Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · BF168 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --served-model-name nemotron --trust-remote-code --tp 1 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 1048576 --mem-fraction-static 0.862866 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-ssm-dtype float16 --cuda-graph-max-bs-decode 16
Prefill
prompt tok/s by context length8K context18.5K443 ms TTFT
16K context19.7K823 ms TTFT
32K context19.8K1,636 ms TTFT
64K context19.3K3,341 ms TTFT
128K context17.7K7,291 ms TTFT
256K context14.8K17,342 ms TTFT
512K context11.1K46,527 ms TTFT
1019K context7,259141,030 ms TTFT
1024K context7,233142,240 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 282.148 ms TTFT | 439.161 ms TTFT | 729.092 ms TTFT | 1,023.6132 ms TTFT | 956.1185 ms TTFT | 1,792.4320 ms TTFT | 2,959.6627 ms TTFT | 4,501.91,134 ms TTFT |
| 16K | 279.774 ms TTFT | 70.5121 ms TTFT | 155.6132 ms TTFT | 888.1237 ms TTFT | 1,556.8344 ms TTFT | 1,807.3655 ms TTFT | 2,816.01,251 ms TTFT | 4,224.62,498 ms TTFT |
| 32K | 276.898 ms TTFT | 66.3147 ms TTFT | 147.6225 ms TTFT | 724.2376 ms TTFT | 1,300.5768 ms TTFT | 1,734.61,404 ms TTFT | 2,487.82,761 ms TTFT | 3,523.45,521 ms TTFT |
| 64K | 272.6140 ms TTFT | 57.2265 ms TTFT | 117.3431 ms TTFT | 315.5783 ms TTFT | 927.21,515 ms TTFT | 1,498.92,965 ms TTFT | 2,147.55,989 ms TTFT | Over capacity |
| 128K | 263.2228 ms TTFT | 67.4411 ms TTFT | 144.2749 ms TTFT | 643.01,462 ms TTFT | 1,074.42,843 ms TTFT | 1,283.65,583 ms TTFT | Over capacity | Over capacity |
| 256K | 246.6405 ms TTFT | 87.0798 ms TTFT | 292.51,480 ms TTFT | 788.92,838 ms TTFT | 1,134.95,600 ms TTFT | Over capacity | Over capacity | Over capacity |
| 512K | 224.1798 ms TTFT | 165.11,548 ms TTFT | 505.62,948 ms TTFT | 521.35,908 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 183.61,647 ms TTFT | 165.53,183 ms TTFT | 410.96,102 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.5K | — | 443 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 19.7K | — | 823 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.8K | — | 1,636 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 19.3K | — | 3,341 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.7K | — | 7,291 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 14.8K | — | 17,342 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 11.1K | — | 46,527 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 7,259.0 | — | 141,030 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 7,233.0 | — | 142,240 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 282.1 | 283.1 | 48 ms | 3.5 ms | 1.0 | 2 | 8,463 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 439.1 | 215.5 | 61 ms | 4.7 ms | 2.2 | 2 | 13,171 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 729.0 | 180.0 | 92 ms | 5.6 ms | 4.0 | 4 | 21,871 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,023.6 | 141.4 | 132 ms | 7.2 ms | 9.3 | 8 | 30,706 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 956.1 | 59.6 | 185 ms | 16.8 ms | 24.0 | 0 | 28,672 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,792.4 | 56.1 | 320 ms | 17.8 ms | 52.5 | 0 | 53,760 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,959.6 | 46.6 | 627 ms | 21.4 ms | 96.0 | 0 | 88,768 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,501.9 | 35.0 | 1,134 ms | 28.6 ms | 199.5 | 0 | 135,040 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 279.7 | 281.1 | 74 ms | 3.6 ms | 1.0 | 2 | 8,392 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 70.5 | 35.2 | 121 ms | 28.4 ms | 189.7 | 0 | 2,114 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 155.6 | 38.9 | 132 ms | 25.7 ms | 134.0 | 0 | 4,668 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 888.1 | 106.5 | 237 ms | 9.4 ms | 11.8 | 0 | 26,640 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,556.8 | 96.6 | 344 ms | 10.4 ms | 16.0 | 0 | 46,704 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,807.3 | 56.2 | 655 ms | 17.8 ms | 35.2 | 0 | 54,224 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,816.0 | 44.3 | 1,251 ms | 22.6 ms | 96.0 | 0 | 84,480 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,224.6 | 32.8 | 2,498 ms | 30.5 ms | 203.7 | 0 | 126,720 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 276.8 | 278.6 | 98 ms | 3.6 ms | 1.0 | 2 | 8,304 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 66.3 | 33.2 | 147 ms | 30.2 ms | 194.0 | 0 | 1,988 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 147.6 | 36.6 | 225 ms | 27.3 ms | 134.0 | 0 | 4,428 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 724.2 | 83.9 | 376 ms | 11.9 ms | 34.4 | 0 | 21,728 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,300.5 | 78.7 | 768 ms | 12.7 ms | 18.9 | 0 | 39,008 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,734.6 | 54.6 | 1,404 ms | 18.3 ms | 41.6 | 0 | 52,032 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,487.8 | 40.0 | 2,761 ms | 25.0 ms | 96.0 | 0 | 74,624 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,523.4 | 28.9 | 5,521 ms | 34.6 ms | 210.0 | 0 | 105,728 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 272.6 | 275.0 | 140 ms | 3.6 ms | 1.0 | 2 | 8,177 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 57.2 | 28.8 | 265 ms | 34.7 ms | 194.0 | 0 | 1,716 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 117.3 | 29.2 | 431 ms | 34.2 ms | 166.4 | 0 | 3,516 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 315.5 | 38.5 | 783 ms | 26.0 ms | 121.8 | 0 | 9,464 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 927.2 | 58.7 | 1,515 ms | 17.0 ms | 27.0 | 0 | 27,808 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,498.9 | 48.2 | 2,965 ms | 20.8 ms | 48.8 | 0 | 44,960 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,147.5 | 35.9 | 5,989 ms | 27.8 ms | 96.0 | 0 | 64,384 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 263.2 | 267.0 | 228 ms | 3.7 ms | 1.0 | 2 | 7,896 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 67.4 | 33.7 | 411 ms | 29.7 ms | 92.8 | 0 | 2,022 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 144.2 | 36.2 | 749 ms | 27.7 ms | 70.0 | 0 | 4,324 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 643.0 | 76.1 | 1,462 ms | 13.1 ms | 17.0 | 0 | 19,288 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,074.4 | 67.5 | 2,843 ms | 14.8 ms | 20.3 | 0 | 32,224 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,283.6 | 43.0 | 5,583 ms | 23.2 ms | 47.8 | 0 | 38,496 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 246.6 | 253.5 | 405 ms | 3.9 ms | 0.9 | 2 | 7,399 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 87.0 | 43.7 | 798 ms | 22.9 ms | 34.0 | 0 | 2,608 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 292.5 | 70.4 | 1,480 ms | 14.2 ms | 23.0 | 0 | 8,776 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 788.9 | 99.9 | 2,838 ms | 10.0 ms | 9.3 | 0 | 23,664 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,134.9 | 72.8 | 5,600 ms | 13.7 ms | 16.1 | 0 | 34,048 | 30.0 s |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 224.1 | 230.1 | 798 ms | 4.3 ms | 1.0 | 1 | 6,722 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 165.1 | 79.6 | 1,548 ms | 12.6 ms | 13.9 | 0 | 4,952 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 505.6 | 131.2 | 2,948 ms | 7.6 ms | 4.0 | 4 | 14,914 | 29.5 s |
| Sustained decode | 512K | 8 | Valid | 521.3 | 70.7 | 5,908 ms | 14.2 ms | 12.0 | 0 | 15,640 | 30.0 s |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 183.6 | 194.3 | 1,647 ms | 5.1 ms | 1.0 | 1 | 5,508 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 165.5 | 80.6 | 3,183 ms | 12.4 ms | 8.1 | 0 | 4,964 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 410.9 | 106.7 | 6,102 ms | 9.4 ms | 4.0 | 0 | 12,328 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.