Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev9335+gdd15fb57b · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nemotron --trust-remote-code --tp 1 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 1048576 --mem-fraction-static 0.852636 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-ssm-dtype float16 --cuda-graph-max-bs-decode 16
Prefill
prompt tok/s by context length8K context26.4K310 ms TTFT
16K context29.2K557 ms TTFT
32K context29.2K1,105 ms TTFT
64K context28.1K2,296 ms TTFT
128K context24.8K5,193 ms TTFT
256K context19.7K13,053 ms TTFT
512K context14K36,697 ms TTFT
1019K context8,886115,208 ms TTFT
1024K context8,851116,237 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 352.463 ms TTFT | 615.185 ms TTFT | 1,019.7108 ms TTFT | 1,497.2141 ms TTFT | 551.0183 ms TTFT | 1,091.2332 ms TTFT | 2,161.1611 ms TTFT | 4,142.01,121 ms TTFT |
| 16K | 349.878 ms TTFT | 65.0123 ms TTFT | 132.0139 ms TTFT | 929.1204 ms TTFT | 2,360.1359 ms TTFT | 1,103.2678 ms TTFT | 2,136.61,283 ms TTFT | 4,199.02,470 ms TTFT |
| 32K | 345.8106 ms TTFT | 64.9171 ms TTFT | 131.1220 ms TTFT | 870.1404 ms TTFT | 2,165.3647 ms TTFT | 1,096.81,387 ms TTFT | 2,166.42,762 ms TTFT | 3,451.15,626 ms TTFT |
| 64K | 339.0146 ms TTFT | 58.0260 ms TTFT | 119.2403 ms TTFT | 384.5790 ms TTFT | 543.91,477 ms TTFT | 1,082.02,980 ms TTFT | 2,083.45,735 ms TTFT | 2,666.711,452 ms TTFT |
| 128K | 327.1237 ms TTFT | 43.0401 ms TTFT | 87.2790 ms TTFT | 200.91,466 ms TTFT | 458.32,908 ms TTFT | 1,088.75,446 ms TTFT | 1,601.311,042 ms TTFT | 1,891.323,291 ms TTFT |
| 256K | 304.3409 ms TTFT | 29.3845 ms TTFT | 61.11,501 ms TTFT | 121.32,931 ms TTFT | 256.65,779 ms TTFT | 496.211,566 ms TTFT | 1,069.522,951 ms TTFT | Over capacity |
| 512K | 266.5795 ms TTFT | 33.31,551 ms TTFT | 66.42,903 ms TTFT | 100.05,733 ms TTFT | 201.611,429 ms TTFT | 403.822,688 ms TTFT | Over capacity | Over capacity |
| 1019K | 220.61,649 ms TTFT | 32.53,099 ms TTFT | 43.26,415 ms TTFT | 86.412,657 ms TTFT | 170.224,995 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 26.4K | — | 310 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.2K | — | 557 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.2K | — | 1,105 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.1K | — | 2,296 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 24.8K | — | 5,193 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.7K | — | 13,053 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 14K | — | 36,697 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,886.0 | — | 115,208 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 8,851.0 | — | 116,237 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 352.4 | 354.0 | 63 ms | 2.8 ms | 1.0 | 2 | 10,571 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 615.1 | 308.9 | 85 ms | 3.2 ms | 2.0 | 4 | 18,452 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,019.7 | 258.3 | 108 ms | 3.9 ms | 4.0 | 8 | 30,588 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,497.2 | 190.9 | 141 ms | 5.3 ms | 9.9 | 8 | 44,912 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 551.0 | 34.4 | 183 ms | 29.1 ms | 24.0 | 0 | 16,528 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,091.2 | 34.1 | 332 ms | 29.3 ms | 48.0 | 0 | 32,737 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,161.1 | 33.7 | 611 ms | 29.7 ms | 112.0 | 0 | 64,833 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,142.0 | 32.3 | 1,121 ms | 31.0 ms | 228.3 | 0 | 124,288 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 349.8 | 351.5 | 78 ms | 2.8 ms | 1.0 | 2 | 10,495 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 65.0 | 32.4 | 123 ms | 30.9 ms | 204.7 | 0 | 1,948 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 132.0 | 32.9 | 139 ms | 30.4 ms | 157.5 | 0 | 3,956 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 929.1 | 105.5 | 204 ms | 9.5 ms | 59.7 | 0 | 27,874 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,360.1 | 151.9 | 359 ms | 6.6 ms | 16.0 | 16 | 70,800 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,103.2 | 34.4 | 678 ms | 29.0 ms | 48.0 | 0 | 33,088 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,136.6 | 33.3 | 1,283 ms | 30.0 ms | 112.0 | 0 | 64,064 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,199.0 | 32.7 | 2,470 ms | 30.6 ms | 226.1 | 0 | 125,984 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 345.8 | 348.3 | 106 ms | 2.9 ms | 1.0 | 2 | 10,374 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 64.9 | 32.4 | 171 ms | 30.8 ms | 204.7 | 0 | 1,946 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 131.1 | 32.7 | 220 ms | 30.6 ms | 159.6 | 0 | 3,932 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 870.1 | 99.5 | 404 ms | 10.1 ms | 59.9 | 0 | 26,104 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,165.3 | 125.0 | 647 ms | 8.0 ms | 16.3 | 16 | 64,961 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,096.8 | 34.4 | 1,387 ms | 29.0 ms | 48.0 | 0 | 32,896 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,166.4 | 34.4 | 2,762 ms | 29.1 ms | 112.0 | 0 | 64,960 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,451.1 | 28.1 | 5,626 ms | 35.6 ms | 240.0 | 0 | 103,540 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 339.0 | 342.0 | 146 ms | 2.9 ms | 1.0 | 2 | 10,170 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 58.0 | 28.8 | 260 ms | 34.7 ms | 211.1 | 0 | 1,738 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 119.2 | 29.8 | 403 ms | 33.6 ms | 178.7 | 0 | 3,572 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 384.5 | 46.3 | 790 ms | 21.6 ms | 116.9 | 0 | 11,536 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 543.9 | 34.4 | 1,477 ms | 29.1 ms | 27.6 | 0 | 16,304 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,082.0 | 34.8 | 2,980 ms | 28.8 ms | 53.9 | 0 | 32,461 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,083.4 | 34.5 | 5,735 ms | 29.0 ms | 112.0 | 0 | 62,464 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 2,666.7 | 23.5 | 11,452 ms | 42.6 ms | 224.0 | 0 | 80,003 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 327.1 | 332.2 | 237 ms | 3.0 ms | 0.9 | 2 | 9,813 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 43.0 | 21.4 | 401 ms | 46.7 ms | 219.0 | 0 | 1,288 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 87.2 | 21.9 | 790 ms | 45.6 ms | 198.0 | 0 | 2,616 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 200.9 | 24.9 | 1,466 ms | 40.1 ms | 160.0 | 0 | 6,024 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 458.3 | 29.2 | 2,908 ms | 34.2 ms | 119.5 | 0 | 13,744 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,088.7 | 36.2 | 5,446 ms | 27.6 ms | 60.4 | 0 | 32,640 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 1,601.3 | 28.5 | 11,042 ms | 35.1 ms | 115.7 | 0 | 48,000 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 1,891.3 | 19.5 | 23,291 ms | 51.4 ms | 225.6 | 0 | 56,703 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 304.3 | 312.5 | 409 ms | 3.2 ms | 0.9 | 2 | 9,130 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 29.3 | 14.7 | 845 ms | 67.8 ms | 226.0 | 0 | 880 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 61.1 | 15.4 | 1,501 ms | 65.1 ms | 203.7 | 0 | 1,832 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 121.3 | 15.6 | 2,931 ms | 63.9 ms | 206.0 | 0 | 3,632 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 256.6 | 16.9 | 5,779 ms | 59.2 ms | 197.5 | 0 | 7,696 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 496.2 | 17.9 | 11,566 ms | 55.9 ms | 190.0 | 0 | 14,880 | 30.0 s |
| Sustained decode | 256K | 64 | Valid | 1,069.5 | 20.6 | 22,951 ms | 48.5 ms | 127.3 | 0 | 32,062 | 30.0 s |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 266.5 | 281.1 | 795 ms | 3.6 ms | 0.9 | 2 | 7,994 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 33.3 | 16.8 | 1,551 ms | 59.6 ms | 126.0 | 0 | 996 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 66.4 | 17.0 | 2,903 ms | 58.7 ms | 125.4 | 0 | 1,992 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 100.0 | 13.4 | 5,733 ms | 74.7 ms | 125.8 | 0 | 3,000 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 201.6 | 14.5 | 11,429 ms | 69.2 ms | 126.0 | 0 | 6,048 | 30.0 s |
| Sustained decode | 512K | 32 | Valid | 403.8 | 16.4 | 22,688 ms | 61.0 ms | 126.0 | 0 | 12,096 | 30.0 s |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 220.6 | 233.3 | 1,649 ms | 4.3 ms | 1.0 | 1 | 6,618 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 32.5 | 16.1 | 3,099 ms | 62.1 ms | 73.6 | 0 | 974 | 29.9 s |
| Sustained decode | 1019K | 4 | Valid | 43.2 | 11.5 | 6,415 ms | 87.2 ms | 66.8 | 0 | 1,296 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 86.4 | 12.5 | 12,657 ms | 80.2 ms | 72.9 | 0 | 2,592 | 30.0 s |
| Sustained decode | 1019K | 16 | Valid | 170.2 | 14.2 | 24,995 ms | 70.6 ms | 84.2 | 0 | 5,104 | 30.0 s |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.