Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on H200 NVL 141GB
sglang · 0.0.0.dev17157+gccfcdd93a · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev17157+gccfcdd93a
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/rystewart-nvidia/sglang
- Build
- cuda-13.2
- Commit
ccfcdd93a8c2
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang-pr-33554/bin/sglang" serve --model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nemotron --trust-remote-code --tp 1 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 1048576 --mem-fraction-static 0.858838 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-ssm-dtype float16 --cuda-graph-max-bs-decode 16 --speculative-algorithm DSPARK --speculative-draft-model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark --speculative-dspark-block-size 3
Prefill
prompt tok/s by context length8K context25.9K316 ms TTFT
16K context29.1K558 ms TTFT
32K context28.7K1,126 ms TTFT
64K context27.4K2,352 ms TTFT
128K context24.4K5,286 ms TTFT
256K context19.6K13,161 ms TTFT
512K context13.9K36,993 ms TTFT
1019K context8,786116,529 ms TTFT
1024K context8,758117,470 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 447.472 ms TTFT | 787.168 ms TTFT | 1,228.179 ms TTFT | 1,937.978 ms TTFT | 1,118.2202 ms TTFT | 2,202.8364 ms TTFT | Not run | Not run |
| 16K | 454.879 ms TTFT | 446.5124 ms TTFT | 1,226.992 ms TTFT | 1,959.2100 ms TTFT | 1,076.9356 ms TTFT | 2,175.7706 ms TTFT | Not run | Not run |
| 32K | 483.8106 ms TTFT | 664.9123 ms TTFT | 1,208.6128 ms TTFT | 1,869.4119 ms TTFT | 1,172.1745 ms TTFT | 2,297.01,413 ms TTFT | Not run | Not run |
| 64K | 433.2151 ms TTFT | 414.7191 ms TTFT | 1,370.7212 ms TTFT | 1,790.1218 ms TTFT | 1,787.21,436 ms TTFT | 2,230.22,780 ms TTFT | Not run | Over capacity |
| 128K | 451.6243 ms TTFT | 345.3327 ms TTFT | 1,112.2253 ms TTFT | 1,734.1275 ms TTFT | 1,091.22,737 ms TTFT | 2,223.85,423 ms TTFT | Over capacity | Over capacity |
| 256K | 396.4457 ms TTFT | 373.7792 ms TTFT | 1,117.5600 ms TTFT | 1,419.02,398 ms TTFT | 1,593.96,076 ms TTFT | Over capacity | Over capacity | Over capacity |
| 512K | 403.5826 ms TTFT | 607.9810 ms TTFT | 975.61,477 ms TTFT | 1,133.03,931 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 354.41,712 ms TTFT | 329.92,332 ms TTFT | 743.3Requests queued4,627 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 25.9K | — | 316 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.1K | — | 558 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 28.7K | — | 1,126 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 27.4K | — | 2,352 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 24.4K | — | 5,286 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 19.6K | — | 13,161 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 13.9K | — | 36,993 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,786.0 | — | 116,529 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 8,758.0 | — | 117,470 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 447.4 | 454.7 | 72 ms | 2.2 ms | 0.9 | 3 | 13,421 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 787.1 | 393.7 | 68 ms | 2.5 ms | 2.0 | 6 | 23,612 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,228.1 | 312.2 | 79 ms | 3.2 ms | 4.1 | 8 | 36,837 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,937.9 | 236.5 | 78 ms | 4.2 ms | 8.1 | 11 | 58,122 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,118.2 | 70.6 | 202 ms | 14.2 ms | 18.5 | 0 | 33,526 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,202.8 | 69.5 | 364 ms | 14.4 ms | 39.7 | 0 | 66,079 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 454.8 | 451.9 | 79 ms | 2.2 ms | 1.0 | 3 | 13,642 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 446.5 | 270.5 | 124 ms | 4.5 ms | 15.7 | 2 | 13,393 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,226.9 | 307.6 | 92 ms | 3.3 ms | 4.0 | 8 | 36,807 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,959.2 | 233.5 | 100 ms | 4.3 ms | 8.2 | 12 | 58,775 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,076.9 | 68.6 | 356 ms | 14.6 ms | 19.7 | 0 | 32,287 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,175.7 | 68.5 | 706 ms | 14.6 ms | 32.3 | 5 | 65,214 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 483.8 | 483.4 | 106 ms | 2.1 ms | 1.0 | 3 | 14,515 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 664.9 | 386.5 | 123 ms | 2.6 ms | 4.3 | 4 | 19,947 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,208.6 | 303.6 | 128 ms | 3.3 ms | 3.9 | 8 | 36,257 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,869.4 | 237.0 | 119 ms | 4.2 ms | 8.4 | 10 | 56,076 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,172.1 | 71.4 | 745 ms | 14.0 ms | 17.6 | 0 | 35,128 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,297.0 | 72.1 | 1,413 ms | 13.9 ms | 37.6 | 0 | 68,857 | 30.0 s |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 433.2 | 441.5 | 151 ms | 2.3 ms | 1.0 | 3 | 12,994 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 414.7 | 256.3 | 191 ms | 4.6 ms | 17.6 | 2 | 12,440 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,370.7 | 318.2 | 212 ms | 3.1 ms | 4.0 | 8 | 41,120 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,790.1 | 230.5 | 218 ms | 4.3 ms | 7.9 | 9 | 53,693 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,787.2 | 111.0 | 1,436 ms | 9.0 ms | 16.5 | 2 | 53,609 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,230.2 | 72.7 | 2,780 ms | 13.8 ms | 33.9 | 0 | 66,856 | 30.0 s |
| Sustained decode | 64K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 451.6 | 483.9 | 243 ms | 2.1 ms | 0.9 | 3 | 13,547 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 345.3 | 266.0 | 327 ms | 4.9 ms | 20.0 | 2 | 10,358 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,112.2 | 284.8 | 253 ms | 3.5 ms | 4.2 | 8 | 33,359 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,734.1 | 213.6 | 275 ms | 4.7 ms | 8.7 | 9 | 52,014 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,091.2 | 71.9 | 2,737 ms | 13.9 ms | 17.7 | 0 | 32,705 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,223.8 | 75.9 | 5,423 ms | 13.2 ms | 38.6 | 0 | 66,661 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 396.4 | 427.3 | 457 ms | 2.3 ms | 1.0 | 3 | 11,891 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 373.7 | 256.4 | 792 ms | 4.8 ms | 16.6 | 2 | 11,211 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 1,117.5 | 284.3 | 600 ms | 3.5 ms | 3.9 | 8 | 33,526 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 1,419.0 | 192.9 | 2,398 ms | 5.2 ms | 8.6 | 8 | 42,557 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,593.9 | 101.8 | 6,076 ms | 9.8 ms | 18.7 | 1 | 47,818 | 30.0 s |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 403.5 | 403.2 | 826 ms | 2.5 ms | 0.9 | 3 | 12,103 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 607.9 | 326.2 | 810 ms | 3.1 ms | 2.2 | 4 | 18,239 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 975.6 | 270.2 | 1,477 ms | 3.7 ms | 3.8 | 7 | 29,268 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 1,133.0 | 166.8 | 3,931 ms | 6.0 ms | 8.8 | 8 | 33,985 | 30.0 s |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 354.4 | 360.8 | 1,712 ms | 2.8 ms | 0.9 | 2 | 10,631 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 329.9 | 226.4 | 2,332 ms | 4.9 ms | 5.9 | 2 | 9,896 | 30.0 s |
| Sustained decode | 1019K | 4 | Requests queued | 743.3 | 203.9 | 4,627 ms | 4.9 ms | 3.8 | 4 | 22,294 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.