Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-nemotron/bin/vllm" serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 256 --max-num-batched-tokens 16384 --async-scheduling --mamba-backend flashinfer --mamba-ssu-algorithm horizontal --mamba-cache-mode align --mamba-ssm-cache-dtype float16 --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5 --linear-backend humming --moe-backend humming
Prefill
prompt tok/s by context length8K context41K199 ms TTFT
16K context46.3K350 ms TTFT
32K context46K702 ms TTFT
64K context43.3K1,486 ms TTFT
128K context36.6K3,510 ms TTFT
256K context26.6K9,654 ms TTFT
512K context17K30,159 ms TTFT
1019K context9,920103,078 ms TTFT
1024K context9,908103,714 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 292.381 ms TTFT | 514.0142 ms TTFT | 805.3204 ms TTFT | 1,197.8334 ms TTFT | 1,696.7495 ms TTFT | 2,451.7744 ms TTFT | 3,805.21,305 ms TTFT | 5,827.82,322 ms TTFT |
| 16K | 290.5106 ms TTFT | 510.7158 ms TTFT | 804.9213 ms TTFT | 1,183.9345 ms TTFT | 1,656.6432 ms TTFT | 2,384.9717 ms TTFT | 3,681.01,289 ms TTFT | 5,602.42,034 ms TTFT |
| 32K | 287.7107 ms TTFT | 508.2117 ms TTFT | 779.3193 ms TTFT | 1,145.8256 ms TTFT | 1,624.9482 ms TTFT | 2,341.1651 ms TTFT | 3,496.91,094 ms TTFT | 5,112.32,205 ms TTFT |
| 64K | 283.0166 ms TTFT | 495.5200 ms TTFT | 759.8294 ms TTFT | 1,100.2337 ms TTFT | 1,545.9895 ms TTFT | 2,175.91,186 ms TTFT | 3,142.52,163 ms TTFT | 4,411.84,118 ms TTFT |
| 128K | 269.9263 ms TTFT | 477.9334 ms TTFT | 722.3341 ms TTFT | 1,013.0350 ms TTFT | 1,393.01,333 ms TTFT | 1,881.82,549 ms TTFT | 2,584.85,114 ms TTFT | 3,413.89,586 ms TTFT |
| 256K | 248.3561 ms TTFT | 444.0705 ms TTFT | 633.9874 ms TTFT | 911.51,683 ms TTFT | 1,174.72,931 ms TTFT | 1,517.65,318 ms TTFT | 1,935.010,372 ms TTFT | 2,370.121,102 ms TTFT |
| 512K | 219.31,037 ms TTFT | 385.21,277 ms TTFT | 530.41,242 ms TTFT | 742.03,569 ms TTFT | 914.76,628 ms TTFT | 1,093.912,094 ms TTFT | 1,285.623,207 ms TTFT | Over capacity |
| 1019K | 170.82,386 ms TTFT | 303.02,613 ms TTFT | 426.54,818 ms TTFT | 529.28,023 ms TTFT | 615.414,312 ms TTFT | 695.826,748 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 41K | — | 199 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 46.3K | — | 350 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 46K | — | 702 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 43.3K | — | 1,486 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 36.6K | — | 3,510 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 26.6K | — | 9,654 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 17K | — | 30,159 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,920.0 | — | 103,078 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,908.0 | — | 103,714 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 292.3 | 294.0 | 81 ms | 3.4 ms | 1.0 | 2 | 8,770 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 514.0 | 259.5 | 142 ms | 3.9 ms | 2.0 | 4 | 15,428 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 805.3 | 203.0 | 204 ms | 4.9 ms | 4.0 | 4 | 24,160 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,197.8 | 152.5 | 334 ms | 6.6 ms | 8.0 | 8 | 35,931 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,696.7 | 107.3 | 495 ms | 9.3 ms | 16.0 | 0 | 50,897 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,451.7 | 77.3 | 744 ms | 12.9 ms | 32.0 | 0 | 73,535 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,805.2 | 59.5 | 1,305 ms | 16.8 ms | 64.0 | 0 | 114,178 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,827.8 | 44.3 | 2,322 ms | 22.6 ms | 128.0 | 0 | 174,798 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 290.5 | 292.7 | 106 ms | 3.4 ms | 1.0 | 2 | 8,713 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 510.7 | 258.3 | 158 ms | 3.9 ms | 2.0 | 4 | 15,320 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 804.9 | 202.9 | 213 ms | 4.9 ms | 4.0 | 4 | 24,144 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,183.9 | 150.2 | 345 ms | 6.7 ms | 8.0 | 8 | 35,518 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,656.6 | 104.8 | 432 ms | 9.5 ms | 16.0 | 0 | 49,698 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,384.9 | 75.3 | 717 ms | 13.3 ms | 32.0 | 0 | 71,551 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,681.0 | 58.0 | 1,289 ms | 17.2 ms | 64.0 | 0 | 110,402 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,602.4 | 43.0 | 2,034 ms | 23.2 ms | 128.0 | 0 | 168,066 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 287.7 | 290.0 | 107 ms | 3.4 ms | 1.0 | 2 | 8,631 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 508.2 | 256.1 | 117 ms | 3.9 ms | 2.0 | 4 | 15,246 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 779.3 | 196.7 | 193 ms | 5.1 ms | 4.0 | 4 | 23,379 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,145.8 | 145.7 | 256 ms | 6.9 ms | 8.0 | 8 | 34,370 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,624.9 | 102.4 | 482 ms | 9.8 ms | 16.0 | 0 | 48,735 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,341.1 | 73.2 | 651 ms | 13.7 ms | 32.0 | 0 | 70,237 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,496.9 | 54.6 | 1,094 ms | 18.3 ms | 64.0 | 0 | 104,900 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 5,112.3 | 39.5 | 2,205 ms | 25.3 ms | 128.0 | 0 | 153,340 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 283.0 | 286.2 | 166 ms | 3.5 ms | 1.0 | 2 | 8,488 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 495.5 | 251.2 | 200 ms | 4.0 ms | 1.9 | 4 | 14,867 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 759.8 | 192.1 | 294 ms | 5.2 ms | 3.9 | 4 | 22,794 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,100.2 | 138.8 | 337 ms | 7.2 ms | 7.9 | 8 | 33,003 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,545.9 | 96.7 | 895 ms | 10.3 ms | 16.0 | 0 | 46,370 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,175.9 | 67.6 | 1,186 ms | 14.8 ms | 32.0 | 0 | 65,279 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,142.5 | 48.4 | 2,163 ms | 20.6 ms | 64.0 | 0 | 94,274 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,411.8 | 33.7 | 4,118 ms | 29.6 ms | 128.0 | 0 | 132,398 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 269.9 | 274.9 | 263 ms | 3.6 ms | 1.0 | 2 | 8,099 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 477.9 | 242.4 | 334 ms | 4.1 ms | 2.0 | 4 | 14,256 | 29.8 s |
| Sustained decode | 128K | 4 | Valid | 722.3 | 182.2 | 341 ms | 5.5 ms | 4.0 | 4 | 21,670 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,013.0 | 128.2 | 350 ms | 7.8 ms | 7.9 | 8 | 30,388 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,393.0 | 86.8 | 1,333 ms | 11.5 ms | 16.0 | 0 | 41,793 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,881.8 | 58.1 | 2,549 ms | 17.2 ms | 32.0 | 0 | 56,448 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,584.8 | 39.6 | 5,114 ms | 25.2 ms | 64.0 | 0 | 77,507 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 3,413.8 | 25.8 | 9,586 ms | 38.8 ms | 128.0 | 0 | 102,410 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 248.3 | 258.0 | 561 ms | 3.9 ms | 0.9 | 2 | 7,448 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 444.0 | 227.6 | 705 ms | 4.4 ms | 1.9 | 2 | 13,319 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 633.9 | 162.3 | 874 ms | 6.2 ms | 3.9 | 4 | 19,017 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 911.5 | 112.9 | 1,683 ms | 8.9 ms | 8.0 | 0 | 27,344 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,174.7 | 72.1 | 2,931 ms | 13.9 ms | 16.0 | 0 | 35,232 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,517.6 | 45.8 | 5,318 ms | 21.8 ms | 32.0 | 0 | 45,505 | 30.0 s |
| Sustained decode | 256K | 64 | Valid | 1,935.0 | 28.7 | 10,372 ms | 34.8 ms | 64.0 | 0 | 58,045 | 30.0 s |
| Sustained decode | 256K | 128 | Valid | 2,370.1 | 17.2 | 21,102 ms | 58.0 ms | 128.0 | 0 | 71,040 | 30.0 s |
| Sustained decode | 512K | 1 | Valid | 219.3 | 228.1 | 1,037 ms | 4.4 ms | 1.0 | 1 | 6,579 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 385.2 | 201.4 | 1,277 ms | 5.0 ms | 1.9 | 2 | 11,557 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 530.4 | 139.3 | 1,242 ms | 7.2 ms | 3.9 | 4 | 15,911 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 742.0 | 92.7 | 3,569 ms | 10.8 ms | 8.0 | 0 | 22,256 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 914.7 | 56.7 | 6,628 ms | 17.6 ms | 16.0 | 0 | 27,442 | 30.0 s |
| Sustained decode | 512K | 32 | Valid | 1,093.9 | 33.4 | 12,094 ms | 29.9 ms | 32.0 | 0 | 32,803 | 30.0 s |
| Sustained decode | 512K | 64 | Valid | 1,285.6 | 19.5 | 23,207 ms | 51.2 ms | 64.0 | 0 | 38,528 | 30.0 s |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 170.8 | 187.1 | 2,386 ms | 5.3 ms | 0.9 | 1 | 5,123 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 303.0 | 167.0 | 2,613 ms | 6.0 ms | 1.9 | 2 | 9,091 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 426.5 | 108.7 | 4,818 ms | 9.2 ms | 4.0 | 1 | 12,795 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 529.2 | 66.6 | 8,023 ms | 15.0 ms | 8.0 | 0 | 15,872 | 30.0 s |
| Sustained decode | 1019K | 16 | Valid | 615.4 | 38.6 | 14,312 ms | 25.9 ms | 16.0 | 0 | 18,449 | 30.0 s |
| Sustained decode | 1019K | 32 | Valid | 695.8 | 21.1 | 26,748 ms | 47.5 ms | 32.0 | 0 | 20,865 | 30.0 s |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.