Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-nemotron/bin/vllm" serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 128 --max-num-batched-tokens 32768 --async-scheduling --mamba-backend flashinfer --mamba-ssm-cache-dtype float16 --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5 --mamba-cache-mode align --moe-backend humming --speculative_config.method dflash --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash --speculative_config.num_speculative_tokens 3
Prefill
prompt tok/s by context length8K context37.6K218 ms TTFT
16K context41.6K390 ms TTFT
32K context43.6K739 ms TTFT
64K context41.8K1,540 ms TTFT
128K context34.8K3,693 ms TTFT
256K context25.7K9,985 ms TTFT
512K context16.7K30,721 ms TTFT
1019K context9,924103,034 ms TTFT
1024K context9,822104,621 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 293.6124 ms TTFT | 473.9315 ms TTFT | 739.6158 ms TTFT | 1,060.5161 ms TTFT | 1,629.61,221 ms TTFT | 2,501.31,529 ms TTFT | 3,855.22,709 ms TTFT | 5,854.14,908 ms TTFT |
| 16K | 339.8122 ms TTFT | 511.5154 ms TTFT | 758.9157 ms TTFT | 1,009.7201 ms TTFT | 1,649.5798 ms TTFT | 2,609.11,429 ms TTFT | 3,782.92,593 ms TTFT | 5,589.54,357 ms TTFT |
| 32K | 300.8204 ms TTFT | 478.0206 ms TTFT | 725.6261 ms TTFT | 1,019.4284 ms TTFT | 1,630.91,169 ms TTFT | 2,532.01,847 ms TTFT | 3,541.33,231 ms TTFT | 5,146.35,938 ms TTFT |
| 64K | 350.8245 ms TTFT | 469.6249 ms TTFT | 726.3293 ms TTFT | 986.8310 ms TTFT | 1,502.81,251 ms TTFT | 2,276.42,002 ms TTFT | 3,185.03,393 ms TTFT | 4,371.96,678 ms TTFT |
| 128K | 290.1375 ms TTFT | 524.4409 ms TTFT | 709.5437 ms TTFT | 939.3760 ms TTFT | 1,357.61,811 ms TTFT | 1,948.93,204 ms TTFT | 2,596.56,192 ms TTFT | Over capacity |
| 256K | 245.0632 ms TTFT | 421.5710 ms TTFT | 643.6681 ms TTFT | 843.81,783 ms TTFT | 1,130.33,374 ms TTFT | 1,518.06,409 ms TTFT | Over capacity | Over capacity |
| 512K | 220.51,268 ms TTFT | 359.71,530 ms TTFT | 491.01,577 ms TTFT | 655.34,327 ms TTFT | 870.77,651 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | 207.12,687 ms TTFT | 278.8Requests queued2,971 ms TTFT | 433.24,272 ms TTFT | 491.08,558 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 37.6K | — | 218 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 41.6K | — | 390 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 43.6K | — | 739 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 41.8K | — | 1,540 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 34.8K | — | 3,693 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 25.7K | — | 9,985 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 16.7K | — | 30,721 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,924.0 | — | 103,034 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,822.0 | — | 104,621 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 293.6 | 296.1 | 124 ms | 3.4 ms | 1.0 | 2 | 8,807 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 473.9 | 242.5 | 315 ms | 4.1 ms | 2.0 | 3 | 14,215 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 739.6 | 184.4 | 158 ms | 5.4 ms | 4.0 | 4 | 22,185 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,060.5 | 136.9 | 161 ms | 7.3 ms | 8.0 | 8 | 31,811 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,629.6 | 101.6 | 1,221 ms | 9.8 ms | 16.0 | 0 | 48,890 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,501.3 | 77.3 | 1,529 ms | 12.9 ms | 32.0 | 1 | 74,963 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,855.2 | 58.9 | 2,709 ms | 17.0 ms | 64.0 | 0 | 115,643 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,854.1 | 42.4 | 4,908 ms | 23.6 ms | 128.0 | 0 | 175,550 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 339.8 | 325.2 | 122 ms | 3.1 ms | 1.0 | 2 | 10,194 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 511.5 | 251.0 | 154 ms | 4.0 ms | 2.0 | 4 | 15,352 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 758.9 | 185.2 | 157 ms | 5.4 ms | 4.0 | 4 | 22,768 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,009.7 | 132.5 | 201 ms | 7.5 ms | 8.0 | 7 | 30,281 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,649.5 | 102.2 | 798 ms | 9.8 ms | 16.0 | 1 | 49,463 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,609.1 | 77.2 | 1,429 ms | 12.9 ms | 32.0 | 2 | 78,234 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,782.9 | 57.7 | 2,593 ms | 17.3 ms | 64.0 | 0 | 113,456 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,589.5 | 41.0 | 4,357 ms | 24.4 ms | 128.0 | 0 | 167,586 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 300.8 | 310.4 | 204 ms | 3.2 ms | 1.0 | 2 | 9,025 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 478.0 | 245.6 | 206 ms | 4.1 ms | 2.0 | 4 | 14,338 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 725.6 | 186.5 | 261 ms | 5.4 ms | 4.0 | 4 | 21,766 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,019.4 | 130.8 | 284 ms | 7.6 ms | 8.0 | 8 | 30,571 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,630.9 | 98.9 | 1,169 ms | 10.1 ms | 16.0 | 2 | 48,915 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,532.0 | 75.2 | 1,847 ms | 13.3 ms | 32.0 | 0 | 75,910 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,541.3 | 53.8 | 3,231 ms | 18.6 ms | 64.0 | 0 | 106,274 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 5,146.3 | 36.4 | 5,938 ms | 27.5 ms | 128.0 | 0 | 154,237 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 350.8 | 321.9 | 245 ms | 3.1 ms | 1.0 | 2 | 10,523 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 469.6 | 240.4 | 249 ms | 4.2 ms | 2.0 | 3 | 14,089 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 726.3 | 184.5 | 293 ms | 5.4 ms | 4.0 | 4 | 21,784 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 986.8 | 128.3 | 310 ms | 7.8 ms | 8.0 | 7 | 29,596 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,502.8 | 94.5 | 1,251 ms | 10.6 ms | 16.0 | 0 | 45,071 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,276.4 | 68.8 | 2,002 ms | 14.5 ms | 32.0 | 0 | 68,275 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,185.0 | 47.9 | 3,393 ms | 20.9 ms | 64.0 | 0 | 95,547 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,371.9 | 31.3 | 6,678 ms | 32.0 ms | 128.0 | 0 | 131,040 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 290.1 | 314.1 | 375 ms | 3.2 ms | 1.0 | 2 | 8,702 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 524.4 | 256.0 | 409 ms | 3.9 ms | 2.0 | 4 | 15,730 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 709.5 | 173.3 | 437 ms | 5.8 ms | 4.0 | 4 | 21,286 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 939.3 | 123.2 | 760 ms | 8.1 ms | 8.0 | 3 | 28,171 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,357.6 | 84.6 | 1,811 ms | 11.8 ms | 16.0 | 0 | 40,727 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,948.9 | 57.9 | 3,204 ms | 17.3 ms | 32.0 | 0 | 58,454 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,596.5 | 37.6 | 6,192 ms | 26.6 ms | 64.0 | 0 | 77,892 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 245.0 | 259.0 | 632 ms | 3.9 ms | 1.0 | 2 | 7,350 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 421.5 | 214.4 | 710 ms | 4.7 ms | 2.0 | 2 | 12,642 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 643.6 | 160.7 | 681 ms | 6.2 ms | 4.0 | 4 | 19,309 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 843.8 | 102.0 | 1,783 ms | 9.8 ms | 8.0 | 1 | 25,306 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,130.3 | 70.6 | 3,374 ms | 14.2 ms | 16.0 | 0 | 33,888 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,518.0 | 46.0 | 6,409 ms | 21.7 ms | 32.0 | 0 | 45,539 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 220.5 | 233.5 | 1,268 ms | 4.3 ms | 1.0 | 1 | 6,614 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 359.7 | 191.3 | 1,530 ms | 5.2 ms | 1.9 | 2 | 10,789 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 491.0 | 131.5 | 1,577 ms | 7.6 ms | 3.9 | 4 | 14,729 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 655.3 | 84.3 | 4,327 ms | 11.9 ms | 8.0 | 0 | 19,651 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 870.7 | 52.3 | 7,651 ms | 19.1 ms | 16.0 | 0 | 26,095 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 207.1 | 226.3 | 2,687 ms | 4.6 ms | 0.9 | 1 | 6,211 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 278.8 | 153.5 | 2,971 ms | 6.5 ms | 1.9 | 2 | 8,362 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 433.2 | 111.6 | 4,272 ms | 9.0 ms | 3.9 | 1 | 12,989 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 491.0 | 59.6 | 8,558 ms | 16.8 ms | 8.0 | 0 | 14,726 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.