Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-nemotron/bin/vllm" serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 128 --max-num-batched-tokens 32768 --async-scheduling --mamba-backend flashinfer --mamba-ssm-cache-dtype float16 --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5 --mamba-cache-mode align --moe-backend humming --speculative_config.method dspark --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark --speculative_config.num_speculative_tokens 3
Prefill
prompt tok/s by context length8K context38.4K213 ms TTFT
16K context41.1K394 ms TTFT
32K context43.3K745 ms TTFT
64K context41.3K1,558 ms TTFT
128K context34.8K3,700 ms TTFT
256K context25.8K9,979 ms TTFT
512K context16.8K30,568 ms TTFT
1019K context9,921103,071 ms TTFT
1024K context9,809104,752 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 413.8124 ms TTFT | 685.8130 ms TTFT | 1,014.9135 ms TTFT | 1,403.3172 ms TTFT | 2,194.8400 ms TTFT | 3,357.71,214 ms TTFT | 5,177.62,456 ms TTFT | 7,888.94,637 ms TTFT |
| 16K | 436.8142 ms TTFT | 661.8151 ms TTFT | 992.0161 ms TTFT | 1,368.7220 ms TTFT | 2,105.2221 ms TTFT | 3,381.11,396 ms TTFT | 5,217.22,579 ms TTFT | 7,685.74,330 ms TTFT |
| 32K | 402.5183 ms TTFT | 656.1192 ms TTFT | 993.1224 ms TTFT | 1,369.8258 ms TTFT | 2,094.6513 ms TTFT | 3,248.62,138 ms TTFT | 5,011.83,090 ms TTFT | 7,496.75,858 ms TTFT |
| 64K | 398.4238 ms TTFT | 640.9249 ms TTFT | 1,004.6295 ms TTFT | 1,298.6320 ms TTFT | 2,020.6700 ms TTFT | 3,133.41,942 ms TTFT | 4,707.53,305 ms TTFT | 6,675.86,503 ms TTFT |
| 128K | 418.7376 ms TTFT | 682.6400 ms TTFT | 957.9418 ms TTFT | 1,312.1464 ms TTFT | 1,906.41,641 ms TTFT | 2,941.12,795 ms TTFT | 4,187.65,546 ms TTFT | 5,687.810,536 ms TTFT |
| 256K | 356.4642 ms TTFT | 622.4737 ms TTFT | 854.8744 ms TTFT | 1,170.0899 ms TTFT | 1,749.73,166 ms TTFT | 2,543.16,237 ms TTFT | 3,399.211,654 ms TTFT | Over capacity |
| 512K | 343.91,287 ms TTFT | 524.6Requests queued1,464 ms TTFT | 731.81,687 ms TTFT | 1,019.22,372 ms TTFT | 1,500.65,902 ms TTFT | 2,042.211,955 ms TTFT | Over capacity | Over capacity |
| 1019K | 288.92,748 ms TTFT | 448.9Requests queued3,149 ms TTFT | 594.8Requests queued4,749 ms TTFT | 812.36,681 ms TTFT | 1,141.813,180 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 38.4K | — | 213 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 41.1K | — | 394 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 43.3K | — | 745 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 41.3K | — | 1,558 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 34.8K | — | 3,700 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 25.8K | — | 9,979 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 16.8K | — | 30,568 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,921.0 | — | 103,071 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,809.0 | — | 104,752 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 413.8 | 416.6 | 124 ms | 2.4 ms | 1.0 | 3 | 12,412 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 685.8 | 343.8 | 130 ms | 2.9 ms | 2.0 | 5 | 20,570 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,014.9 | 259.0 | 135 ms | 3.9 ms | 4.0 | 7 | 30,457 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,403.3 | 179.2 | 172 ms | 5.6 ms | 8.0 | 8 | 42,099 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,194.8 | 133.0 | 400 ms | 7.5 ms | 16.0 | 16 | 65,831 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,357.7 | 101.8 | 1,214 ms | 9.8 ms | 32.0 | 4 | 100,734 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,177.6 | 77.9 | 2,456 ms | 12.8 ms | 64.0 | 0 | 155,321 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,888.9 | 56.6 | 4,637 ms | 17.7 ms | 128.0 | 0 | 236,585 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 436.8 | 429.6 | 142 ms | 2.3 ms | 1.0 | 3 | 13,105 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 661.8 | 339.5 | 151 ms | 2.9 ms | 2.0 | 4 | 19,852 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 992.0 | 249.0 | 161 ms | 4.0 ms | 4.0 | 7 | 29,760 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,368.7 | 175.1 | 220 ms | 5.7 ms | 8.0 | 8 | 41,054 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,105.2 | 130.8 | 221 ms | 7.6 ms | 15.9 | 16 | 63,090 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,381.1 | 103.5 | 1,396 ms | 9.7 ms | 32.0 | 3 | 101,415 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,217.2 | 78.3 | 2,579 ms | 12.8 ms | 64.0 | 0 | 156,524 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 7,685.7 | 56.5 | 4,330 ms | 17.7 ms | 128.0 | 0 | 230,580 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 402.5 | 410.4 | 183 ms | 2.4 ms | 1.0 | 3 | 12,074 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 656.1 | 330.9 | 192 ms | 3.0 ms | 2.0 | 4 | 19,681 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 993.1 | 249.1 | 224 ms | 4.0 ms | 4.0 | 7 | 29,756 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,369.8 | 174.9 | 258 ms | 5.7 ms | 8.0 | 8 | 41,091 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,094.6 | 127.8 | 513 ms | 7.8 ms | 16.0 | 13 | 62,770 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,248.6 | 99.6 | 2,138 ms | 10.0 ms | 32.0 | 1 | 97,416 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,011.8 | 74.6 | 3,090 ms | 13.4 ms | 64.0 | 0 | 150,382 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 7,496.7 | 52.6 | 5,858 ms | 19.0 ms | 128.0 | 0 | 224,789 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 398.4 | 420.0 | 238 ms | 2.4 ms | 0.9 | 3 | 11,953 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 640.9 | 331.3 | 249 ms | 3.0 ms | 2.0 | 4 | 19,223 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,004.6 | 246.9 | 295 ms | 4.1 ms | 3.9 | 6 | 30,140 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,298.6 | 168.2 | 320 ms | 5.9 ms | 8.0 | 8 | 38,951 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,020.6 | 124.6 | 700 ms | 8.0 ms | 15.9 | 12 | 60,430 | 29.9 s |
| Sustained decode | 64K | 32 | Valid | 3,133.4 | 94.0 | 1,942 ms | 10.6 ms | 32.0 | 3 | 93,949 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,707.5 | 69.5 | 3,305 ms | 14.4 ms | 64.0 | 0 | 141,266 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 6,675.8 | 48.0 | 6,503 ms | 20.9 ms | 128.0 | 0 | 200,186 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 418.7 | 402.5 | 376 ms | 2.5 ms | 1.0 | 3 | 12,560 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 682.6 | 321.6 | 400 ms | 3.1 ms | 2.0 | 4 | 20,488 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 957.9 | 240.2 | 418 ms | 4.2 ms | 4.0 | 6 | 28,737 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,312.1 | 164.6 | 464 ms | 6.1 ms | 8.0 | 8 | 39,357 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,906.4 | 116.5 | 1,641 ms | 8.6 ms | 16.0 | 4 | 57,192 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,941.1 | 85.9 | 2,795 ms | 11.6 ms | 32.0 | 2 | 88,172 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 4,187.6 | 59.5 | 5,546 ms | 16.8 ms | 64.0 | 0 | 125,574 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 5,687.8 | 37.5 | 10,536 ms | 26.7 ms | 128.0 | 0 | 170,711 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 356.4 | 400.8 | 642 ms | 2.5 ms | 1.0 | 3 | 10,691 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 622.4 | 317.0 | 737 ms | 3.2 ms | 2.0 | 4 | 18,670 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 854.8 | 215.8 | 744 ms | 4.6 ms | 3.9 | 4 | 25,643 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 1,170.0 | 150.7 | 899 ms | 6.6 ms | 7.9 | 8 | 35,088 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,749.7 | 105.8 | 3,166 ms | 9.5 ms | 16.0 | 2 | 52,477 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 2,543.1 | 73.6 | 6,237 ms | 13.6 ms | 32.0 | 1 | 76,300 | 30.0 s |
| Sustained decode | 256K | 64 | Valid | 3,399.2 | 48.7 | 11,654 ms | 20.5 ms | 64.0 | 0 | 101,951 | 30.0 s |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 343.9 | 346.4 | 1,287 ms | 2.9 ms | 0.9 | 2 | 10,317 | 30.0 s |
| Sustained decode | 512K | 2 | Requests queued | 524.6 | 294.7 | 1,464 ms | 3.4 ms | 1.8 | 4 | 15,739 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 731.8 | 197.8 | 1,687 ms | 5.1 ms | 3.9 | 4 | 21,950 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 1,019.2 | 128.0 | 2,372 ms | 7.8 ms | 7.7 | 8 | 30,575 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,500.6 | 87.0 | 5,902 ms | 11.5 ms | 15.9 | 3 | 45,006 | 30.0 s |
| Sustained decode | 512K | 32 | Valid | 2,042.2 | 57.1 | 11,955 ms | 17.5 ms | 32.0 | 1 | 61,241 | 30.0 s |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 288.9 | 331.5 | 2,748 ms | 3.0 ms | 0.8 | 2 | 8,666 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 448.9 | 251.8 | 3,149 ms | 4.0 ms | 1.9 | 2 | 13,468 | 30.0 s |
| Sustained decode | 1019K | 4 | Requests queued | 594.8 | 180.7 | 4,749 ms | 5.6 ms | 3.5 | 4 | 17,841 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 812.3 | 105.9 | 6,681 ms | 9.4 ms | 7.8 | 4 | 24,259 | 29.9 s |
| Sustained decode | 1019K | 16 | Valid | 1,141.8 | 67.2 | 13,180 ms | 14.9 ms | 15.9 | 2 | 34,219 | 30.0 s |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.