Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF168 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 "$HOME/env_h200-nvidia-nemotron/bin/vllm" serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 128 --async-scheduling --mamba-backend flashinfer --mamba-cache-mode align --mamba-ssm-cache-dtype float16 --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5
Prefill
prompt tok/s by context length8K context43.1K190 ms TTFT
16K context43.5K373 ms TTFT
32K context44.7K722 ms TTFT
64K context42.1K1,529 ms TTFT
128K context34.2K3,759 ms TTFT
256K context24.2K10,636 ms TTFT
512K context14.6K35,156 ms TTFT
1019K context8,164125,244 ms TTFT
1024K context8,109126,723 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 292.175 ms TTFT | 482.6116 ms TTFT | 722.9161 ms TTFT | 1,112.7222 ms TTFT | 1,653.9363 ms TTFT | 2,450.1534 ms TTFT | 3,799.7787 ms TTFT | 5,929.61,184 ms TTFT |
| 16K | 291.578 ms TTFT | 482.2119 ms TTFT | 718.5184 ms TTFT | 1,096.7204 ms TTFT | 1,613.6353 ms TTFT | 2,411.2405 ms TTFT | 3,733.8730 ms TTFT | 5,691.21,195 ms TTFT |
| 32K | 288.4101 ms TTFT | 476.5111 ms TTFT | 710.3221 ms TTFT | 1,085.6203 ms TTFT | 1,577.3386 ms TTFT | 2,330.8638 ms TTFT | 3,533.91,094 ms TTFT | 5,236.92,020 ms TTFT |
| 64K | 282.1139 ms TTFT | 468.6191 ms TTFT | 694.4196 ms TTFT | 1,034.0206 ms TTFT | 1,509.3707 ms TTFT | 2,180.01,280 ms TTFT | 3,182.52,408 ms TTFT | 4,492.94,689 ms TTFT |
| 128K | 270.8302 ms TTFT | 450.8383 ms TTFT | 659.1393 ms TTFT | 971.8766 ms TTFT | 1,389.81,489 ms TTFT | 1,949.42,765 ms TTFT | 2,655.75,452 ms TTFT | Over capacity |
| 256K | 249.8525 ms TTFT | 420.6676 ms TTFT | 602.5612 ms TTFT | 886.21,682 ms TTFT | 1,192.62,920 ms TTFT | 1,577.65,361 ms TTFT | Over capacity | Over capacity |
| 512K | 221.31,044 ms TTFT | 366.01,206 ms TTFT | 512.21,166 ms TTFT | 736.23,624 ms TTFT | 915.46,453 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | 172.22,429 ms TTFT | 286.7Requests queued2,446 ms TTFT | 418.34,632 ms TTFT | 534.47,792 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 43.1K | — | 190 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 43.5K | — | 373 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.7K | — | 722 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 42.1K | — | 1,529 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 34.2K | — | 3,759 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 24.2K | — | 10,636 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 14.6K | — | 35,156 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,164.0 | — | 125,244 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 8,109.0 | — | 126,723 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 292.1 | 293.9 | 75 ms | 3.4 ms | 1.0 | 2 | 8,764 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 482.6 | 243.4 | 116 ms | 4.1 ms | 2.0 | 4 | 14,478 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 722.9 | 182.0 | 161 ms | 5.5 ms | 4.0 | 4 | 21,689 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,112.7 | 143.2 | 222 ms | 7.0 ms | 8.0 | 8 | 33,382 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,653.9 | 104.8 | 363 ms | 9.5 ms | 16.0 | 0 | 49,617 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,450.1 | 78.1 | 534 ms | 12.8 ms | 32.0 | 0 | 73,504 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,799.7 | 60.5 | 787 ms | 16.5 ms | 64.0 | 0 | 113,983 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,929.6 | 46.6 | 1,184 ms | 21.5 ms | 128.0 | 0 | 177,922 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 291.5 | 293.0 | 78 ms | 3.4 ms | 1.0 | 2 | 8,746 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 482.2 | 243.1 | 119 ms | 4.1 ms | 2.0 | 4 | 14,467 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 718.5 | 181.0 | 184 ms | 5.5 ms | 3.9 | 4 | 21,556 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,096.7 | 141.2 | 204 ms | 7.1 ms | 7.8 | 8 | 32,899 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,613.6 | 102.5 | 353 ms | 9.8 ms | 16.0 | 0 | 48,402 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,411.2 | 76.5 | 405 ms | 13.1 ms | 32.0 | 0 | 72,323 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,733.8 | 58.8 | 730 ms | 17.0 ms | 64.0 | 0 | 111,997 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,691.2 | 44.6 | 1,195 ms | 22.4 ms | 128.0 | 0 | 170,757 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 288.4 | 290.4 | 101 ms | 3.4 ms | 1.0 | 2 | 8,651 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 476.5 | 240.4 | 111 ms | 4.2 ms | 2.0 | 4 | 14,294 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 710.3 | 179.2 | 221 ms | 5.6 ms | 4.0 | 4 | 21,309 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,085.6 | 137.5 | 203 ms | 7.3 ms | 8.0 | 8 | 32,566 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,577.3 | 99.1 | 386 ms | 10.1 ms | 16.0 | 0 | 47,311 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,330.8 | 73.1 | 638 ms | 13.7 ms | 32.0 | 0 | 69,926 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,533.9 | 55.2 | 1,094 ms | 18.1 ms | 64.0 | 0 | 105,985 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 5,236.9 | 40.8 | 2,020 ms | 24.5 ms | 128.0 | 0 | 157,058 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 282.1 | 285.0 | 139 ms | 3.5 ms | 1.0 | 2 | 8,464 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 468.6 | 236.0 | 191 ms | 4.2 ms | 2.0 | 2 | 14,057 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 694.4 | 174.7 | 196 ms | 5.7 ms | 4.0 | 4 | 20,830 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,034.0 | 130.6 | 206 ms | 7.7 ms | 7.9 | 8 | 31,018 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,509.3 | 94.6 | 707 ms | 10.6 ms | 16.0 | 0 | 45,279 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,180.0 | 68.3 | 1,280 ms | 14.6 ms | 32.0 | 0 | 65,380 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,182.5 | 49.8 | 2,408 ms | 20.1 ms | 64.0 | 0 | 95,488 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,492.9 | 35.2 | 4,689 ms | 28.4 ms | 128.0 | 0 | 134,777 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 270.8 | 276.3 | 302 ms | 3.6 ms | 1.0 | 2 | 8,124 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 450.8 | 228.6 | 383 ms | 4.4 ms | 2.0 | 2 | 13,522 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 659.1 | 166.9 | 393 ms | 6.0 ms | 4.0 | 4 | 19,770 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 971.8 | 121.9 | 766 ms | 8.2 ms | 8.0 | 4 | 29,068 | 29.9 s |
| Sustained decode | 128K | 16 | Valid | 1,389.8 | 86.5 | 1,489 ms | 11.6 ms | 16.0 | 0 | 41,697 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,949.4 | 60.0 | 2,765 ms | 16.7 ms | 32.0 | 0 | 58,463 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,655.7 | 40.5 | 5,452 ms | 24.7 ms | 64.0 | 0 | 79,686 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 249.8 | 259.1 | 525 ms | 3.9 ms | 1.0 | 2 | 7,494 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 420.6 | 215.6 | 676 ms | 4.6 ms | 1.9 | 2 | 12,618 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 602.5 | 153.9 | 612 ms | 6.5 ms | 4.0 | 4 | 18,073 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 886.2 | 110.8 | 1,682 ms | 9.0 ms | 8.0 | 0 | 26,585 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,192.6 | 74.3 | 2,920 ms | 13.5 ms | 16.0 | 0 | 35,778 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,577.6 | 48.8 | 5,361 ms | 20.5 ms | 32.0 | 0 | 47,331 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 221.3 | 230.1 | 1,044 ms | 4.3 ms | 1.0 | 1 | 6,640 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 366.0 | 191.7 | 1,206 ms | 5.2 ms | 1.9 | 2 | 10,980 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 512.2 | 134.1 | 1,166 ms | 7.5 ms | 3.9 | 4 | 15,373 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 736.2 | 92.4 | 3,624 ms | 10.8 ms | 8.0 | 0 | 22,080 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 915.4 | 57.7 | 6,453 ms | 17.3 ms | 16.0 | 0 | 27,456 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 172.2 | 189.6 | 2,429 ms | 5.3 ms | 0.9 | 1 | 5,166 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 286.7 | 159.6 | 2,446 ms | 6.3 ms | 1.9 | 2 | 8,602 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 418.3 | 105.8 | 4,632 ms | 9.4 ms | 4.0 | 0 | 12,548 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 534.4 | 67.5 | 7,792 ms | 14.8 ms | 8.0 | 0 | 16,033 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.