Speed test matrix
Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-nemotron/bin/vllm" serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --video-pruning-rate 0.5 --max-num-seqs 384 --allowed-local-media-path / --media-io-kwargs '{"video":{"fps":2,"num_frames":256}}'Prefill
prompt tok/s by context length8K context45.3K181 ms TTFT
16K context42.7K380 ms TTFT
32K context44.3K728 ms TTFT
64K context42.5K1,516 ms TTFT
128K context34.2K3,760 ms TTFT
251K context24.4K10,346 ms TTFT
256K context24K10,700 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 309.779 ms TTFT | 591.4121 ms TTFT | 647.2144 ms TTFT | 1,156.2204 ms TTFT | 2,139.5303 ms TTFT | 1,285.5564 ms TTFT | 1,419.6834 ms TTFT | 2,257.01,312 ms TTFT |
| 16K | 308.188 ms TTFT | 529.6104 ms TTFT | 663.2128 ms TTFT | 1,357.9141 ms TTFT | 2,101.8286 ms TTFT | 1,011.2505 ms TTFT | 1,932.1683 ms TTFT | 2,405.91,310 ms TTFT |
| 32K | 304.6104 ms TTFT | 543.8131 ms TTFT | 613.3197 ms TTFT | 1,240.3254 ms TTFT | 2,142.5241 ms TTFT | 1,210.4643 ms TTFT | 1,714.01,156 ms TTFT | 2,467.21,822 ms TTFT |
| 64K | 299.3139 ms TTFT | 496.5182 ms TTFT | 786.0189 ms TTFT | 1,121.7246 ms TTFT | 1,243.2770 ms TTFT | 1,486.71,401 ms TTFT | 2,213.62,423 ms TTFT | 3,462.03,930 ms TTFT |
| 128K | 285.7273 ms TTFT | 365.3335 ms TTFT | 695.5339 ms TTFT | 1,006.1483 ms TTFT | 1,442.81,483 ms TTFT | 1,971.02,714 ms TTFT | 2,536.85,303 ms TTFT | Over capacity |
| 251K | 264.0484 ms TTFT | 447.0586 ms TTFT | 606.0588 ms TTFT | 949.31,294 ms TTFT | 1,277.12,705 ms TTFT | 1,599.35,009 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 45.3K | — | 181 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 42.7K | — | 380 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.3K | — | 728 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 42.5K | — | 1,516 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 34.2K | — | 3,760 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 24.4K | — | 10,346 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 24K | — | 10,700 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 309.7 | 1006.7 | 79 ms | 1.0 ms | 1.0 | 2 | 7,551 | 24.4 s |
| Sustained decode | 8K | 2 | Valid | 591.4 | 2789.1 | 121 ms | 0.4 ms | 2.0 | 4 | 17,750 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 647.2 | 193.5 | 144 ms | 5.2 ms | 4.0 | 4 | 19,415 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,156.2 | 190.2 | 204 ms | 5.3 ms | 8.0 | 8 | 34,685 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,139.5 | 129.4 | 303 ms | 7.7 ms | 16.0 | 16 | 64,187 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,285.5 | 91.7 | 564 ms | 10.9 ms | 32.0 | 0 | 38,558 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,419.6 | 68.3 | 834 ms | 14.6 ms | 64.0 | 0 | 42,585 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,257.0 | 50.0 | 1,312 ms | 20.0 ms | 128.0 | 0 | 67,701 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 308.1 | 316.1 | 88 ms | 3.2 ms | 1.0 | 2 | 7,338 | 23.8 s |
| Sustained decode | 16K | 2 | Valid | 529.6 | 960.5 | 104 ms | 1.1 ms | 2.0 | 4 | 15,886 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 663.2 | 193.7 | 128 ms | 5.2 ms | 4.0 | 4 | 19,897 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,357.9 | 175.4 | 141 ms | 5.8 ms | 8.0 | 8 | 40,737 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,101.8 | 123.2 | 286 ms | 8.1 ms | 15.7 | 16 | 63,043 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,011.2 | 87.8 | 505 ms | 11.4 ms | 32.0 | 0 | 30,334 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,932.1 | 64.0 | 683 ms | 15.6 ms | 64.0 | 0 | 57,960 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,405.9 | 50.1 | 1,310 ms | 19.9 ms | 128.0 | 0 | 72,153 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 304.6 | 307.2 | 104 ms | 3.3 ms | 1.0 | 2 | 7,483 | 24.6 s |
| Sustained decode | 32K | 2 | Valid | 543.8 | 541.9 | 131 ms | 2.5 ms | 1.9 | 4 | 15,506 | 28.5 s |
| Sustained decode | 32K | 4 | Valid | 613.3 | 371.5 | 197 ms | 3.5 ms | 4.0 | 4 | 18,394 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,240.3 | 159.8 | 254 ms | 6.3 ms | 8.0 | 8 | 37,205 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,142.5 | 122.7 | 241 ms | 8.2 ms | 15.9 | 16 | 64,279 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,210.4 | 81.7 | 643 ms | 12.2 ms | 32.0 | 0 | 36,306 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,714.0 | 59.6 | 1,156 ms | 16.8 ms | 64.0 | 0 | 51,405 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,467.2 | 43.6 | 1,822 ms | 22.9 ms | 128.0 | 0 | 74,013 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 299.3 | 302.1 | 139 ms | 3.3 ms | 1.0 | 2 | 8,978 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 496.5 | 260.5 | 182 ms | 3.8 ms | 2.0 | 4 | 14,896 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 786.0 | 185.9 | 189 ms | 5.4 ms | 4.0 | 4 | 23,579 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,121.7 | 147.2 | 246 ms | 6.8 ms | 8.0 | 8 | 33,647 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,243.2 | 109.5 | 770 ms | 9.1 ms | 16.0 | 0 | 37,294 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,486.7 | 77.5 | 1,401 ms | 12.9 ms | 32.0 | 0 | 44,592 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,213.6 | 54.2 | 2,423 ms | 18.4 ms | 64.0 | 0 | 66,395 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,462.0 | 35.8 | 3,930 ms | 28.0 ms | 128.0 | 0 | 103,802 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 285.7 | 310.2 | 273 ms | 3.2 ms | 1.0 | 2 | 7,501 | 26.3 s |
| Sustained decode | 128K | 2 | Valid | 365.3 | 242.4 | 335 ms | 4.1 ms | 2.0 | 2 | 10,959 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 695.5 | 175.5 | 339 ms | 5.7 ms | 3.9 | 4 | 20,865 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,006.1 | 127.8 | 483 ms | 7.8 ms | 7.9 | 8 | 30,178 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,442.8 | 89.9 | 1,483 ms | 11.1 ms | 16.0 | 0 | 43,278 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,971.0 | 62.6 | 2,714 ms | 16.0 ms | 32.0 | 0 | 59,115 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,536.8 | 41.7 | 5,303 ms | 24.0 ms | 64.0 | 0 | 76,103 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 264.0 | 272.8 | 484 ms | 3.7 ms | 1.0 | 2 | 7,921 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 447.0 | 228.6 | 586 ms | 4.4 ms | 1.9 | 2 | 13,409 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 606.0 | 163.0 | 588 ms | 6.1 ms | 3.9 | 4 | 18,178 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 949.3 | 119.5 | 1,294 ms | 8.4 ms | 8.0 | 3 | 28,477 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,277.1 | 79.9 | 2,705 ms | 12.5 ms | 16.0 | 0 | 38,301 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,599.3 | 51.7 | 5,009 ms | 19.4 ms | 32.0 | 0 | 47,986 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.