Speed test matrix
Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-vllm-73a5831127a9/bin/vllm" serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --video-pruning-rate 0.5 --max-num-seqs 384 --allowed-local-media-path / --media-io-kwargs '{"video":{"fps":2,"num_frames":256}}' --kv-cache-dtype fp8 --attention-backend FLASHINFER --moe-backend cutlassPrefill
prompt tok/s by context length8K context44K186 ms TTFT
16K context43.9K370 ms TTFT
32K context41.1K785 ms TTFT
64K context35K1,840 ms TTFT
128K context26.3K4,880 ms TTFT
251K context17.7K14,197 ms TTFT
256K context17.5K14,694 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 269.577 ms TTFT | 531.7146 ms TTFT | 743.8194 ms TTFT | 1,144.3255 ms TTFT | 920.9418 ms TTFT | 1,245.9741 ms TTFT | 2,040.11,377 ms TTFT | 2,958.22,598 ms TTFT |
| 16K | 268.493 ms TTFT | 450.4159 ms TTFT | 729.3189 ms TTFT | 1,116.1292 ms TTFT | 1,999.8481 ms TTFT | 1,139.3703 ms TTFT | 1,832.41,192 ms TTFT | 2,936.32,219 ms TTFT |
| 32K | 246.676 ms TTFT | 453.991 ms TTFT | Looped | 1,064.0189 ms TTFT | 960.3320 ms TTFT | Looped | Looped | 3,802.82,114 ms TTFT |
| 64K | Looped | 431.3234 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped |
| 128K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Over capacity |
| 251K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 44K | — | 186 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 43.9K | — | 370 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 41.1K | — | 785 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 35K | — | 1,840 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 26.3K | — | 4,880 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.7K | — | 14,197 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.5K | — | 14,694 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 269.5 | 252.3 | 77 ms | 4.0 ms | 1.0 | 2 | 8,084 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 531.7 | 1189.9 | 146 ms | 2.6 ms | 2.0 | 2 | 8,202 | 15.4 s |
| Sustained decode | 8K | 4 | Valid | 743.8 | 890.0 | 194 ms | 3.1 ms | 4.0 | 4 | 22,312 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,144.3 | 155.5 | 255 ms | 6.4 ms | 8.0 | 8 | 34,327 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 920.9 | 115.0 | 418 ms | 8.7 ms | 16.0 | 0 | 27,627 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,245.9 | 86.4 | 741 ms | 11.6 ms | 32.0 | 0 | 37,391 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,040.1 | 65.1 | 1,377 ms | 15.4 ms | 64.0 | 0 | 61,189 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,958.2 | 46.7 | 2,598 ms | 21.4 ms | 128.0 | 0 | 88,729 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 268.4 | 253.1 | 93 ms | 4.0 ms | 1.0 | 2 | 8,053 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 450.4 | 213.7 | 159 ms | 4.7 ms | 2.0 | 2 | 7,853 | 17.4 s |
| Sustained decode | 16K | 4 | Valid | 729.3 | 183.7 | 189 ms | 5.4 ms | 4.0 | 4 | 16,093 | 22.1 s |
| Sustained decode | 16K | 8 | Valid | 1,116.1 | 158.9 | 292 ms | 6.3 ms | 7.7 | 8 | 33,478 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,999.8 | 125.7 | 481 ms | 8.0 ms | 16.0 | 16 | 59,994 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,139.3 | 86.9 | 703 ms | 11.5 ms | 32.0 | 0 | 34,171 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,832.4 | 63.7 | 1,192 ms | 15.7 ms | 64.0 | 0 | 54,973 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,936.3 | 45.8 | 2,219 ms | 21.8 ms | 128.0 | 0 | 88,056 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 246.6 | 247.9 | 76 ms | 4.0 ms | 1.0 | 2 | 7,397 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 453.9 | 456.2 | 91 ms | 3.1 ms | 2.0 | 2 | 13,614 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 1,064.0 | 139.6 | 189 ms | 7.2 ms | 7.9 | 8 | 31,916 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 960.3 | 112.5 | 320 ms | 8.9 ms | 16.0 | 0 | 28,804 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Valid | 3,802.8 | 42.9 | 2,114 ms | 23.3 ms | 128.0 | 0 | 114,022 | 30.0 s |
| Sustained decode | 64K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 2 | Valid | 431.3 | 210.6 | 234 ms | 4.7 ms | 2.0 | 2 | 7,667 | 17.8 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.