Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-vllm-73a5831127a9/bin/vllm" serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nemotron --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 128 --max-num-batched-tokens 32768 --async-scheduling --mamba-backend flashinfer --mamba-ssm-cache-dtype float16 --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5 --mamba-cache-mode align --attention-backend FLASHINFER --moe-backend marlin --speculative_config.method dflash --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash --speculative_config.num_speculative_tokens 3 --speculative_config.attention_backend FLASHINFER
Prefill
prompt tok/s by context length8K context31.5K260 ms TTFT
16K context33.2K489 ms TTFT
32K context32.1K1,007 ms TTFT
64K context28.5K2,256 ms TTFT
128K context22.8K5,651 ms TTFT
256K context16.4K15,660 ms TTFT
512K context10.3K49,773 ms TTFT
1019K context5,940172,117 ms TTFT
1024K context5,927173,323 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 317.5132 ms TTFT | 515.2138 ms TTFT | 799.4293 ms TTFT | Looped | 1,775.51,708 ms TTFT | Looped | Looped | 5,609.67,810 ms TTFT |
| 16K | 309.5133 ms TTFT | 533.1140 ms TTFT | 771.8280 ms TTFT | 1,183.0514 ms TTFT | 1,869.51,523 ms TTFT | 2,760.52,704 ms TTFT | 3,924.14,447 ms TTFT | 5,189.47,238 ms TTFT |
| 32K | 311.8193 ms TTFT | Looped | Looped | 1,143.3595 ms TTFT | Looped | Looped | 3,386.76,365 ms TTFT | 4,486.29,597 ms TTFT |
| 64K | 288.6232 ms TTFT | 517.7254 ms TTFT | Looped | 1,159.4386 ms TTFT | 1,637.42,308 ms TTFT | Looped | 2,669.66,760 ms TTFT | Over capacity |
| 128K | 295.6444 ms TTFT | 446.5580 ms TTFT | 675.8608 ms TTFT | 941.91,927 ms TTFT | Looped | 1,540.57,620 ms TTFT | Over capacity | Over capacity |
| 256K | 259.6682 ms TTFT | 409.3859 ms TTFT | 567.4842 ms TTFT | Looped | 976.73,218 ms TTFT | 1,010.95,881 ms TTFT | Over capacity | Over capacity |
| 512K | 216.31,593 ms TTFT | 314.41,803 ms TTFT | 440.93,474 ms TTFT | Looped | 640.68,263 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | 203.9Requests queued3,345 ms TTFT | 250.43,370 ms TTFT | Looped | 350.99,184 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.5K | — | 260 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.2K | — | 489 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 32.1K | — | 1,007 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.5K | — | 2,256 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.8K | — | 5,651 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.4K | — | 15,660 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 10.3K | — | 49,773 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 5,940.0 | — | 172,117 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 5,927.0 | — | 173,323 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 317.5 | 320.6 | 132 ms | 3.1 ms | 1.0 | 2 | 9,524 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 515.2 | 267.7 | 138 ms | 3.7 ms | 2.0 | 4 | 15,452 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 799.4 | 202.4 | 293 ms | 4.9 ms | 4.0 | 4 | 23,977 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 1,775.5 | 113.0 | 1,708 ms | 8.8 ms | 16.0 | 0 | 53,266 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Valid | 5,609.6 | 39.6 | 7,810 ms | 25.2 ms | 128.0 | 0 | 168,162 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 309.5 | 316.1 | 133 ms | 3.2 ms | 1.0 | 2 | 9,284 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 533.1 | 262.2 | 140 ms | 3.8 ms | 2.0 | 4 | 15,992 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 771.8 | 194.4 | 280 ms | 5.1 ms | 4.0 | 4 | 23,150 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,183.0 | 150.4 | 514 ms | 6.6 ms | 8.0 | 8 | 35,477 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,869.5 | 117.3 | 1,523 ms | 8.5 ms | 16.0 | 2 | 56,067 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,760.5 | 86.4 | 2,704 ms | 11.6 ms | 32.0 | 0 | 82,782 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,924.1 | 59.8 | 4,447 ms | 16.7 ms | 64.0 | 0 | 117,726 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,189.4 | 37.5 | 7,238 ms | 26.7 ms | 128.0 | 0 | 155,588 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 311.8 | 320.6 | 193 ms | 3.1 ms | 1.0 | 2 | 9,354 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 1,143.3 | 146.0 | 595 ms | 6.9 ms | 7.9 | 8 | 34,295 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Valid | 3,386.7 | 51.1 | 6,365 ms | 19.6 ms | 64.0 | 0 | 101,526 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,486.2 | 30.4 | 9,597 ms | 32.9 ms | 128.0 | 0 | 134,391 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 288.6 | 303.3 | 232 ms | 3.3 ms | 1.0 | 2 | 8,656 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 517.7 | 263.1 | 254 ms | 3.8 ms | 2.0 | 4 | 15,528 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 1,159.4 | 139.7 | 386 ms | 7.2 ms | 7.9 | 8 | 34,779 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,637.4 | 100.9 | 2,308 ms | 9.9 ms | 16.0 | 2 | 49,108 | 30.0 s |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Valid | 2,669.6 | 41.0 | 6,760 ms | 24.4 ms | 64.0 | 0 | 80,045 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 295.6 | 300.7 | 444 ms | 3.3 ms | 1.0 | 2 | 8,868 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 446.5 | 230.3 | 580 ms | 4.3 ms | 2.0 | 2 | 13,394 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 675.8 | 172.9 | 608 ms | 5.8 ms | 4.0 | 4 | 20,268 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 941.9 | 121.3 | 1,927 ms | 8.2 ms | 7.9 | 3 | 28,248 | 30.0 s |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Valid | 1,540.5 | 47.2 | 7,620 ms | 21.2 ms | 32.0 | 0 | 46,196 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 259.6 | 280.2 | 682 ms | 3.6 ms | 1.0 | 2 | 7,788 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 409.3 | 208.7 | 859 ms | 4.8 ms | 1.9 | 2 | 12,276 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 567.4 | 151.3 | 842 ms | 6.6 ms | 3.9 | 4 | 17,018 | 30.0 s |
| Sustained decode | 256K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 16 | Valid | 976.7 | 57.2 | 3,218 ms | 17.5 ms | 16.0 | 0 | 29,289 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,010.9 | 30.1 | 5,881 ms | 33.3 ms | 32.0 | 0 | 30,298 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 216.3 | 232.9 | 1,593 ms | 4.3 ms | 0.9 | 1 | 6,488 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 314.4 | 166.9 | 1,803 ms | 6.0 ms | 1.9 | 2 | 9,431 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 440.9 | 110.4 | 3,474 ms | 9.1 ms | 4.0 | 0 | 13,226 | 30.0 s |
| Sustained decode | 512K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 16 | Valid | 640.6 | 36.9 | 8,263 ms | 27.1 ms | 16.0 | 0 | 19,217 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 203.9 | 225.8 | 3,345 ms | 4.5 ms | 0.9 | 1 | 6,116 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 250.4 | 151.4 | 3,370 ms | 6.6 ms | 1.9 | 1 | 7,511 | 30.0 s |
| Sustained decode | 1019K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 8 | Valid | 350.9 | 41.3 | 9,184 ms | 24.2 ms | 8.0 | 0 | 10,525 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.