Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-nvidia-vllm-73a5831127a9/bin/vllm" serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nemotron --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 128 --max-num-batched-tokens 32768 --async-scheduling --mamba-backend flashinfer --mamba-ssm-cache-dtype float16 --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5 --mamba-cache-mode align --attention-backend FLASHINFER --moe-backend marlin --speculative_config.method dspark --speculative_config.model nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark --speculative_config.num_speculative_tokens 3 --speculative_config.attention_backend FLASHINFER
Prefill
prompt tok/s by context length8K context4,6011,778 ms TTFT
16K context33K491 ms TTFT
32K context31.9K1,012 ms TTFT
64K context28.3K2,276 ms TTFT
128K context22.9K5,625 ms TTFT
256K context16.4K15,661 ms TTFT
512K context10.4K49,425 ms TTFT
1019K context6,011170,071 ms TTFT
1024K context5,993171,436 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 445.7256 ms TTFT | Looped | 1,011.5381 ms TTFT | 1,482.8721 ms TTFT | 2,156.0951 ms TTFT | 3,700.45,378 ms TTFT | Looped | Looped |
| 16K | 260.0262 ms TTFT | 633.5275 ms TTFT | Looped | 1,184.71,751 ms TTFT | Looped | Looped | Looped | 6,904.015,252 ms TTFT |
| 32K | 423.1272 ms TTFT | Looped | 988.6286 ms TTFT | 1,403.2512 ms TTFT | 2,039.0719 ms TTFT | 3,430.35,085 ms TTFT | 4,642.78,356 ms TTFT | 6,426.014,875 ms TTFT |
| 64K | 416.4245 ms TTFT | 625.0257 ms TTFT | 990.9257 ms TTFT | 1,453.2356 ms TTFT | 1,872.71,258 ms TTFT | Looped | 4,099.05,977 ms TTFT | Over capacity |
| 128K | 394.7565 ms TTFT | 653.3584 ms TTFT | 914.0794 ms TTFT | 1,229.7660 ms TTFT | 1,737.74,593 ms TTFT | 2,581.69,109 ms TTFT | Over capacity | Over capacity |
| 256K | 396.1729 ms TTFT | Looped | 821.9810 ms TTFT | 1,116.3884 ms TTFT | 1,304.33,456 ms TTFT | Over capacity | Over capacity | Over capacity |
| 512K | 332.61,936 ms TTFT | 489.5Requests queued2,643 ms TTFT | 697.72,265 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | Over capacity | Over capacity | Looped | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,601.0 | — | 1,778 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33K | — | 491 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.9K | — | 1,012 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.3K | — | 2,276 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.9K | — | 5,625 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.4K | — | 15,661 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 10.4K | — | 49,425 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,011.0 | — | 170,071 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 5,993.0 | — | 171,436 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 445.7 | 457.2 | 256 ms | 2.2 ms | 1.0 | 3 | 13,370 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Valid | 1,011.5 | 257.2 | 381 ms | 3.9 ms | 4.0 | 8 | 30,340 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,482.8 | 185.3 | 721 ms | 5.4 ms | 8.0 | 8 | 44,479 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,156.0 | 132.5 | 951 ms | 7.5 ms | 15.9 | 16 | 64,667 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,700.4 | 112.6 | 5,378 ms | 8.9 ms | 31.9 | 7 | 110,545 | 29.9 s |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 260.0 | 223.5 | 262 ms | 4.5 ms | 1.0 | 2 | 7,798 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 633.5 | 341.6 | 275 ms | 2.9 ms | 2.0 | 4 | 19,003 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 1,184.7 | 161.5 | 1,751 ms | 6.2 ms | 8.0 | 7 | 35,530 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Valid | 6,904.0 | 41.6 | 15,252 ms | 24.1 ms | 128.0 | 0 | 207,043 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 423.1 | 423.2 | 272 ms | 2.4 ms | 1.0 | 3 | 12,692 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Valid | 988.6 | 254.6 | 286 ms | 3.9 ms | 3.9 | 8 | 29,657 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,403.2 | 177.5 | 512 ms | 5.6 ms | 8.0 | 8 | 42,098 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,039.0 | 131.0 | 719 ms | 7.6 ms | 15.9 | 16 | 61,173 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,430.3 | 105.4 | 5,085 ms | 9.5 ms | 32.0 | 1 | 102,854 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,642.7 | 66.5 | 8,356 ms | 15.0 ms | 64.0 | 0 | 139,258 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,426.0 | 40.1 | 14,875 ms | 24.9 ms | 128.0 | 0 | 192,580 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 416.4 | 435.3 | 245 ms | 2.3 ms | 1.0 | 3 | 12,490 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 625.0 | 335.9 | 257 ms | 3.0 ms | 1.9 | 4 | 18,749 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 990.9 | 254.1 | 257 ms | 3.9 ms | 4.0 | 8 | 29,721 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,453.2 | 184.3 | 356 ms | 5.4 ms | 8.0 | 8 | 43,589 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,872.7 | 121.2 | 1,258 ms | 8.2 ms | 15.9 | 10 | 56,183 | 30.0 s |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Valid | 4,099.0 | 62.4 | 5,977 ms | 16.0 ms | 64.0 | 0 | 122,881 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 394.7 | 421.0 | 565 ms | 2.4 ms | 0.9 | 3 | 11,839 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 653.3 | 332.8 | 584 ms | 3.0 ms | 2.0 | 4 | 19,598 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 914.0 | 235.5 | 794 ms | 4.2 ms | 4.0 | 4 | 27,435 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,229.7 | 162.3 | 660 ms | 6.2 ms | 7.9 | 8 | 36,886 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,737.7 | 111.2 | 4,593 ms | 9.0 ms | 15.9 | 3 | 51,904 | 29.9 s |
| Sustained decode | 128K | 32 | Valid | 2,581.6 | 76.5 | 9,109 ms | 13.1 ms | 32.0 | 0 | 77,420 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 396.1 | 410.2 | 729 ms | 2.4 ms | 1.0 | 3 | 11,881 | 30.0 s |
| Sustained decode | 256K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 4 | Valid | 821.9 | 211.2 | 810 ms | 4.7 ms | 3.9 | 4 | 24,653 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 1,116.3 | 134.9 | 884 ms | 7.4 ms | 7.8 | 8 | 33,478 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,304.3 | 83.9 | 3,456 ms | 11.9 ms | 16.0 | 0 | 39,117 | 30.0 s |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 332.6 | 376.0 | 1,936 ms | 2.7 ms | 0.9 | 2 | 9,978 | 30.0 s |
| Sustained decode | 512K | 2 | Requests queued | 489.5 | 292.0 | 2,643 ms | 3.4 ms | 1.8 | 4 | 14,685 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 697.7 | 187.1 | 2,265 ms | 5.4 ms | 3.9 | 4 | 20,923 | 30.0 s |
| Sustained decode | 512K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 1019K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.