Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-nvidia-vllm-73a5831127a9/bin/vllm" serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nemotron --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 256 --max-num-batched-tokens 16384 --async-scheduling --mamba-backend flashinfer --mamba-ssu-algorithm horizontal --mamba-cache-mode align --mamba-ssm-cache-dtype float16 --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5 --attention-backend FLASHINFER --moe-backend marlin
Prefill
prompt tok/s by context length8K context32.1K255 ms TTFT
16K context33.8K480 ms TTFT
32K context31.9K1,013 ms TTFT
64K context28.5K2,261 ms TTFT
128K context22.9K5,607 ms TTFT
256K context16.4K15,712 ms TTFT
512K context10.4K49,279 ms TTFT
1019K context6,022169,778 ms TTFT
1024K context5,989171,553 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 322.979 ms TTFT | 546.6139 ms TTFT | 877.6235 ms TTFT | 1,313.5415 ms TTFT | 1,940.3500 ms TTFT | Looped | 4,374.01,826 ms TTFT | 6,567.43,646 ms TTFT |
| 16K | Looped | 543.2127 ms TTFT | 868.9205 ms TTFT | 1,307.6290 ms TTFT | Looped | 2,758.11,127 ms TTFT | 4,268.11,618 ms TTFT | 6,288.62,918 ms TTFT |
| 32K | 318.571 ms TTFT | 541.983 ms TTFT | 865.6109 ms TTFT | 1,298.8166 ms TTFT | Looped | 2,661.8561 ms TTFT | 4,054.91,033 ms TTFT | 5,804.52,316 ms TTFT |
| 64K | 310.4139 ms TTFT | 526.7169 ms TTFT | 837.9226 ms TTFT | 1,231.1379 ms TTFT | 1,765.4640 ms TTFT | 2,486.71,155 ms TTFT | 3,616.32,443 ms TTFT | 4,960.24,406 ms TTFT |
| 128K | 295.4264 ms TTFT | 502.7319 ms TTFT | 806.6422 ms TTFT | 1,161.5313 ms TTFT | Looped | 2,160.02,611 ms TTFT | 2,992.55,212 ms TTFT | 3,839.19,495 ms TTFT |
| 256K | 266.3670 ms TTFT | 465.5818 ms TTFT | 729.21,051 ms TTFT | 985.7754 ms TTFT | 1,418.43,121 ms TTFT | 1,735.66,164 ms TTFT | 2,224.011,186 ms TTFT | Over capacity |
| 512K | 233.01,207 ms TTFT | 403.41,410 ms TTFT | Looped | Looped | 1,109.67,158 ms TTFT | 1,217.113,921 ms TTFT | Over capacity | Over capacity |
| 1019K | 176.62,550 ms TTFT | 309.0Requests queued2,767 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32.1K | — | 255 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.8K | — | 480 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.9K | — | 1,013 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.5K | — | 2,261 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.9K | — | 5,607 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.4K | — | 15,712 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 10.4K | — | 49,279 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,022.0 | — | 169,778 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 5,989.0 | — | 171,553 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 322.9 | 324.6 | 79 ms | 3.1 ms | 1.0 | 2 | 9,687 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 546.6 | 275.6 | 139 ms | 3.6 ms | 2.0 | 4 | 16,398 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 877.6 | 220.8 | 235 ms | 4.5 ms | 4.0 | 4 | 26,327 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,313.5 | 166.7 | 415 ms | 6.0 ms | 8.0 | 8 | 39,406 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,940.3 | 122.0 | 500 ms | 8.2 ms | 16.0 | 16 | 57,674 | 29.7 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 4,374.0 | 67.0 | 1,826 ms | 14.9 ms | 64.0 | 0 | 131,204 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,567.4 | 48.3 | 3,646 ms | 20.7 ms | 128.0 | 0 | 196,995 | 30.0 s |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 543.2 | 274.1 | 127 ms | 3.6 ms | 2.0 | 4 | 16,294 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 868.9 | 218.5 | 205 ms | 4.6 ms | 4.0 | 4 | 26,064 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,307.6 | 165.3 | 290 ms | 6.1 ms | 8.0 | 8 | 39,228 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 2,758.1 | 87.1 | 1,127 ms | 11.5 ms | 32.0 | 0 | 82,722 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,268.1 | 66.2 | 1,618 ms | 15.1 ms | 64.0 | 0 | 128,043 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 6,288.6 | 46.9 | 2,918 ms | 21.3 ms | 128.0 | 0 | 188,661 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 318.5 | 320.0 | 71 ms | 3.1 ms | 1.0 | 2 | 9,554 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 541.9 | 272.5 | 83 ms | 3.7 ms | 2.0 | 4 | 16,257 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 865.6 | 217.4 | 109 ms | 4.6 ms | 4.0 | 4 | 25,965 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,298.8 | 163.3 | 166 ms | 6.1 ms | 8.0 | 8 | 38,960 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 2,661.8 | 83.3 | 561 ms | 12.0 ms | 32.0 | 0 | 79,843 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,054.9 | 63.1 | 1,033 ms | 15.8 ms | 64.0 | 0 | 121,703 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 5,804.5 | 44.3 | 2,316 ms | 22.6 ms | 128.0 | 0 | 174,087 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 310.4 | 313.8 | 139 ms | 3.2 ms | 1.0 | 2 | 9,311 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 526.7 | 266.3 | 169 ms | 3.8 ms | 2.0 | 4 | 15,801 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 837.9 | 210.8 | 226 ms | 4.7 ms | 3.9 | 4 | 25,135 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,231.1 | 155.5 | 379 ms | 6.4 ms | 8.0 | 8 | 36,930 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,765.4 | 109.9 | 640 ms | 9.1 ms | 16.0 | 0 | 52,959 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,486.7 | 76.7 | 1,155 ms | 13.0 ms | 32.0 | 0 | 74,591 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,616.3 | 54.8 | 2,443 ms | 18.2 ms | 64.0 | 0 | 108,488 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,960.2 | 36.2 | 4,406 ms | 27.6 ms | 128.0 | 0 | 148,737 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 295.4 | 300.8 | 264 ms | 3.3 ms | 1.0 | 2 | 8,862 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 502.7 | 256.7 | 319 ms | 3.9 ms | 2.0 | 4 | 15,081 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 806.6 | 203.9 | 422 ms | 4.9 ms | 3.9 | 4 | 24,194 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,161.5 | 146.5 | 313 ms | 6.8 ms | 8.0 | 8 | 34,841 | 30.0 s |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Valid | 2,160.0 | 66.1 | 2,611 ms | 15.1 ms | 32.0 | 0 | 64,803 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,992.5 | 44.8 | 5,212 ms | 22.3 ms | 64.0 | 0 | 89,729 | 30.0 s |
| Sustained decode | 128K | 128 | Valid | 3,839.1 | 27.7 | 9,495 ms | 36.1 ms | 128.0 | 0 | 115,179 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 266.3 | 278.7 | 670 ms | 3.6 ms | 0.9 | 2 | 7,989 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 465.5 | 238.8 | 818 ms | 4.2 ms | 2.0 | 2 | 13,968 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 729.2 | 187.0 | 1,051 ms | 5.3 ms | 4.0 | 4 | 21,872 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 985.7 | 127.1 | 754 ms | 7.9 ms | 7.9 | 8 | 29,569 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,418.4 | 84.6 | 3,121 ms | 11.8 ms | 16.0 | 0 | 42,544 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,735.6 | 49.5 | 6,164 ms | 20.2 ms | 32.0 | 0 | 52,063 | 30.0 s |
| Sustained decode | 256K | 64 | Valid | 2,224.0 | 29.1 | 11,186 ms | 34.3 ms | 64.0 | 0 | 66,693 | 30.0 s |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 233.0 | 243.5 | 1,207 ms | 4.1 ms | 1.0 | 1 | 6,992 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 403.4 | 211.5 | 1,410 ms | 4.7 ms | 1.9 | 2 | 12,103 | 30.0 s |
| Sustained decode | 512K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 512K | 16 | Valid | 1,109.6 | 68.1 | 7,158 ms | 14.7 ms | 16.0 | 0 | 33,280 | 30.0 s |
| Sustained decode | 512K | 32 | Valid | 1,217.1 | 37.5 | 13,921 ms | 26.7 ms | 32.0 | 0 | 36,510 | 30.0 s |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 176.6 | 195.0 | 2,550 ms | 5.1 ms | 0.9 | 1 | 5,297 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 309.0 | 170.0 | 2,767 ms | 5.9 ms | 1.8 | 2 | 9,270 | 30.0 s |
| Sustained decode | 1019K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.