Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF168 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- nemotron
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 "$HOME/env_rtxpro6k-nvidia-vllm-73a5831127a9/bin/vllm" serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 --served-model-name nemotron --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 1048576 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 128 --async-scheduling --mamba-backend flashinfer --mamba-cache-mode align --mamba-ssm-cache-dtype float16 --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5 --attention-backend FLASH_ATTN --moe-backend flashinfer_cutlass
Prefill
prompt tok/s by context length8K context31.7K258 ms TTFT
16K context32.4K501 ms TTFT
32K context32.4K996 ms TTFT
64K context28.8K2,236 ms TTFT
128K context22.4K5,742 ms TTFT
256K context15.2K16,873 ms TTFT
512K context9,27555,392 ms TTFT
1019K context5,216195,996 ms TTFT
1024K context5,194197,804 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 178.074 ms TTFT | 267.6217 ms TTFT | 386.4157 ms TTFT | 525.4194 ms TTFT | 730.4327 ms TTFT | 1,090.5540 ms TTFT | 1,761.8791 ms TTFT | 2,924.11,516 ms TTFT |
| 16K | 177.669 ms TTFT | 266.993 ms TTFT | 385.1138 ms TTFT | 529.1196 ms TTFT | 723.7291 ms TTFT | 1,081.0418 ms TTFT | 1,722.2704 ms TTFT | Over capacity |
| 32K | 175.291 ms TTFT | 263.4109 ms TTFT | 383.2163 ms TTFT | 520.6267 ms TTFT | 710.6398 ms TTFT | 1,047.8646 ms TTFT | Over capacity | Over capacity |
| 64K | 170.0126 ms TTFT | 256.2150 ms TTFT | Looped | 508.0399 ms TTFT | 694.3637 ms TTFT | 1,009.31,268 ms TTFT | Over capacity | Over capacity |
| 128K | 160.0286 ms TTFT | 246.9375 ms TTFT | 355.8541 ms TTFT | 482.1823 ms TTFT | 647.31,449 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 145.5518 ms TTFT | 213.3715 ms TTFT | 320.61,040 ms TTFT | 439.81,635 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 122.9982 ms TTFT | 177.91,550 ms TTFT | 276.12,205 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 91.0Requests queued2,605 ms TTFT | 121.3Requests queued3,379 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.7K | — | 258 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.4K | — | 501 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 32.4K | — | 996 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.8K | — | 2,236 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.4K | — | 5,742 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 15.2K | — | 16,873 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,275.0 | — | 55,392 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 5,216.0 | — | 195,996 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 5,194.0 | — | 197,804 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 178.0 | 178.5 | 74 ms | 5.6 ms | 1.0 | 1 | 5,341 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 267.6 | 134.4 | 217 ms | 7.4 ms | 2.0 | 2 | 8,026 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 386.4 | 97.4 | 157 ms | 10.3 ms | 4.0 | 0 | 11,592 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 525.4 | 66.6 | 194 ms | 15.0 ms | 8.0 | 0 | 15,761 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 730.4 | 47.1 | 327 ms | 21.2 ms | 16.0 | 0 | 21,904 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,090.5 | 35.5 | 540 ms | 28.2 ms | 32.0 | 0 | 32,702 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,761.8 | 28.6 | 791 ms | 34.9 ms | 64.0 | 0 | 52,802 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,924.1 | 23.4 | 1,516 ms | 42.8 ms | 128.0 | 0 | 87,682 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 177.6 | 178.1 | 69 ms | 5.6 ms | 1.0 | 1 | 5,328 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 266.9 | 134.4 | 93 ms | 7.4 ms | 2.0 | 2 | 8,008 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 385.1 | 96.9 | 138 ms | 10.3 ms | 4.0 | 0 | 11,552 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 529.1 | 66.7 | 196 ms | 15.0 ms | 8.0 | 0 | 15,872 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 723.7 | 46.3 | 291 ms | 21.6 ms | 16.0 | 0 | 21,710 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,081.0 | 34.9 | 418 ms | 28.7 ms | 32.0 | 0 | 32,416 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,722.2 | 27.7 | 704 ms | 36.2 ms | 64.0 | 0 | 51,654 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 175.2 | 175.8 | 91 ms | 5.7 ms | 1.0 | 1 | 5,255 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 263.4 | 132.8 | 109 ms | 7.5 ms | 2.0 | 2 | 7,906 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 383.2 | 96.0 | 163 ms | 10.4 ms | 4.0 | 0 | 11,496 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 520.6 | 65.5 | 267 ms | 15.3 ms | 8.0 | 0 | 15,616 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 710.6 | 45.2 | 398 ms | 22.1 ms | 16.0 | 0 | 21,315 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,047.8 | 33.3 | 646 ms | 30.0 ms | 32.0 | 0 | 31,430 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 170.0 | 170.7 | 126 ms | 5.9 ms | 1.0 | 1 | 5,098 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 256.2 | 129.8 | 150 ms | 7.7 ms | 2.0 | 2 | 7,684 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 508.0 | 63.6 | 399 ms | 15.7 ms | 8.0 | 0 | 15,239 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 694.3 | 43.6 | 637 ms | 22.9 ms | 16.0 | 0 | 20,815 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,009.3 | 31.3 | 1,268 ms | 31.9 ms | 32.0 | 0 | 30,298 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 160.0 | 161.8 | 286 ms | 6.2 ms | 1.0 | 1 | 4,800 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 246.9 | 123.5 | 375 ms | 8.1 ms | 2.0 | 1 | 7,408 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 355.8 | 88.8 | 541 ms | 11.3 ms | 4.0 | 0 | 10,675 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 482.1 | 59.9 | 823 ms | 16.7 ms | 8.0 | 0 | 14,457 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 647.3 | 39.9 | 1,449 ms | 25.1 ms | 16.0 | 0 | 19,407 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 145.5 | 148.8 | 518 ms | 6.7 ms | 1.0 | 1 | 4,366 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 213.3 | 106.8 | 715 ms | 9.4 ms | 2.0 | 0 | 6,398 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 320.6 | 80.0 | 1,040 ms | 12.5 ms | 4.0 | 0 | 9,616 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 439.8 | 54.5 | 1,635 ms | 18.4 ms | 8.0 | 0 | 13,190 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 122.9 | 146.3 | 982 ms | 7.0 ms | 1.0 | 1 | 3,688 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 177.9 | 89.4 | 1,550 ms | 11.2 ms | 2.0 | 0 | 5,338 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 276.1 | 69.1 | 2,205 ms | 14.5 ms | 4.0 | 0 | 8,280 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 91.0 | 97.8 | 2,605 ms | 10.2 ms | 0.9 | 0 | 2,729 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 121.3 | 67.1 | 3,379 ms | 14.9 ms | 1.8 | 0 | 3,640 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.