Speed test matrix
Qwen3.6-35B-A3B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-nvidia-vllm-73a5831127a9/bin/vllm" serve nvidia/Qwen3.6-35B-A3B-NVFP4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --quantization modelopt
Prefill
prompt tok/s by context length8K context35.7K230 ms TTFT
16K context34.8K467 ms TTFT
32K context32K1,010 ms TTFT
64K context25.5K2,530 ms TTFT
128K context18.1K7,117 ms TTFT
251K context11.4K22,047 ms TTFT
256K context11.3K22,808 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 236.484 ms TTFT | 355.1151 ms TTFT | 667.5156 ms TTFT | 1,075.3234 ms TTFT | 1,697.8597 ms TTFT | 2,501.6572 ms TTFT | 3,446.11,082 ms TTFT | 4,431.01,787 ms TTFT |
| 16K | 230.362 ms TTFT | 348.3117 ms TTFT | 650.1130 ms TTFT | 1,036.6224 ms TTFT | 1,608.0383 ms TTFT | 2,314.3578 ms TTFT | 3,100.9890 ms TTFT | 3,844.11,575 ms TTFT |
| 32K | 216.4120 ms TTFT | 328.0210 ms TTFT | 606.5281 ms TTFT | 956.7425 ms TTFT | 1,425.2564 ms TTFT | 1,962.2897 ms TTFT | 2,516.71,562 ms TTFT | Over capacity |
| 64K | 193.1143 ms TTFT | 297.8179 ms TTFT | 539.1219 ms TTFT | 847.0462 ms TTFT | 1,181.2789 ms TTFT | 1,536.61,422 ms TTFT | Over capacity | Over capacity |
| 128K | 162.0363 ms TTFT | 224.9480 ms TTFT | 423.1665 ms TTFT | 679.2972 ms TTFT | 879.41,661 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 123.6745 ms TTFT | 167.8978 ms TTFT | 311.71,492 ms TTFT | 489.22,065 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 35.7K | — | 230 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.8K | — | 467 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 32K | — | 1,010 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.5K | — | 2,530 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.1K | — | 7,117 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 11.4K | — | 22,047 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.3K | — | 22,808 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 236.4 | 237.3 | 84 ms | 4.2 ms | 1.0 | 1 | 7,093 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 355.1 | 178.9 | 151 ms | 5.6 ms | 2.0 | 2 | 10,652 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 667.5 | 169.0 | 156 ms | 5.9 ms | 3.9 | 4 | 20,024 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,075.3 | 137.9 | 234 ms | 7.3 ms | 8.0 | 8 | 32,256 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,697.8 | 106.9 | 597 ms | 9.4 ms | 16.0 | 0 | 50,928 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,501.6 | 78.7 | 572 ms | 12.7 ms | 32.0 | 0 | 75,040 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,446.1 | 53.9 | 1,082 ms | 18.5 ms | 64.0 | 0 | 103,415 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,431.0 | 33.9 | 1,787 ms | 29.5 ms | 128.0 | 0 | 132,864 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 230.3 | 231.1 | 62 ms | 4.3 ms | 1.0 | 1 | 6,910 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 348.3 | 175.1 | 117 ms | 5.7 ms | 2.0 | 2 | 10,448 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 650.1 | 163.6 | 130 ms | 6.1 ms | 4.0 | 4 | 19,500 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,036.6 | 131.3 | 224 ms | 7.6 ms | 8.0 | 8 | 31,096 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,608.0 | 101.2 | 383 ms | 9.9 ms | 16.0 | 0 | 48,240 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,314.3 | 72.8 | 578 ms | 13.7 ms | 32.0 | 0 | 69,428 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,100.9 | 48.3 | 890 ms | 20.7 ms | 64.0 | 0 | 92,992 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,844.1 | 29.6 | 1,575 ms | 33.8 ms | 128.0 | 0 | 115,329 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 216.4 | 217.3 | 120 ms | 4.6 ms | 1.0 | 1 | 6,491 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 328.0 | 165.4 | 210 ms | 6.0 ms | 2.0 | 2 | 9,838 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 606.5 | 153.8 | 281 ms | 6.5 ms | 4.0 | 4 | 18,196 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 956.7 | 121.0 | 425 ms | 8.3 ms | 8.0 | 8 | 28,639 | 29.9 s |
| Sustained decode | 32K | 16 | Valid | 1,425.2 | 89.3 | 564 ms | 11.2 ms | 16.0 | 0 | 42,752 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,962.2 | 61.0 | 897 ms | 16.4 ms | 32.0 | 0 | 58,848 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,516.7 | 38.5 | 1,562 ms | 26.0 ms | 64.0 | 0 | 75,456 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 193.1 | 194.1 | 143 ms | 5.2 ms | 1.0 | 1 | 5,792 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 297.8 | 150.0 | 179 ms | 6.7 ms | 2.0 | 2 | 8,934 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 539.1 | 135.8 | 219 ms | 7.4 ms | 4.0 | 4 | 16,168 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 847.0 | 105.8 | 462 ms | 9.5 ms | 8.0 | 0 | 25,409 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,181.2 | 73.4 | 789 ms | 13.6 ms | 16.0 | 0 | 35,424 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,536.6 | 47.1 | 1,422 ms | 21.2 ms | 32.0 | 0 | 46,080 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 162.0 | 164.2 | 363 ms | 6.1 ms | 1.0 | 1 | 4,861 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 224.9 | 112.3 | 480 ms | 8.9 ms | 2.0 | 0 | 6,747 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 423.1 | 105.4 | 665 ms | 9.5 ms | 4.0 | 0 | 12,692 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 679.2 | 83.8 | 972 ms | 11.9 ms | 8.0 | 0 | 20,376 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 879.4 | 53.2 | 1,661 ms | 18.8 ms | 16.0 | 0 | 26,368 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 123.6 | 131.0 | 745 ms | 7.6 ms | 1.0 | 1 | 3,707 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 167.8 | 83.8 | 978 ms | 11.9 ms | 2.0 | 0 | 5,032 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 311.7 | 77.3 | 1,492 ms | 12.9 ms | 4.0 | 0 | 9,348 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 489.2 | 59.8 | 2,065 ms | 16.7 ms | 8.0 | 0 | 14,672 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.