Speed test matrix
Qwen3.6-27B-int4-AutoRound on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT4+BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4+BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-intel-vllm-73a5831127a9/bin/vllm" serve Intel/Qwen3.6-27B-int4-AutoRound --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context6,1041,342 ms TTFT
16K context5,9592,723 ms TTFT
32K context5,6175,749 ms TTFT
64K context5,02512,820 ms TTFT
128K context4,13331,138 ms TTFT
251K context3,08181,856 ms TTFT
256K context3,06084,048 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 94.5190 ms TTFT | 192.2760 ms TTFT | 362.31,536 ms TTFT | 543.02,722 ms TTFT | 851.22,398 ms TTFT | 1,076.43,899 ms TTFT | 1,305.06,674 ms TTFT | Over capacity |
| 16K | 76.7222 ms TTFT | 158.2382 ms TTFT | 304.1764 ms TTFT | 427.21,451 ms TTFT | 645.62,919 ms TTFT | 795.35,203 ms TTFT | Over capacity | Over capacity |
| 32K | 57.1285 ms TTFT | 118.6496 ms TTFT | 229.3963 ms TTFT | 290.71,662 ms TTFT | 426.03,353 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 38.6410 ms TTFT | 80.0679 ms TTFT | 155.51,264 ms TTFT | 179.22,577 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 24.4757 ms TTFT | 46.81,221 ms TTFT | 92.42,161 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 13.11,084 ms TTFT | 25.41,507 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,104.0 | — | 1,342 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,959.0 | — | 2,723 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,617.0 | — | 5,749 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,025.0 | — | 12,820 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,133.0 | — | 31,138 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,081.0 | — | 81,856 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,060.0 | — | 84,048 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 94.5 | 95.1 | 190 ms | 10.5 ms | 1.0 | 0 | 2,833 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 192.2 | 96.0 | 760 ms | 10.4 ms | 2.0 | 0 | 5,765 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 362.3 | 90.3 | 1,536 ms | 11.1 ms | 4.0 | 0 | 10,861 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 543.0 | 68.8 | 2,722 ms | 14.5 ms | 8.0 | 0 | 16,284 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 851.2 | 53.2 | 2,398 ms | 18.8 ms | 16.0 | 0 | 25,516 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,076.4 | 33.2 | 3,899 ms | 30.1 ms | 32.0 | 0 | 32,268 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,305.0 | 19.1 | 6,674 ms | 52.3 ms | 64.0 | 0 | 39,146 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 76.7 | 77.7 | 222 ms | 12.9 ms | 1.0 | 0 | 2,302 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 158.2 | 80.1 | 382 ms | 12.5 ms | 2.0 | 0 | 4,743 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 304.1 | 77.0 | 764 ms | 13.0 ms | 4.0 | 0 | 9,119 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 427.2 | 53.8 | 1,451 ms | 18.6 ms | 8.0 | 0 | 12,816 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 645.6 | 40.8 | 2,919 ms | 24.5 ms | 16.0 | 0 | 19,354 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 795.3 | 24.7 | 5,203 ms | 40.5 ms | 32.0 | 0 | 23,793 | 29.9 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 57.1 | 57.8 | 285 ms | 17.3 ms | 1.0 | 0 | 1,711 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 118.6 | 60.1 | 496 ms | 16.6 ms | 2.0 | 0 | 3,554 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 229.3 | 58.1 | 963 ms | 17.2 ms | 4.0 | 0 | 6,876 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 290.7 | 36.5 | 1,662 ms | 27.4 ms | 8.0 | 0 | 8,717 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 426.0 | 27.0 | 3,353 ms | 37.1 ms | 16.0 | 0 | 12,774 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 38.6 | 39.2 | 410 ms | 25.5 ms | 1.0 | 0 | 1,156 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 80.0 | 40.4 | 679 ms | 24.8 ms | 2.0 | 0 | 2,400 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 155.5 | 39.4 | 1,264 ms | 25.4 ms | 4.0 | 0 | 4,664 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 179.2 | 22.8 | 2,577 ms | 43.8 ms | 8.0 | 0 | 5,366 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 24.4 | 24.8 | 757 ms | 40.3 ms | 1.0 | 0 | 732 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 46.8 | 23.9 | 1,221 ms | 42.0 ms | 2.0 | 0 | 1,401 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 92.4 | 23.4 | 2,161 ms | 42.7 ms | 4.0 | 0 | 2,771 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 13.1 | 13.3 | 1,084 ms | 75.2 ms | 1.0 | 0 | 391 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 25.4 | 13.0 | 1,507 ms | 77.0 ms | 2.0 | 0 | 758 | 29.9 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.