Speed test matrix
Qwen3.6-27B-int4-AutoRound on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT4+BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4+BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-intel-vllm-73a5831127a9/bin/vllm" serve Intel/Qwen3.6-27B-int4-AutoRound --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context6,1671,328 ms TTFT
16K context6,0532,680 ms TTFT
32K context5,7155,651 ms TTFT
64K context5,13012,559 ms TTFT
128K context4,25230,265 ms TTFT
251K context3,18779,130 ms TTFT
256K context3,16481,304 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 75.193 ms TTFT | 132.6134 ms TTFT | 252.1272 ms TTFT | 498.7424 ms TTFT | 851.11,005 ms TTFT | 1,297.71,819 ms TTFT | 1,737.43,626 ms TTFT | Over capacity |
| 16K | 73.0146 ms TTFT | 128.3241 ms TTFT | 243.1442 ms TTFT | 471.7759 ms TTFT | 772.51,618 ms TTFT | 1,124.52,802 ms TTFT | Over capacity | Over capacity |
| 32K | 69.6113 ms TTFT | 121.2155 ms TTFT | 227.5216 ms TTFT | 424.7352 ms TTFT | 654.7612 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 63.6177 ms TTFT | 102.4237 ms TTFT | 193.4335 ms TTFT | 352.8514 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 54.2322 ms TTFT | 81.4412 ms TTFT | 153.0596 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 41.4866 ms TTFT | 58.11,131 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,167.0 | — | 1,328 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,053.0 | — | 2,680 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,715.0 | — | 5,651 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,130.0 | — | 12,559 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,252.0 | — | 30,265 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,187.0 | — | 79,130 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,164.0 | — | 81,304 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 75.1 | 75.1 | 93 ms | 13.3 ms | 1.0 | 0 | 2,252 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 132.6 | 66.3 | 134 ms | 15.1 ms | 2.0 | 0 | 3,976 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 252.1 | 63.1 | 272 ms | 15.8 ms | 4.0 | 0 | 7,560 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 498.7 | 62.3 | 424 ms | 16.1 ms | 8.0 | 0 | 14,952 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 851.1 | 53.3 | 1,005 ms | 18.8 ms | 16.0 | 0 | 25,520 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,297.7 | 40.5 | 1,819 ms | 24.7 ms | 32.0 | 0 | 38,912 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,737.4 | 27.0 | 3,626 ms | 37.0 ms | 64.0 | 0 | 52,096 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 73.0 | 73.0 | 146 ms | 13.7 ms | 1.0 | 0 | 2,190 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 128.3 | 64.1 | 241 ms | 15.6 ms | 2.0 | 0 | 3,848 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 243.1 | 60.8 | 442 ms | 16.4 ms | 4.0 | 0 | 7,288 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 471.7 | 58.9 | 759 ms | 17.0 ms | 8.0 | 0 | 14,144 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 772.5 | 48.4 | 1,618 ms | 20.7 ms | 16.0 | 0 | 23,168 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,124.5 | 34.9 | 2,802 ms | 28.7 ms | 32.0 | 0 | 33,728 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 69.6 | 69.6 | 113 ms | 14.4 ms | 1.0 | 0 | 2,087 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 121.2 | 60.6 | 155 ms | 16.5 ms | 2.0 | 0 | 3,634 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 227.5 | 56.8 | 216 ms | 17.6 ms | 4.0 | 0 | 6,824 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 424.7 | 53.0 | 352 ms | 18.9 ms | 8.0 | 0 | 12,741 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 654.7 | 40.7 | 612 ms | 24.6 ms | 16.0 | 0 | 19,632 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 63.6 | 63.7 | 177 ms | 15.7 ms | 1.0 | 0 | 1,909 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 102.4 | 51.2 | 237 ms | 19.5 ms | 2.0 | 0 | 3,070 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 193.4 | 48.3 | 335 ms | 20.7 ms | 4.0 | 0 | 5,800 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 352.8 | 43.8 | 514 ms | 22.8 ms | 8.0 | 0 | 10,576 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 54.2 | 54.3 | 322 ms | 18.4 ms | 1.0 | 0 | 1,627 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 81.4 | 40.7 | 412 ms | 24.6 ms | 2.0 | 0 | 2,442 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 153.0 | 38.2 | 596 ms | 26.2 ms | 4.0 | 0 | 4,588 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 41.4 | 41.5 | 866 ms | 24.1 ms | 1.0 | 0 | 1,243 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 58.1 | 28.9 | 1,131 ms | 34.6 ms | 2.0 | 0 | 1,742 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.