Speed test matrix
Qwen3.6-27B on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.6-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context8,391976 ms TTFT
16K context8,3981,932 ms TTFT
32K context8,1053,984 ms TTFT
64K context7,5018,588 ms TTFT
128K context6,51519,753 ms TTFT
251K context5,16948,794 ms TTFT
256K context5,14150,031 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 73.6150 ms TTFT | 143.4238 ms TTFT | 297.2547 ms TTFT | 544.41,020 ms TTFT | 869.31,111 ms TTFT | 1,346.92,057 ms TTFT | 1,830.83,955 ms TTFT | 2,108.17,852 ms TTFT |
| 16K | 64.1174 ms TTFT | 126.1287 ms TTFT | 255.4601 ms TTFT | 474.71,082 ms TTFT | 716.81,209 ms TTFT | 1,069.32,717 ms TTFT | 1,421.94,653 ms TTFT | Over capacity |
| 32K | 50.8228 ms TTFT | 97.8358 ms TTFT | 197.2734 ms TTFT | 376.31,022 ms TTFT | 519.11,868 ms TTFT | 748.63,441 ms TTFT | Over capacity | Over capacity |
| 64K | 35.7337 ms TTFT | 71.4505 ms TTFT | 139.51,001 ms TTFT | 265.71,126 ms TTFT | 333.52,535 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 21.5557 ms TTFT | 44.3801 ms TTFT | 88.31,358 ms TTFT | 171.02,989 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 12.9912 ms TTFT | 26.71,248 ms TTFT | 51.42,042 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,391.0 | — | 976 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,398.0 | — | 1,932 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,105.0 | — | 3,984 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,501.0 | — | 8,588 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,515.0 | — | 19,753 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,169.0 | — | 48,794 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,141.0 | — | 50,031 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 73.6 | 74.6 | 150 ms | 13.4 ms | 1.0 | 0 | 2,208 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 143.4 | 72.3 | 238 ms | 13.8 ms | 2.0 | 0 | 4,300 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 297.2 | 74.2 | 547 ms | 13.5 ms | 4.0 | 0 | 8,915 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 544.4 | 68.6 | 1,020 ms | 14.6 ms | 8.0 | 0 | 16,315 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 869.3 | 54.8 | 1,111 ms | 18.2 ms | 16.0 | 0 | 26,078 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,346.9 | 41.3 | 2,057 ms | 24.2 ms | 32.0 | 0 | 40,342 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,830.8 | 26.8 | 3,955 ms | 37.3 ms | 64.0 | 0 | 54,863 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,108.1 | 14.4 | 7,852 ms | 69.4 ms | 128.0 | 0 | 63,189 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 64.1 | 65.2 | 174 ms | 15.3 ms | 1.0 | 0 | 1,922 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 126.1 | 63.5 | 287 ms | 15.7 ms | 2.0 | 0 | 3,783 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 255.4 | 64.2 | 601 ms | 15.6 ms | 4.0 | 0 | 7,654 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 474.7 | 60.1 | 1,082 ms | 16.6 ms | 8.0 | 0 | 14,229 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 716.8 | 44.7 | 1,209 ms | 22.4 ms | 16.0 | 0 | 21,495 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,069.3 | 32.6 | 2,717 ms | 30.7 ms | 32.0 | 0 | 32,065 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,421.9 | 20.7 | 4,653 ms | 48.4 ms | 64.0 | 0 | 42,528 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 50.8 | 51.0 | 228 ms | 19.6 ms | 1.0 | 0 | 1,523 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 97.8 | 49.6 | 358 ms | 20.2 ms | 2.0 | 0 | 2,931 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 197.2 | 49.7 | 734 ms | 20.1 ms | 4.0 | 0 | 5,909 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 376.3 | 47.1 | 1,022 ms | 21.2 ms | 8.0 | 0 | 11,271 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 519.1 | 32.5 | 1,868 ms | 30.8 ms | 16.0 | 0 | 15,567 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 748.6 | 22.5 | 3,441 ms | 44.4 ms | 32.0 | 0 | 22,445 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 35.7 | 36.2 | 337 ms | 27.6 ms | 1.0 | 0 | 1,071 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 71.4 | 35.9 | 505 ms | 27.9 ms | 2.0 | 0 | 2,139 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 139.5 | 35.5 | 1,001 ms | 28.2 ms | 4.0 | 0 | 4,181 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 265.7 | 33.5 | 1,126 ms | 29.9 ms | 8.0 | 0 | 7,959 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 333.5 | 20.7 | 2,535 ms | 48.3 ms | 16.0 | 0 | 9,970 | 29.9 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 21.5 | 22.1 | 557 ms | 45.3 ms | 1.0 | 0 | 646 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 44.3 | 22.5 | 801 ms | 44.5 ms | 2.0 | 0 | 1,328 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 88.3 | 22.5 | 1,358 ms | 44.4 ms | 4.0 | 0 | 2,646 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 171.0 | 21.8 | 2,989 ms | 46.0 ms | 8.0 | 0 | 5,116 | 29.9 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 12.9 | 13.1 | 912 ms | 76.3 ms | 1.0 | 0 | 386 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 26.7 | 13.5 | 1,248 ms | 74.0 ms | 2.0 | 0 | 801 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 51.4 | 13.0 | 2,042 ms | 77.1 ms | 4.0 | 0 | 1,538 | 29.9 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.