Speed test matrix
Qwen3.6-27B on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.6-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context8,599952 ms TTFT
16K context8,5831,890 ms TTFT
32K context8,2303,924 ms TTFT
64K context7,6508,421 ms TTFT
128K context6,65519,339 ms TTFT
251K context5,33347,287 ms TTFT
256K context5,27748,746 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 47.180 ms TTFT | 85.7122 ms TTFT | 168.2243 ms TTFT | 322.8410 ms TTFT | 624.2456 ms TTFT | 1,078.51,441 ms TTFT | 1,760.01,594 ms TTFT | 2,450.73,492 ms TTFT |
| 16K | 46.8119 ms TTFT | 84.8189 ms TTFT | 166.1353 ms TTFT | 316.7559 ms TTFT | 602.1923 ms TTFT | 1,012.51,697 ms TTFT | 1,595.02,675 ms TTFT | Over capacity |
| 32K | 45.8116 ms TTFT | 83.3169 ms TTFT | 162.5221 ms TTFT | 305.5353 ms TTFT | 561.5462 ms TTFT | 904.4917 ms TTFT | Over capacity | Over capacity |
| 64K | 44.1189 ms TTFT | 80.3258 ms TTFT | 155.1348 ms TTFT | 284.3545 ms TTFT | 495.1831 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 41.4347 ms TTFT | 71.2457 ms TTFT | 138.3627 ms TTFT | 250.0955 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 37.0721 ms TTFT | 60.0936 ms TTFT | 116.31,353 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,599.0 | — | 952 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,583.0 | — | 1,890 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,230.0 | — | 3,924 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,650.0 | — | 8,421 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,655.0 | — | 19,339 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,333.0 | — | 47,287 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,277.0 | — | 48,746 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 47.1 | 47.1 | 80 ms | 21.2 ms | 1.0 | 0 | 1,413 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 85.7 | 42.8 | 122 ms | 23.4 ms | 2.0 | 0 | 2,570 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 168.2 | 42.1 | 243 ms | 23.8 ms | 4.0 | 0 | 5,044 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 322.8 | 40.4 | 410 ms | 24.8 ms | 8.0 | 0 | 9,680 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 624.2 | 38.7 | 456 ms | 25.8 ms | 16.0 | 0 | 18,720 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,078.5 | 33.7 | 1,441 ms | 29.6 ms | 32.0 | 0 | 32,352 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,760.0 | 26.7 | 1,594 ms | 37.5 ms | 64.0 | 0 | 52,800 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,450.7 | 18.2 | 3,492 ms | 55.1 ms | 128.0 | 0 | 73,472 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 46.8 | 46.7 | 119 ms | 21.4 ms | 1.0 | 0 | 1,402 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 84.8 | 42.3 | 189 ms | 23.6 ms | 2.0 | 0 | 2,542 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 166.1 | 41.6 | 353 ms | 24.1 ms | 4.0 | 0 | 4,984 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 316.7 | 39.5 | 559 ms | 25.3 ms | 8.0 | 0 | 9,496 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 602.1 | 37.4 | 923 ms | 26.8 ms | 16.0 | 0 | 18,048 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,012.5 | 31.2 | 1,697 ms | 32.1 ms | 32.0 | 0 | 30,368 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,595.0 | 23.8 | 2,675 ms | 42.1 ms | 64.0 | 0 | 47,808 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 45.8 | 45.8 | 116 ms | 21.8 ms | 1.0 | 0 | 1,375 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 83.3 | 41.6 | 169 ms | 24.0 ms | 2.0 | 0 | 2,498 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 162.5 | 40.6 | 221 ms | 24.6 ms | 4.0 | 0 | 4,872 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 305.5 | 38.1 | 353 ms | 26.2 ms | 8.0 | 0 | 9,160 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 561.5 | 35.0 | 462 ms | 28.6 ms | 16.0 | 0 | 16,832 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 904.4 | 28.0 | 917 ms | 35.7 ms | 32.0 | 0 | 27,104 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 44.1 | 44.2 | 189 ms | 22.6 ms | 1.0 | 0 | 1,324 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 80.3 | 40.1 | 258 ms | 24.9 ms | 2.0 | 0 | 2,408 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 155.1 | 38.7 | 348 ms | 25.8 ms | 4.0 | 0 | 4,652 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 284.3 | 35.5 | 545 ms | 28.2 ms | 8.0 | 0 | 8,528 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 495.1 | 30.7 | 831 ms | 32.6 ms | 16.0 | 0 | 14,848 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 41.4 | 41.4 | 347 ms | 24.1 ms | 1.0 | 0 | 1,241 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 71.2 | 35.6 | 457 ms | 28.1 ms | 2.0 | 0 | 2,134 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 138.3 | 34.5 | 627 ms | 29.0 ms | 4.0 | 0 | 4,144 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 250.0 | 31.0 | 955 ms | 32.2 ms | 8.0 | 0 | 7,496 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 37.0 | 37.1 | 721 ms | 27.0 ms | 1.0 | 0 | 1,111 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 60.0 | 29.9 | 936 ms | 33.4 ms | 2.0 | 0 | 1,800 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 116.3 | 28.8 | 1,353 ms | 34.7 ms | 4.0 | 0 | 3,488 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.