Speed test matrix
Qwen3.5-27B on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context8,620950 ms TTFT
16K context8,5581,895 ms TTFT
32K context8,2303,924 ms TTFT
64K context7,6488,423 ms TTFT
128K context6,65419,340 ms TTFT
251K context5,33647,269 ms TTFT
256K context5,28148,705 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 47.278 ms TTFT | 85.8121 ms TTFT | 168.4244 ms TTFT | 323.2413 ms TTFT | 623.8457 ms TTFT | 1,076.21,443 ms TTFT | 1,757.31,595 ms TTFT | 2,451.63,502 ms TTFT |
| 16K | 46.8118 ms TTFT | 84.9199 ms TTFT | 166.2355 ms TTFT | 316.9558 ms TTFT | 601.7922 ms TTFT | 1,011.71,675 ms TTFT | 1,592.02,678 ms TTFT | Over capacity |
| 32K | 45.8113 ms TTFT | 83.4167 ms TTFT | 162.5217 ms TTFT | 305.6334 ms TTFT | 562.0475 ms TTFT | 903.6857 ms TTFT | Over capacity | Over capacity |
| 64K | 44.2181 ms TTFT | 80.3248 ms TTFT | 155.1343 ms TTFT | 284.5536 ms TTFT | 495.3842 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 41.4331 ms TTFT | 71.2431 ms TTFT | 138.4593 ms TTFT | 249.9947 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 37.1703 ms TTFT | 60.0917 ms TTFT | 116.31,273 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,620.0 | — | 950 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,558.0 | — | 1,895 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,230.0 | — | 3,924 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,648.0 | — | 8,423 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,654.0 | — | 19,340 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,336.0 | — | 47,269 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,281.0 | — | 48,705 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 47.2 | 47.1 | 78 ms | 21.2 ms | 1.0 | 0 | 1,414 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 85.8 | 42.9 | 121 ms | 23.3 ms | 2.0 | 0 | 2,572 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 168.4 | 42.1 | 244 ms | 23.8 ms | 4.0 | 0 | 5,052 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 323.2 | 40.4 | 413 ms | 24.8 ms | 8.0 | 0 | 9,696 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 623.8 | 38.7 | 457 ms | 25.8 ms | 16.0 | 0 | 18,704 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,076.2 | 33.7 | 1,443 ms | 29.7 ms | 32.0 | 0 | 32,288 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,757.3 | 26.6 | 1,595 ms | 37.5 ms | 64.0 | 0 | 52,672 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,451.6 | 18.1 | 3,502 ms | 55.1 ms | 128.0 | 0 | 73,472 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 46.8 | 46.8 | 118 ms | 21.4 ms | 1.0 | 0 | 1,403 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 84.9 | 42.4 | 199 ms | 23.6 ms | 2.0 | 0 | 2,544 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 166.2 | 41.6 | 355 ms | 24.1 ms | 4.0 | 0 | 4,984 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 316.9 | 39.5 | 558 ms | 25.3 ms | 8.0 | 0 | 9,504 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 601.7 | 37.3 | 922 ms | 26.8 ms | 16.0 | 0 | 18,048 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,011.7 | 31.1 | 1,675 ms | 32.1 ms | 32.0 | 0 | 30,336 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,592.0 | 23.7 | 2,678 ms | 42.1 ms | 64.0 | 0 | 47,744 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 45.8 | 45.9 | 113 ms | 21.8 ms | 1.0 | 0 | 1,375 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 83.4 | 41.7 | 167 ms | 24.0 ms | 2.0 | 0 | 2,500 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 162.5 | 40.6 | 217 ms | 24.6 ms | 4.0 | 0 | 4,872 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 305.6 | 38.2 | 334 ms | 26.2 ms | 8.0 | 0 | 9,160 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 562.0 | 34.9 | 475 ms | 28.7 ms | 16.0 | 0 | 16,848 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 903.6 | 27.9 | 857 ms | 35.8 ms | 32.0 | 0 | 27,104 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 44.2 | 44.2 | 181 ms | 22.6 ms | 1.0 | 0 | 1,324 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 80.3 | 40.1 | 248 ms | 24.9 ms | 2.0 | 0 | 2,408 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 155.1 | 38.7 | 343 ms | 25.8 ms | 4.0 | 0 | 4,652 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 284.5 | 35.4 | 536 ms | 28.2 ms | 8.0 | 0 | 8,536 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 495.3 | 30.6 | 842 ms | 32.7 ms | 16.0 | 0 | 14,848 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 41.4 | 41.4 | 331 ms | 24.1 ms | 1.0 | 0 | 1,242 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 71.2 | 35.6 | 431 ms | 28.1 ms | 2.0 | 0 | 2,136 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 138.4 | 34.5 | 593 ms | 29.0 ms | 4.0 | 0 | 4,148 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 249.9 | 31.0 | 947 ms | 32.3 ms | 8.0 | 0 | 7,496 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 37.1 | 37.1 | 703 ms | 27.0 ms | 1.0 | 0 | 1,112 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 60.0 | 30.0 | 917 ms | 33.4 ms | 2.0 | 0 | 1,800 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 116.3 | 28.8 | 1,273 ms | 34.7 ms | 4.0 | 0 | 3,484 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.