Speed test matrix
Qwen3.5-9B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-9B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context20.5K400 ms TTFT
16K context20.2K803 ms TTFT
32K context19K1,704 ms TTFT
64K context16.5K3,900 ms TTFT
128K context13.2K9,761 ms TTFT
251K context9,44226,714 ms TTFT
256K context9,34627,522 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 131.458 ms TTFT | 264.283 ms TTFT | 536.8112 ms TTFT | 959.7201 ms TTFT | 1,719.9336 ms TTFT | 2,766.5708 ms TTFT | 3,971.11,154 ms TTFT | Over capacity |
| 16K | 125.594 ms TTFT | 264.4119 ms TTFT | 515.7148 ms TTFT | 926.0458 ms TTFT | 1,635.8679 ms TTFT | 2,574.21,135 ms TTFT | 3,570.72,057 ms TTFT | Over capacity |
| 32K | 123.1110 ms TTFT | 257.3152 ms TTFT | 499.2133 ms TTFT | 903.5379 ms TTFT | 1,508.7605 ms TTFT | 2,307.21,039 ms TTFT | Over capacity | Over capacity |
| 64K | 115.2191 ms TTFT | 239.8270 ms TTFT | 482.8217 ms TTFT | 832.4607 ms TTFT | 1,324.9957 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 109.6374 ms TTFT | 220.7482 ms TTFT | 445.7625 ms TTFT | 737.1939 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 90.0691 ms TTFT | 169.4978 ms TTFT | 364.01,327 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.5K | — | 400 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 20.2K | — | 803 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19K | — | 1,704 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.5K | — | 3,900 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.2K | — | 9,761 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 9,442.0 | — | 26,714 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,346.0 | — | 27,522 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 131.4 | 140.3 | 58 ms | 7.2 ms | 1.0 | 1 | 3,942 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 264.2 | 136.0 | 83 ms | 7.4 ms | 2.0 | 2 | 7,923 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 536.8 | 137.5 | 112 ms | 7.3 ms | 4.0 | 4 | 16,099 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 959.7 | 124.9 | 201 ms | 8.0 ms | 8.0 | 6 | 28,778 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,719.9 | 109.3 | 336 ms | 9.1 ms | 16.0 | 0 | 51,587 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,766.5 | 87.3 | 708 ms | 11.5 ms | 32.0 | 0 | 82,982 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,971.1 | 62.1 | 1,154 ms | 16.1 ms | 64.0 | 0 | 119,097 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 125.5 | 146.1 | 94 ms | 7.0 ms | 1.0 | 1 | 3,763 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 264.4 | 140.3 | 119 ms | 7.1 ms | 2.0 | 2 | 7,928 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 515.7 | 134.8 | 148 ms | 7.4 ms | 4.0 | 4 | 15,469 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 926.0 | 117.9 | 458 ms | 8.5 ms | 8.0 | 0 | 27,767 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,635.8 | 103.7 | 679 ms | 9.6 ms | 16.0 | 0 | 49,057 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,574.2 | 80.5 | 1,135 ms | 12.4 ms | 32.0 | 0 | 77,199 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,570.7 | 54.8 | 2,057 ms | 18.2 ms | 64.0 | 0 | 107,022 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 123.1 | 148.0 | 110 ms | 6.9 ms | 1.0 | 1 | 3,693 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 257.3 | 138.5 | 152 ms | 7.2 ms | 2.0 | 2 | 7,717 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 499.2 | 128.9 | 133 ms | 7.8 ms | 4.0 | 3 | 14,976 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 903.5 | 114.4 | 379 ms | 8.7 ms | 8.0 | 0 | 27,094 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,508.7 | 95.4 | 605 ms | 10.5 ms | 16.0 | 0 | 45,254 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,307.2 | 72.5 | 1,039 ms | 13.8 ms | 32.0 | 0 | 69,155 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 115.2 | 116.7 | 191 ms | 8.6 ms | 1.0 | 0 | 3,457 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 239.8 | 121.1 | 270 ms | 8.3 ms | 2.0 | 0 | 7,194 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 482.8 | 121.9 | 217 ms | 8.2 ms | 4.0 | 4 | 14,477 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 832.4 | 105.4 | 607 ms | 9.5 ms | 8.0 | 0 | 24,967 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,324.9 | 83.1 | 957 ms | 12.0 ms | 16.0 | 0 | 39,728 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 109.6 | 111.0 | 374 ms | 9.0 ms | 1.0 | 0 | 3,287 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 220.7 | 110.3 | 482 ms | 9.1 ms | 2.0 | 0 | 6,620 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 445.7 | 112.6 | 625 ms | 8.9 ms | 4.0 | 0 | 13,370 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 737.1 | 92.5 | 939 ms | 10.8 ms | 8.0 | 0 | 22,108 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 90.0 | 90.7 | 691 ms | 11.0 ms | 1.0 | 0 | 2,700 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 169.4 | 84.7 | 978 ms | 11.8 ms | 2.0 | 0 | 5,080 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 364.0 | 91.8 | 1,327 ms | 10.9 ms | 4.0 | 0 | 10,912 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.