Speed test matrix
Qwen3.5-9B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-9B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.94 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context20.4K401 ms TTFT
16K context20.3K801 ms TTFT
32K context19K1,701 ms TTFT
64K context16.6K3,885 ms TTFT
128K context13.2K9,753 ms TTFT
251K context9,45526,676 ms TTFT
256K context9,31227,624 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 112.556 ms TTFT | 223.092 ms TTFT | 451.5150 ms TTFT | 818.7274 ms TTFT | 1,488.6333 ms TTFT | 2,424.0737 ms TTFT | 3,485.61,152 ms TTFT | 4,683.42,188 ms TTFT |
| 16K | 116.394 ms TTFT | 227.0164 ms TTFT | 465.0231 ms TTFT | 816.7424 ms TTFT | 1,393.7676 ms TTFT | 2,247.51,133 ms TTFT | 3,135.82,020 ms TTFT | Over capacity |
| 32K | 107.2109 ms TTFT | 215.9145 ms TTFT | 441.5251 ms TTFT | 776.7389 ms TTFT | 1,319.8659 ms TTFT | Looped | Over capacity | Over capacity |
| 64K | 100.8186 ms TTFT | 216.9276 ms TTFT | 418.8398 ms TTFT | 739.9580 ms TTFT | 1,167.61,003 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 93.5360 ms TTFT | 190.3441 ms TTFT | 374.6642 ms TTFT | 646.4971 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 77.2683 ms TTFT | 144.1944 ms TTFT | 317.31,334 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.4K | — | 401 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 20.3K | — | 801 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19K | — | 1,701 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.6K | — | 3,885 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.2K | — | 9,753 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 9,455.0 | — | 26,676 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,312.0 | — | 27,624 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 112.5 | 113.5 | 56 ms | 8.8 ms | 1.0 | 0 | 3,375 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 223.0 | 111.6 | 92 ms | 9.0 ms | 2.0 | 0 | 6,687 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 451.5 | 115.1 | 150 ms | 8.7 ms | 4.0 | 0 | 13,541 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 818.7 | 103.6 | 274 ms | 9.7 ms | 8.0 | 0 | 24,560 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,488.6 | 93.8 | 333 ms | 10.7 ms | 16.0 | 0 | 44,659 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,424.0 | 75.6 | 737 ms | 13.2 ms | 32.0 | 0 | 72,716 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,485.6 | 54.0 | 1,152 ms | 18.5 ms | 64.0 | 0 | 104,490 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,683.4 | 35.5 | 2,188 ms | 28.1 ms | 128.0 | 0 | 140,401 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 116.3 | 115.7 | 94 ms | 8.6 ms | 1.0 | 0 | 3,489 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 227.0 | 113.4 | 164 ms | 8.8 ms | 2.0 | 0 | 6,808 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 465.0 | 116.4 | 231 ms | 8.6 ms | 4.0 | 0 | 13,949 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 816.7 | 102.2 | 424 ms | 9.8 ms | 8.0 | 0 | 24,503 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,393.7 | 87.4 | 676 ms | 11.4 ms | 16.0 | 0 | 41,790 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,247.5 | 69.5 | 1,133 ms | 14.4 ms | 32.0 | 0 | 67,403 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,135.8 | 47.3 | 2,020 ms | 21.1 ms | 64.0 | 0 | 93,984 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 107.2 | 107.7 | 109 ms | 9.3 ms | 1.0 | 0 | 3,214 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 215.9 | 108.3 | 145 ms | 9.2 ms | 2.0 | 0 | 6,476 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 441.5 | 110.2 | 251 ms | 9.1 ms | 4.0 | 0 | 13,246 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 776.7 | 97.5 | 389 ms | 10.3 ms | 8.0 | 0 | 23,291 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,319.8 | 83.0 | 659 ms | 12.0 ms | 16.0 | 0 | 39,576 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 100.8 | 101.1 | 186 ms | 9.9 ms | 1.0 | 0 | 3,024 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 216.9 | 108.6 | 276 ms | 9.2 ms | 2.0 | 0 | 6,506 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 418.8 | 104.5 | 398 ms | 9.6 ms | 4.0 | 0 | 12,557 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 739.9 | 93.6 | 580 ms | 10.7 ms | 8.0 | 0 | 22,195 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,167.6 | 71.9 | 1,003 ms | 13.9 ms | 16.0 | 0 | 35,002 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 93.5 | 93.9 | 360 ms | 10.6 ms | 1.0 | 0 | 2,802 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 190.3 | 94.9 | 441 ms | 10.5 ms | 2.0 | 0 | 5,708 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 374.6 | 94.1 | 642 ms | 10.6 ms | 4.0 | 0 | 11,234 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 646.4 | 81.5 | 971 ms | 12.3 ms | 8.0 | 0 | 19,383 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 77.2 | 76.7 | 683 ms | 13.0 ms | 1.0 | 0 | 2,314 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 144.1 | 72.8 | 944 ms | 13.7 ms | 2.0 | 0 | 4,321 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 317.3 | 79.5 | 1,334 ms | 12.6 ms | 4.0 | 0 | 9,515 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.