Speed test matrix
Qwen3.5-9B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-z-lab-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-9B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.5-9B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context20.6K397 ms TTFT
16K context20.7K784 ms TTFT
32K context19.2K1,678 ms TTFT
64K context17K3,788 ms TTFT
128K context13.8K9,319 ms TTFT
251K context10.1K25,079 ms TTFT
256K context9,96025,825 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 142.676 ms TTFT | 284.799 ms TTFT | 539.3104 ms TTFT | 951.9246 ms TTFT | 1,511.1495 ms TTFT | 2,118.1941 ms TTFT | 2,332.21,741 ms TTFT | Over capacity |
| 16K | 124.793 ms TTFT | 245.2136 ms TTFT | 480.1151 ms TTFT | 868.5416 ms TTFT | 1,299.6652 ms TTFT | 1,777.91,102 ms TTFT | 1,959.02,274 ms TTFT | Over capacity |
| 32K | 105.6125 ms TTFT | 206.6192 ms TTFT | 417.1295 ms TTFT | 725.8480 ms TTFT | 1,009.3692 ms TTFT | 1,364.71,430 ms TTFT | Over capacity | Over capacity |
| 64K | 75.0158 ms TTFT | 147.9227 ms TTFT | 290.2341 ms TTFT | 539.0527 ms TTFT | 666.6878 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 45.6346 ms TTFT | 98.7470 ms TTFT | 189.1723 ms TTFT | 341.11,040 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 26.8670 ms TTFT | 52.0927 ms TTFT | 99.31,396 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.6K | — | 397 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 20.7K | — | 784 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.2K | — | 1,678 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17K | — | 3,788 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.8K | — | 9,319 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.1K | — | 25,079 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,960.0 | — | 25,825 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 142.6 | 148.7 | 76 ms | 6.7 ms | 1.0 | 1 | 4,275 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 284.7 | 150.2 | 99 ms | 6.7 ms | 2.0 | 2 | 8,539 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 539.3 | 137.7 | 104 ms | 7.3 ms | 4.0 | 4 | 16,177 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 951.9 | 125.7 | 246 ms | 8.0 ms | 8.0 | 5 | 28,537 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,511.1 | 96.6 | 495 ms | 10.3 ms | 16.0 | 0 | 45,315 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,118.1 | 67.1 | 941 ms | 14.9 ms | 32.0 | 0 | 63,544 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,332.2 | 37.2 | 1,741 ms | 26.9 ms | 64.0 | 0 | 69,875 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 124.7 | 153.2 | 93 ms | 6.7 ms | 1.0 | 1 | 3,739 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 245.2 | 159.4 | 136 ms | 6.6 ms | 2.0 | 2 | 7,352 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 480.1 | 125.7 | 151 ms | 8.0 ms | 4.0 | 2 | 14,400 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 868.5 | 110.5 | 416 ms | 9.0 ms | 8.0 | 0 | 26,055 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,299.6 | 82.8 | 652 ms | 12.1 ms | 16.0 | 0 | 38,972 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,777.9 | 57.0 | 1,102 ms | 17.5 ms | 32.0 | 0 | 53,276 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,959.0 | 31.2 | 2,274 ms | 32.0 ms | 64.0 | 0 | 58,697 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 105.6 | 108.7 | 125 ms | 9.2 ms | 1.0 | 0 | 3,167 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 206.6 | 105.6 | 192 ms | 9.5 ms | 2.0 | 0 | 6,192 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 417.1 | 104.0 | 295 ms | 9.6 ms | 4.0 | 0 | 12,505 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 725.8 | 93.0 | 480 ms | 10.8 ms | 8.0 | 0 | 21,751 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,009.3 | 63.9 | 692 ms | 15.6 ms | 16.0 | 0 | 30,278 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,364.7 | 44.2 | 1,430 ms | 22.6 ms | 32.0 | 0 | 40,930 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 75.0 | 77.2 | 158 ms | 13.0 ms | 1.0 | 0 | 2,251 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 147.9 | 76.3 | 227 ms | 13.1 ms | 2.0 | 0 | 4,433 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 290.2 | 74.5 | 341 ms | 13.4 ms | 4.0 | 0 | 8,704 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 539.0 | 69.3 | 527 ms | 14.4 ms | 8.0 | 0 | 16,170 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 666.6 | 43.9 | 878 ms | 22.8 ms | 16.0 | 0 | 19,990 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 45.6 | 45.5 | 346 ms | 22.0 ms | 1.0 | 0 | 1,367 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 98.7 | 50.5 | 470 ms | 19.8 ms | 2.0 | 0 | 2,961 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 189.1 | 49.2 | 723 ms | 20.3 ms | 4.0 | 0 | 5,671 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 341.1 | 45.3 | 1,040 ms | 22.1 ms | 8.0 | 0 | 10,217 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 26.8 | 28.4 | 670 ms | 35.2 ms | 1.0 | 0 | 803 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 52.0 | 27.9 | 927 ms | 36.1 ms | 2.0 | 0 | 1,560 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 99.3 | 26.2 | 1,396 ms | 38.2 ms | 4.0 | 0 | 2,977 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.