Speed test matrix
Qwen3.5-9B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-9B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER
Prefill
prompt tok/s by context length8K context21.1K387 ms TTFT
16K context21.3K763 ms TTFT
32K context19.8K1,634 ms TTFT
64K context17.5K3,677 ms TTFT
128K context14K9,166 ms TTFT
251K context10.2K24,751 ms TTFT
256K context10.1K25,519 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 83.145 ms TTFT | 155.165 ms TTFT | 308.689 ms TTFT | 617.7133 ms TTFT | 1,119.3200 ms TTFT | 1,887.2362 ms TTFT | 2,960.6688 ms TTFT | 4,431.91,178 ms TTFT |
| 16K | 81.760 ms TTFT | 153.2109 ms TTFT | 304.2132 ms TTFT | 604.9224 ms TTFT | 1,070.2348 ms TTFT | 1,751.8562 ms TTFT | 2,624.1996 ms TTFT | Over capacity |
| 32K | 79.677 ms TTFT | 149.6140 ms TTFT | 296.9169 ms TTFT | 581.4326 ms TTFT | 994.2529 ms TTFT | 1,555.8974 ms TTFT | Over capacity | Over capacity |
| 64K | 75.0123 ms TTFT | 141.6184 ms TTFT | 281.0246 ms TTFT | 531.4411 ms TTFT | 860.2687 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 67.3273 ms TTFT | 126.9397 ms TTFT | 251.5538 ms TTFT | 452.2935 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 56.1557 ms TTFT | 98.5781 ms TTFT | 193.21,135 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 21.1K | — | 387 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.3K | — | 763 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.8K | — | 1,634 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.5K | — | 3,677 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14K | — | 9,166 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.2K | — | 24,751 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.1K | — | 25,519 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 83.1 | 83.1 | 45 ms | 12.0 ms | 1.0 | 0 | 2,491 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 155.1 | 77.6 | 65 ms | 12.9 ms | 2.0 | 0 | 4,650 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 308.6 | 77.2 | 89 ms | 12.9 ms | 4.0 | 0 | 9,258 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 617.7 | 77.4 | 133 ms | 12.9 ms | 8.0 | 0 | 18,531 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,119.3 | 70.1 | 200 ms | 14.3 ms | 16.0 | 0 | 33,568 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,887.2 | 59.1 | 362 ms | 16.9 ms | 32.0 | 0 | 56,608 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,960.6 | 46.2 | 688 ms | 21.7 ms | 64.0 | 0 | 88,768 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,431.9 | 34.1 | 1,178 ms | 29.3 ms | 128.0 | 0 | 132,864 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 81.7 | 81.8 | 60 ms | 12.2 ms | 1.0 | 0 | 2,452 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 153.2 | 76.6 | 109 ms | 13.0 ms | 2.0 | 0 | 4,594 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 304.2 | 76.1 | 132 ms | 13.1 ms | 4.0 | 0 | 9,124 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 604.9 | 75.7 | 224 ms | 13.2 ms | 8.0 | 0 | 18,144 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,070.2 | 66.9 | 348 ms | 15.0 ms | 16.0 | 0 | 32,096 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,751.8 | 54.6 | 562 ms | 18.3 ms | 32.0 | 0 | 52,544 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,624.1 | 40.6 | 996 ms | 24.7 ms | 64.0 | 0 | 78,720 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 79.6 | 79.6 | 77 ms | 12.6 ms | 1.0 | 0 | 2,389 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 149.6 | 74.9 | 140 ms | 13.4 ms | 2.0 | 0 | 4,488 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 296.9 | 74.3 | 169 ms | 13.5 ms | 4.0 | 0 | 8,904 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 581.4 | 72.7 | 326 ms | 13.8 ms | 8.0 | 0 | 17,440 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 994.2 | 62.2 | 529 ms | 16.1 ms | 16.0 | 0 | 29,824 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,555.8 | 48.6 | 974 ms | 20.6 ms | 32.0 | 0 | 46,656 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 75.0 | 75.1 | 123 ms | 13.3 ms | 1.0 | 0 | 2,251 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 141.6 | 70.8 | 184 ms | 14.1 ms | 2.0 | 0 | 4,248 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 281.0 | 70.3 | 246 ms | 14.2 ms | 4.0 | 0 | 8,428 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 531.4 | 66.4 | 411 ms | 15.1 ms | 8.0 | 0 | 15,936 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 860.2 | 53.8 | 687 ms | 18.6 ms | 16.0 | 0 | 25,792 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 67.3 | 67.4 | 273 ms | 14.8 ms | 1.0 | 0 | 2,019 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 126.9 | 63.5 | 397 ms | 15.8 ms | 2.0 | 0 | 3,807 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 251.5 | 62.8 | 538 ms | 15.9 ms | 4.0 | 0 | 7,544 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 452.2 | 56.3 | 935 ms | 17.8 ms | 8.0 | 0 | 13,568 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 56.1 | 56.2 | 557 ms | 17.8 ms | 1.0 | 0 | 1,683 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 98.5 | 49.3 | 781 ms | 20.3 ms | 2.0 | 0 | 2,954 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 193.2 | 48.3 | 1,135 ms | 20.7 ms | 4.0 | 0 | 5,796 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.