Speed test matrix
Qwen3.5-4B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-4B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.92 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context32.9K249 ms TTFT
16K context32.4K500 ms TTFT
32K context29.4K1,099 ms TTFT
64K context23.9K2,695 ms TTFT
128K context17.5K7,371 ms TTFT
251K context11.5K21,984 ms TTFT
256K context11.3K22,775 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 174.545 ms TTFT | 333.169 ms TTFT | 662.768 ms TTFT | 1,139.190 ms TTFT | 1,952.1149 ms TTFT | 2,893.3551 ms TTFT | 4,062.8834 ms TTFT | 5,307.71,434 ms TTFT |
| 16K | 138.171 ms TTFT | 313.5109 ms TTFT | 596.0103 ms TTFT | 1,076.4112 ms TTFT | 1,749.3429 ms TTFT | 2,701.1706 ms TTFT | 3,579.71,374 ms TTFT | Over capacity |
| 32K | 150.488 ms TTFT | 276.1104 ms TTFT | 588.5102 ms TTFT | Looped | 1,638.9466 ms TTFT | 2,344.4777 ms TTFT | Over capacity | Over capacity |
| 64K | 148.7172 ms TTFT | 282.4184 ms TTFT | 569.9184 ms TTFT | 912.8343 ms TTFT | 1,367.5853 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 115.1327 ms TTFT | 246.2354 ms TTFT | 466.7521 ms TTFT | 780.61,065 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 93.0662 ms TTFT | 156.1901 ms TTFT | 365.71,236 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32.9K | — | 249 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.4K | — | 500 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.4K | — | 1,099 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 23.9K | — | 2,695 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.5K | — | 7,371 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 11.5K | — | 21,984 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.3K | — | 22,775 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 174.5 | 175.6 | 45 ms | 5.7 ms | 1.0 | 1 | 5,235 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 333.1 | 170.4 | 69 ms | 5.9 ms | 2.0 | 2 | 9,989 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 662.7 | 168.2 | 68 ms | 5.9 ms | 4.0 | 4 | 19,880 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,139.1 | 147.7 | 90 ms | 6.8 ms | 7.9 | 8 | 34,170 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,952.1 | 129.4 | 149 ms | 7.7 ms | 15.9 | 12 | 58,534 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,893.3 | 91.4 | 551 ms | 10.9 ms | 32.0 | 0 | 86,765 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,062.8 | 65.2 | 834 ms | 15.3 ms | 64.0 | 0 | 121,863 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,307.7 | 40.8 | 1,434 ms | 24.5 ms | 128.0 | 0 | 159,181 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 138.1 | 159.8 | 71 ms | 6.4 ms | 1.0 | 1 | 4,141 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 313.5 | 167.0 | 109 ms | 6.0 ms | 2.0 | 2 | 9,405 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 596.0 | 159.6 | 103 ms | 6.3 ms | 4.0 | 4 | 17,876 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,076.4 | 149.0 | 112 ms | 6.7 ms | 8.0 | 7 | 32,292 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,749.3 | 112.5 | 429 ms | 8.9 ms | 16.0 | 0 | 52,462 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,701.1 | 84.8 | 706 ms | 11.8 ms | 32.0 | 0 | 81,009 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,579.7 | 55.4 | 1,374 ms | 18.0 ms | 64.0 | 0 | 107,387 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 150.4 | 156.1 | 88 ms | 6.4 ms | 1.0 | 1 | 4,511 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 276.1 | 149.2 | 104 ms | 6.7 ms | 2.0 | 2 | 8,279 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 588.5 | 159.0 | 102 ms | 6.3 ms | 4.0 | 4 | 17,650 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 1,638.9 | 105.6 | 466 ms | 9.5 ms | 16.0 | 0 | 49,154 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,344.4 | 74.7 | 777 ms | 13.4 ms | 32.0 | 0 | 70,331 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 148.7 | 151.7 | 172 ms | 6.6 ms | 1.0 | 1 | 4,461 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 282.4 | 152.7 | 184 ms | 6.6 ms | 2.0 | 2 | 8,470 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 569.9 | 148.6 | 184 ms | 6.7 ms | 4.0 | 4 | 17,089 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 912.8 | 121.7 | 343 ms | 8.2 ms | 8.0 | 4 | 27,373 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,367.5 | 86.0 | 853 ms | 11.6 ms | 16.0 | 0 | 41,019 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 115.1 | 118.7 | 327 ms | 8.4 ms | 1.0 | 0 | 3,453 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 246.2 | 132.0 | 354 ms | 7.6 ms | 2.0 | 1 | 7,383 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 466.7 | 121.9 | 521 ms | 8.2 ms | 4.0 | 2 | 14,000 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 780.6 | 99.7 | 1,065 ms | 10.0 ms | 8.0 | 0 | 23,409 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 93.0 | 95.2 | 662 ms | 10.5 ms | 1.0 | 0 | 2,791 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 156.1 | 79.8 | 901 ms | 12.7 ms | 2.0 | 0 | 4,681 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 365.7 | 89.7 | 1,236 ms | 11.2 ms | 4.0 | 0 | 10,971 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.