Speed test matrix
Qwen3.5-2B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-2B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.93 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context66.2K124 ms TTFT
16K context70.4K230 ms TTFT
32K context66.2K488 ms TTFT
64K context54.9K1,173 ms TTFT
128K context41.2K3,126 ms TTFT
251K context27.2K9,265 ms TTFT
256K context26.6K9,655 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 209.639 ms TTFT | 440.953 ms TTFT | 852.284 ms TTFT | 1,742.5135 ms TTFT | Looped | Looped | Looped | Looped |
| 16K | 264.551 ms TTFT | 424.686 ms TTFT | 861.780 ms TTFT | 1,724.2157 ms TTFT | 2,934.6253 ms TTFT | Looped | 5,881.6807 ms TTFT | Looped |
| 32K | 229.574 ms TTFT | 469.392 ms TTFT | Looped | 1,671.7170 ms TTFT | 2,661.5326 ms TTFT | Looped | Looped | 6,426.72,496 ms TTFT |
| 64K | 210.8130 ms TTFT | 410.8184 ms TTFT | 868.9164 ms TTFT | 1,556.5297 ms TTFT | 2,254.0406 ms TTFT | Looped | Looped | Over capacity |
| 128K | 169.8254 ms TTFT | 368.5312 ms TTFT | 755.2300 ms TTFT | Looped | 1,865.31,010 ms TTFT | Looped | Over capacity | Over capacity |
| 251K | 163.5567 ms TTFT | 303.2582 ms TTFT | 530.8719 ms TTFT | 1,028.21,017 ms TTFT | Looped | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 66.2K | — | 124 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 70.4K | — | 230 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 66.2K | — | 488 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 54.9K | — | 1,173 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 41.2K | — | 3,126 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 27.2K | — | 9,265 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 26.6K | — | 9,655 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 209.6 | 218.4 | 39 ms | 4.6 ms | 1.0 | 1 | 6,287 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 440.9 | 234.1 | 53 ms | 4.3 ms | 2.0 | 3 | 13,224 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 852.2 | 216.3 | 84 ms | 4.6 ms | 4.0 | 5 | 25,565 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,742.5 | 226.9 | 135 ms | 4.4 ms | 8.0 | 9 | 52,277 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 264.5 | 283.7 | 51 ms | 3.5 ms | 1.0 | 2 | 7,934 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 424.6 | 219.7 | 86 ms | 4.6 ms | 2.0 | 2 | 12,738 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 861.7 | 222.3 | 80 ms | 4.5 ms | 4.0 | 4 | 25,851 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,724.2 | 216.5 | 157 ms | 4.6 ms | 8.0 | 9 | 51,723 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,934.6 | 190.4 | 253 ms | 5.3 ms | 15.9 | 16 | 88,038 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Valid | 5,881.6 | 93.5 | 807 ms | 10.7 ms | 63.9 | 2 | 178,425 | 30.3 s |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 229.5 | 235.2 | 74 ms | 4.3 ms | 1.0 | 1 | 6,884 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 469.3 | 243.3 | 92 ms | 4.1 ms | 2.0 | 3 | 14,078 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 1,671.7 | 213.0 | 170 ms | 4.7 ms | 8.0 | 8 | 50,153 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,661.5 | 171.7 | 326 ms | 5.8 ms | 15.9 | 16 | 79,846 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Valid | 6,426.7 | 51.7 | 2,496 ms | 19.3 ms | 128.0 | 0 | 192,850 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 210.8 | 225.6 | 130 ms | 4.6 ms | 1.0 | 1 | 6,325 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 410.8 | 205.1 | 184 ms | 4.9 ms | 2.0 | 2 | 12,322 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 868.9 | 228.4 | 164 ms | 4.4 ms | 4.0 | 5 | 26,066 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,556.5 | 202.5 | 297 ms | 4.9 ms | 8.0 | 9 | 46,696 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,254.0 | 157.3 | 406 ms | 6.4 ms | 16.0 | 16 | 67,621 | 30.0 s |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 169.8 | 182.9 | 254 ms | 5.5 ms | 1.0 | 1 | 5,093 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 368.5 | 194.7 | 312 ms | 5.2 ms | 2.0 | 2 | 11,054 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 755.2 | 191.7 | 300 ms | 5.2 ms | 3.9 | 4 | 22,657 | 30.0 s |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Valid | 1,865.3 | 124.2 | 1,010 ms | 8.1 ms | 15.8 | 8 | 55,959 | 30.0 s |
| Sustained decode | 128K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 163.5 | 172.3 | 567 ms | 5.8 ms | 1.0 | 1 | 4,908 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 303.2 | 169.6 | 582 ms | 5.9 ms | 2.0 | 2 | 9,096 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 530.8 | 148.7 | 719 ms | 6.7 ms | 3.9 | 4 | 15,926 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,028.2 | 140.5 | 1,017 ms | 7.1 ms | 7.8 | 7 | 30,844 | 30.0 s |
| Sustained decode | 251K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.