Speed test matrix
Qwen3.5-2B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-2B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --default-chat-template-kwargs '{"enable_thinking":false}' --attention-backend FLASHINFERPrefill
prompt tok/s by context length8K context72.1K114 ms TTFT
16K context75.2K216 ms TTFT
32K context71.1K454 ms TTFT
64K context60.7K1,061 ms TTFT
128K context45.4K2,835 ms TTFT
251K context30.8K8,202 ms TTFT
256K context30.3K8,496 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 286.632 ms TTFT | 518.348 ms TTFT | 1,012.671 ms TTFT | 1,895.8105 ms TTFT | Looped | Looped | Looped | Looped |
| 16K | 281.444 ms TTFT | 508.351 ms TTFT | Looped | 1,846.4142 ms TTFT | Looped | Looped | Looped | Looped |
| 32K | Looped | 494.291 ms TTFT | 957.7Requests queued129 ms TTFT | 1,777.1236 ms TTFT | Looped | Looped | 6,493.21,373 ms TTFT | Looped |
| 64K | 252.0116 ms TTFT | 465.1157 ms TTFT | 905.3184 ms TTFT | Looped | 2,581.2432 ms TTFT | 3,703.71,235 ms TTFT | 4,809.62,437 ms TTFT | Over capacity |
| 128K | 220.5244 ms TTFT | 408.3336 ms TTFT | 797.1359 ms TTFT | 1,360.0478 ms TTFT | 2,007.9502 ms TTFT | Looped | Over capacity | Over capacity |
| 251K | 176.8525 ms TTFT | 330.6626 ms TTFT | 646.3676 ms TTFT | 1,044.3680 ms TTFT | Looped | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 72.1K | — | 114 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 75.2K | — | 216 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 71.1K | — | 454 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 60.7K | — | 1,061 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 45.4K | — | 2,835 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 30.8K | — | 8,202 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 30.3K | — | 8,496 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 286.6 | 287.2 | 32 ms | 3.5 ms | 1.0 | 2 | 8,597 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 518.3 | 260.0 | 48 ms | 3.8 ms | 2.0 | 4 | 15,550 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,012.6 | 254.6 | 71 ms | 3.9 ms | 3.9 | 8 | 30,376 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,895.8 | 238.8 | 105 ms | 4.2 ms | 8.0 | 16 | 56,826 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 281.4 | 282.2 | 44 ms | 3.5 ms | 1.0 | 2 | 8,443 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 508.3 | 255.2 | 51 ms | 3.9 ms | 2.0 | 4 | 15,248 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 1,846.4 | 232.6 | 142 ms | 4.3 ms | 8.0 | 8 | 55,392 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Valid | 494.2 | 248.8 | 91 ms | 4.0 ms | 2.0 | 4 | 14,827 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 957.7 | 241.9 | 129 ms | 4.1 ms | 3.8 | 8 | 28,728 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,777.1 | 224.2 | 236 ms | 4.5 ms | 7.8 | 8 | 53,313 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Valid | 6,493.2 | 102.0 | 1,373 ms | 9.8 ms | 64.0 | 0 | 196,416 | 30.2 s |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Valid | 252.0 | 254.1 | 116 ms | 3.9 ms | 1.0 | 2 | 7,561 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 465.1 | 233.9 | 157 ms | 4.3 ms | 2.0 | 2 | 13,951 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 905.3 | 228.0 | 184 ms | 4.4 ms | 4.0 | 4 | 27,224 | 30.1 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 2,581.2 | 163.4 | 432 ms | 6.1 ms | 15.8 | 16 | 77,436 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,703.7 | 116.5 | 1,235 ms | 8.6 ms | 31.9 | 7 | 111,613 | 30.1 s |
| Sustained decode | 64K | 64 | Valid | 4,809.6 | 75.8 | 2,437 ms | 13.2 ms | 64.0 | 0 | 144,288 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 220.5 | 222.6 | 244 ms | 4.5 ms | 1.0 | 1 | 6,615 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 408.3 | 206.5 | 336 ms | 4.8 ms | 2.0 | 2 | 12,249 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 797.1 | 201.4 | 359 ms | 5.0 ms | 4.0 | 4 | 23,914 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,360.0 | 172.3 | 478 ms | 5.8 ms | 7.9 | 8 | 40,801 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,007.9 | 127.8 | 502 ms | 7.8 ms | 15.7 | 16 | 60,236 | 30.0 s |
| Sustained decode | 128K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 176.8 | 180.4 | 525 ms | 5.5 ms | 1.0 | 1 | 5,303 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 330.6 | 168.8 | 626 ms | 5.9 ms | 2.0 | 2 | 9,917 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 646.3 | 165.0 | 676 ms | 6.1 ms | 3.9 | 4 | 19,388 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,044.3 | 133.3 | 680 ms | 7.5 ms | 7.9 | 8 | 31,327 | 30.0 s |
| Sustained decode | 251K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.