Speed test matrix
Qwen3.5-4B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-4B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.93 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER
Prefill
prompt tok/s by context length8K context34.5K237 ms TTFT
16K context34.1K475 ms TTFT
32K context30.6K1,055 ms TTFT
64K context25.5K2,529 ms TTFT
128K context18.8K6,861 ms TTFT
251K context12.5K20,230 ms TTFT
256K context12.3K20,869 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 141.237 ms TTFT | 257.857 ms TTFT | 495.372 ms TTFT | 916.5118 ms TTFT | 1,623.2159 ms TTFT | 2,620.1251 ms TTFT | 3,873.0475 ms TTFT | 5,403.3875 ms TTFT |
| 16K | 137.552 ms TTFT | 252.582 ms TTFT | Looped | 888.6169 ms TTFT | 1,525.8239 ms TTFT | 2,389.0406 ms TTFT | 3,341.0714 ms TTFT | Over capacity |
| 32K | 131.272 ms TTFT | 241.6111 ms TTFT | 465.9137 ms TTFT | 842.0229 ms TTFT | 1,378.3371 ms TTFT | 2,044.8694 ms TTFT | Over capacity | Over capacity |
| 64K | 119.5123 ms TTFT | 222.5196 ms TTFT | Looped | 745.5396 ms TTFT | 1,138.2697 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 101.2268 ms TTFT | 188.0400 ms TTFT | 366.7537 ms TTFT | 598.3884 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 78.0557 ms TTFT | Looped | 254.31,215 ms TTFT | 427.41,900 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.5K | — | 237 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.1K | — | 475 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.6K | — | 1,055 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.5K | — | 2,529 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.8K | — | 6,861 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.5K | — | 20,230 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.3K | — | 20,869 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 141.2 | 141.8 | 37 ms | 7.1 ms | 1.0 | 1 | 4,235 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 257.8 | 129.9 | 57 ms | 7.7 ms | 2.0 | 2 | 7,732 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 495.3 | 125.7 | 72 ms | 8.0 ms | 4.0 | 4 | 14,856 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 916.5 | 115.3 | 118 ms | 8.7 ms | 8.0 | 0 | 27,496 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,623.2 | 102.3 | 159 ms | 9.8 ms | 16.0 | 0 | 48,688 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,620.1 | 82.7 | 251 ms | 12.1 ms | 32.0 | 0 | 78,592 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,873.0 | 60.8 | 475 ms | 16.4 ms | 64.0 | 0 | 116,161 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,403.3 | 42.0 | 875 ms | 23.8 ms | 128.0 | 0 | 162,047 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 137.5 | 138.1 | 52 ms | 7.2 ms | 1.0 | 1 | 4,125 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 252.5 | 127.4 | 82 ms | 7.8 ms | 2.0 | 2 | 7,575 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 888.6 | 111.6 | 169 ms | 9.0 ms | 8.0 | 0 | 26,656 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,525.8 | 95.9 | 239 ms | 10.4 ms | 16.0 | 0 | 45,760 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,389.0 | 75.0 | 406 ms | 13.3 ms | 32.0 | 0 | 71,648 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,341.0 | 52.0 | 714 ms | 19.2 ms | 64.0 | 0 | 100,224 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 131.2 | 132.2 | 72 ms | 7.6 ms | 1.0 | 1 | 3,937 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 241.6 | 121.9 | 111 ms | 8.2 ms | 2.0 | 2 | 7,245 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 465.9 | 116.7 | 137 ms | 8.6 ms | 4.0 | 0 | 13,976 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 842.0 | 105.5 | 229 ms | 9.5 ms | 8.0 | 0 | 25,256 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,378.3 | 86.5 | 371 ms | 11.6 ms | 16.0 | 0 | 41,344 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,044.8 | 64.1 | 694 ms | 15.6 ms | 32.0 | 0 | 61,345 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 119.5 | 119.6 | 123 ms | 8.4 ms | 1.0 | 0 | 3,586 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 222.5 | 111.4 | 196 ms | 9.0 ms | 2.0 | 0 | 6,674 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 745.5 | 93.2 | 396 ms | 10.7 ms | 8.0 | 0 | 22,360 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,138.2 | 71.4 | 697 ms | 14.0 ms | 16.0 | 0 | 34,161 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 101.2 | 101.3 | 268 ms | 9.9 ms | 1.0 | 0 | 3,035 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 188.0 | 94.1 | 400 ms | 10.6 ms | 2.0 | 0 | 5,638 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 366.7 | 91.6 | 537 ms | 10.9 ms | 4.0 | 0 | 11,000 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 598.3 | 74.6 | 884 ms | 13.4 ms | 8.0 | 0 | 17,944 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 78.0 | 78.1 | 557 ms | 12.8 ms | 1.0 | 0 | 2,338 | 30.0 s |
| Sustained decode | 251K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 4 | Valid | 254.3 | 63.5 | 1,215 ms | 15.7 ms | 4.0 | 0 | 7,624 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 427.4 | 53.3 | 1,900 ms | 18.8 ms | 8.0 | 0 | 12,816 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.