Speed test matrix
Qwen3.5-4B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-z-lab-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-4B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.5-4B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context32.7K250 ms TTFT
16K context32.9K493 ms TTFT
32K context29.9K1,079 ms TTFT
64K context24.9K2,588 ms TTFT
128K context18.6K6,923 ms TTFT
251K context12.4K20,344 ms TTFT
256K context12.3K20,999 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 203.956 ms TTFT | 400.881 ms TTFT | 754.186 ms TTFT | 1,327.392 ms TTFT | 1,829.9305 ms TTFT | 2,433.6657 ms TTFT | 2,692.11,064 ms TTFT | 2,777.1Requests queued2,079 ms TTFT |
| 16K | 176.873 ms TTFT | 330.8103 ms TTFT | 670.5101 ms TTFT | 1,160.7127 ms TTFT | 1,504.0520 ms TTFT | 2,038.5869 ms TTFT | 2,185.11,525 ms TTFT | Over capacity |
| 32K | 133.3105 ms TTFT | 258.0133 ms TTFT | 511.8130 ms TTFT | 864.2332 ms TTFT | 1,152.7529 ms TTFT | 1,480.5922 ms TTFT | Over capacity | Over capacity |
| 64K | 87.9147 ms TTFT | 173.2230 ms TTFT | 343.3317 ms TTFT | 613.1485 ms TTFT | 742.4827 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 53.9332 ms TTFT | 100.3445 ms TTFT | 202.9608 ms TTFT | 382.4956 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 28.4654 ms TTFT | 62.4911 ms TTFT | 117.61,308 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32.7K | — | 250 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.9K | — | 493 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.9K | — | 1,079 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 24.9K | — | 2,588 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.6K | — | 6,923 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.4K | — | 20,344 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.3K | — | 20,999 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 203.9 | 208.2 | 56 ms | 4.8 ms | 1.0 | 1 | 6,116 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 400.8 | 207.8 | 81 ms | 4.8 ms | 2.0 | 2 | 12,023 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 754.1 | 190.8 | 86 ms | 5.2 ms | 4.0 | 4 | 22,622 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,327.3 | 169.8 | 92 ms | 5.9 ms | 8.0 | 8 | 39,800 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,829.9 | 118.6 | 305 ms | 8.4 ms | 16.0 | 4 | 54,886 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,433.6 | 77.6 | 657 ms | 12.9 ms | 32.0 | 0 | 72,933 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,692.1 | 43.5 | 1,064 ms | 23.0 ms | 64.0 | 0 | 80,687 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 2,777.1 | 23.1 | 2,079 ms | 43.3 ms | 127.4 | 0 | 83,240 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 176.8 | 183.3 | 73 ms | 5.5 ms | 1.0 | 1 | 5,300 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 330.8 | 167.8 | 103 ms | 6.0 ms | 2.0 | 2 | 9,918 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 670.5 | 169.6 | 101 ms | 5.9 ms | 4.0 | 4 | 20,112 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,160.7 | 152.4 | 127 ms | 6.6 ms | 8.0 | 8 | 34,810 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,504.0 | 95.9 | 520 ms | 10.4 ms | 16.0 | 0 | 45,088 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,038.5 | 65.5 | 869 ms | 15.3 ms | 32.0 | 0 | 61,135 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,185.1 | 35.6 | 1,525 ms | 28.1 ms | 64.0 | 0 | 65,411 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 133.3 | 142.0 | 105 ms | 7.1 ms | 1.0 | 1 | 3,999 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 258.0 | 145.9 | 133 ms | 6.9 ms | 2.0 | 2 | 7,739 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 511.8 | 136.1 | 130 ms | 7.4 ms | 4.0 | 4 | 15,352 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 864.2 | 110.5 | 332 ms | 9.1 ms | 8.0 | 0 | 25,925 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,152.7 | 73.3 | 529 ms | 13.6 ms | 16.0 | 0 | 34,566 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,480.5 | 47.8 | 922 ms | 20.9 ms | 32.0 | 0 | 44,397 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 87.9 | 89.0 | 147 ms | 11.2 ms | 1.0 | 0 | 2,634 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 173.2 | 88.2 | 230 ms | 11.3 ms | 2.0 | 0 | 5,194 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 343.3 | 87.7 | 317 ms | 11.4 ms | 4.0 | 0 | 10,289 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 613.1 | 78.8 | 485 ms | 12.7 ms | 8.0 | 0 | 18,375 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 742.4 | 47.9 | 827 ms | 20.9 ms | 16.0 | 0 | 22,236 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 53.9 | 56.3 | 332 ms | 17.7 ms | 1.0 | 0 | 1,614 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 100.3 | 51.4 | 445 ms | 19.5 ms | 2.0 | 0 | 3,006 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 202.9 | 51.8 | 608 ms | 19.4 ms | 4.0 | 0 | 6,079 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 382.4 | 49.7 | 956 ms | 20.1 ms | 8.0 | 0 | 11,464 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 28.4 | 29.7 | 654 ms | 33.7 ms | 1.0 | 0 | 853 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 62.4 | 32.8 | 911 ms | 30.6 ms | 2.0 | 0 | 1,869 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 117.6 | 30.3 | 1,308 ms | 33.0 ms | 4.0 | 0 | 3,527 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.