Speed test matrix
Qwen3.5-4B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-4B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context33K248 ms TTFT
16K context32.5K500 ms TTFT
32K context29.3K1,101 ms TTFT
64K context23.9K2,695 ms TTFT
128K context17.4K7,384 ms TTFT
251K context11.5K21,991 ms TTFT
256K context11.3K22,745 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 206.846 ms TTFT | 415.272 ms TTFT | 786.677 ms TTFT | 1,465.296 ms TTFT | 2,450.3116 ms TTFT | 3,555.0504 ms TTFT | 4,923.0874 ms TTFT | 6,224.81,450 ms TTFT |
| 16K | 203.672 ms TTFT | 408.4114 ms TTFT | 773.8158 ms TTFT | 1,407.9130 ms TTFT | 2,267.0130 ms TTFT | 3,266.7797 ms TTFT | 4,331.91,387 ms TTFT | Over capacity |
| 32K | 193.489 ms TTFT | 392.5108 ms TTFT | 756.6123 ms TTFT | 1,338.9147 ms TTFT | 2,022.1264 ms TTFT | 2,871.1809 ms TTFT | Over capacity | Over capacity |
| 64K | 172.7170 ms TTFT | 360.7197 ms TTFT | 693.1195 ms TTFT | 1,184.5236 ms TTFT | 1,723.5837 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 157.8357 ms TTFT | 299.5356 ms TTFT | 582.3378 ms TTFT | 962.2585 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 118.8654 ms TTFT | 213.9878 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33K | — | 248 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.5K | — | 500 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.3K | — | 1,101 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 23.9K | — | 2,695 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.4K | — | 7,384 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 11.5K | — | 21,991 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.3K | — | 22,745 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 206.8 | 209.1 | 46 ms | 4.8 ms | 1.0 | 1 | 6,203 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 415.2 | 209.7 | 72 ms | 4.8 ms | 2.0 | 2 | 12,456 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 786.6 | 198.7 | 77 ms | 5.0 ms | 4.0 | 4 | 23,596 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,465.2 | 185.7 | 96 ms | 5.4 ms | 8.0 | 8 | 43,956 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,450.3 | 157.7 | 116 ms | 6.3 ms | 16.0 | 16 | 73,511 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,555.0 | 113.3 | 504 ms | 8.8 ms | 32.0 | 0 | 106,624 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,923.0 | 78.2 | 874 ms | 12.8 ms | 64.0 | 0 | 147,583 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,224.8 | 48.8 | 1,450 ms | 20.5 ms | 128.0 | 0 | 186,584 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 203.6 | 206.6 | 72 ms | 4.8 ms | 1.0 | 1 | 6,108 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 408.4 | 206.2 | 114 ms | 4.8 ms | 2.0 | 2 | 12,250 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 773.8 | 197.3 | 158 ms | 5.1 ms | 4.0 | 4 | 23,207 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,407.9 | 178.4 | 130 ms | 5.6 ms | 7.9 | 8 | 42,227 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,267.0 | 145.3 | 130 ms | 6.9 ms | 15.9 | 16 | 67,986 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,266.7 | 103.5 | 797 ms | 9.7 ms | 32.0 | 0 | 97,980 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,331.9 | 67.4 | 1,387 ms | 14.8 ms | 64.0 | 0 | 129,869 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 193.4 | 195.8 | 89 ms | 5.1 ms | 1.0 | 1 | 5,803 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 392.5 | 197.4 | 108 ms | 5.1 ms | 2.0 | 2 | 11,776 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 756.6 | 190.6 | 123 ms | 5.2 ms | 4.0 | 4 | 22,694 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,338.9 | 171.5 | 147 ms | 5.8 ms | 8.0 | 8 | 40,168 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,022.1 | 128.8 | 264 ms | 7.8 ms | 16.0 | 16 | 60,654 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,871.1 | 90.3 | 809 ms | 11.1 ms | 32.0 | 0 | 86,096 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 172.7 | 176.5 | 170 ms | 5.7 ms | 1.0 | 1 | 5,179 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 360.7 | 183.4 | 197 ms | 5.5 ms | 2.0 | 2 | 10,820 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 693.1 | 176.7 | 195 ms | 5.7 ms | 4.0 | 4 | 20,793 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,184.5 | 151.7 | 236 ms | 6.6 ms | 8.0 | 8 | 35,525 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,723.5 | 108.0 | 837 ms | 9.3 ms | 16.0 | 0 | 51,680 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 157.8 | 162.3 | 357 ms | 6.2 ms | 1.0 | 1 | 4,734 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 299.5 | 155.6 | 356 ms | 6.4 ms | 2.0 | 2 | 8,985 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 582.3 | 149.8 | 378 ms | 6.7 ms | 4.0 | 4 | 17,465 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 962.2 | 121.8 | 585 ms | 8.2 ms | 7.9 | 8 | 28,865 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 118.8 | 120.9 | 654 ms | 8.3 ms | 1.0 | 1 | 3,561 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 213.9 | 107.9 | 878 ms | 9.3 ms | 2.0 | 0 | 6,416 | 30.0 s |
| Sustained decode | 251K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.