Speed test matrix
Qwen3.5-4B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-4B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER
Prefill
prompt tok/s by context length8K context33.9K242 ms TTFT
16K context33.9K478 ms TTFT
32K context30.6K1,055 ms TTFT
64K context25.5K2,525 ms TTFT
128K context18.8K6,861 ms TTFT
251K context12.4K20,269 ms TTFT
256K context12.3K20,921 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 140.951 ms TTFT | 257.958 ms TTFT | 495.3115 ms TTFT | 916.0120 ms TTFT | 1,623.7164 ms TTFT | 2,618.4240 ms TTFT | 3,852.3476 ms TTFT | 5,354.8806 ms TTFT |
| 16K | 137.552 ms TTFT | 252.681 ms TTFT | 483.094 ms TTFT | 889.3160 ms TTFT | 1,526.3233 ms TTFT | 2,390.5405 ms TTFT | 3,345.2711 ms TTFT | Over capacity |
| 32K | 131.270 ms TTFT | 241.884 ms TTFT | 466.2161 ms TTFT | 842.5215 ms TTFT | 1,378.2366 ms TTFT | 2,046.7664 ms TTFT | Over capacity | Over capacity |
| 64K | 119.5120 ms TTFT | 222.2185 ms TTFT | 429.5264 ms TTFT | 744.8436 ms TTFT | 1,139.4764 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 101.1317 ms TTFT | 188.3389 ms TTFT | 364.5571 ms TTFT | 596.9949 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 77.8553 ms TTFT | 130.6793 ms TTFT | 247.71,190 ms TTFT | 426.32,033 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.9K | — | 242 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.9K | — | 478 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.6K | — | 1,055 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.5K | — | 2,525 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.8K | — | 6,861 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 12.4K | — | 20,269 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.3K | — | 20,921 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 140.9 | 141.7 | 51 ms | 7.1 ms | 1.0 | 1 | 4,228 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 257.9 | 129.9 | 58 ms | 7.7 ms | 2.0 | 2 | 7,736 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 495.3 | 126.2 | 115 ms | 7.9 ms | 4.0 | 4 | 14,856 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 916.0 | 115.2 | 120 ms | 8.7 ms | 8.0 | 0 | 27,481 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,623.7 | 102.4 | 164 ms | 9.8 ms | 16.0 | 0 | 48,704 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,618.4 | 82.6 | 240 ms | 12.1 ms | 32.0 | 0 | 78,528 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,852.3 | 60.5 | 476 ms | 16.5 ms | 64.0 | 0 | 115,520 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,354.8 | 41.6 | 806 ms | 24.0 ms | 128.0 | 0 | 160,640 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 137.5 | 138.1 | 52 ms | 7.2 ms | 1.0 | 1 | 4,124 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 252.6 | 127.4 | 81 ms | 7.8 ms | 2.0 | 2 | 7,576 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 483.0 | 121.4 | 94 ms | 8.2 ms | 4.0 | 4 | 14,488 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 889.3 | 111.7 | 160 ms | 9.0 ms | 8.0 | 0 | 26,672 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,526.3 | 95.9 | 233 ms | 10.4 ms | 16.0 | 0 | 45,776 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,390.5 | 75.0 | 405 ms | 13.3 ms | 32.0 | 0 | 71,712 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,345.2 | 52.1 | 711 ms | 19.2 ms | 64.0 | 0 | 100,357 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 131.2 | 132.1 | 70 ms | 7.6 ms | 1.0 | 1 | 3,935 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 241.8 | 122.3 | 84 ms | 8.2 ms | 2.0 | 2 | 7,254 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 466.2 | 116.8 | 161 ms | 8.6 ms | 4.0 | 0 | 13,984 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 842.5 | 105.5 | 215 ms | 9.5 ms | 8.0 | 0 | 25,272 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,378.2 | 86.5 | 366 ms | 11.6 ms | 16.0 | 0 | 41,344 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,046.7 | 64.1 | 664 ms | 15.6 ms | 32.0 | 0 | 61,376 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 119.5 | 119.6 | 120 ms | 8.4 ms | 1.0 | 0 | 3,584 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 222.2 | 111.3 | 185 ms | 9.0 ms | 2.0 | 0 | 6,666 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 429.5 | 107.5 | 264 ms | 9.3 ms | 4.0 | 0 | 12,884 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 744.8 | 93.1 | 436 ms | 10.7 ms | 8.0 | 0 | 22,344 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,139.4 | 71.4 | 764 ms | 14.0 ms | 16.0 | 0 | 34,176 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 101.1 | 101.2 | 317 ms | 9.9 ms | 1.0 | 0 | 3,032 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 188.3 | 94.2 | 389 ms | 10.6 ms | 2.0 | 0 | 5,646 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 364.5 | 91.1 | 571 ms | 11.0 ms | 4.0 | 0 | 10,932 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 596.9 | 74.4 | 949 ms | 13.4 ms | 8.0 | 0 | 17,904 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 77.8 | 77.9 | 553 ms | 12.8 ms | 1.0 | 0 | 2,333 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 130.6 | 65.4 | 793 ms | 15.3 ms | 2.0 | 0 | 3,918 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 247.7 | 61.9 | 1,190 ms | 16.1 ms | 4.0 | 0 | 7,428 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 426.3 | 53.2 | 2,033 ms | 18.8 ms | 8.0 | 0 | 12,784 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.