Speed test matrix
Qwen3.5-0.8B-Base on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-0.8B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER
Prefill
prompt tok/s by context length8K context104.5K78 ms TTFT
16K context116.9K139 ms TTFT
32K context106.3K304 ms TTFT
64K context85.1K757 ms TTFT
128K context59K2,182 ms TTFT
251K context36.3K6,942 ms TTFT
256K context35.6K7,222 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 509.131 ms TTFT | 940.8Requests queued47 ms TTFT | Looped | 3,278.3104 ms TTFT | 5,376.5161 ms TTFT | 7,776.0273 ms TTFT | Looped | Looped |
| 16K | 490.545 ms TTFT | 919.166 ms TTFT | 1,764.296 ms TTFT | 3,113.7135 ms TTFT | 4,962.1213 ms TTFT | 6,986.7400 ms TTFT | 8,966.3729 ms TTFT | Looped |
| 32K | 458.766 ms TTFT | 872.2101 ms TTFT | Looped | 2,869.3202 ms TTFT | Looped | 6,026.0703 ms TTFT | 7,452.81,394 ms TTFT | Looped |
| 64K | Looped | 774.0173 ms TTFT | 1,475.2211 ms TTFT | 2,538.6318 ms TTFT | 3,707.4555 ms TTFT | 4,518.3957 ms TTFT | Looped | Over capacity |
| 128K | 330.2233 ms TTFT | 634.7314 ms TTFT | 1,219.0412 ms TTFT | 1,944.1506 ms TTFT | Looped | Looped | Over capacity | Over capacity |
| 251K | 242.6510 ms TTFT | 469.0629 ms TTFT | 901.2782 ms TTFT | 1,380.0758 ms TTFT | 1,749.62,818 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 104.5K | — | 78 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 116.9K | — | 139 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 106.3K | — | 304 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 85.1K | — | 757 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 59K | — | 2,182 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 36.3K | — | 6,942 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 35.6K | — | 7,222 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 509.1 | 511.5 | 31 ms | 2.0 ms | 1.0 | 4 | 15,272 | 30.0 s |
| Sustained decode | 8K | 2 | Requests queued | 940.8 | 476.6 | 47 ms | 2.1 ms | 1.9 | 8 | 28,223 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Valid | 3,278.3 | 415.7 | 104 ms | 2.4 ms | 8.0 | 24 | 98,350 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 5,376.5 | 344.0 | 161 ms | 2.9 ms | 16.0 | 32 | 161,393 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 7,776.0 | 251.9 | 273 ms | 4.0 ms | 30.8 | 64 | 233,280 | 30.0 s |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 490.5 | 493.8 | 45 ms | 2.0 ms | 1.0 | 4 | 14,716 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 919.1 | 463.2 | 66 ms | 2.2 ms | 1.9 | 6 | 27,572 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,764.2 | 447.0 | 96 ms | 2.2 ms | 3.9 | 12 | 53,064 | 30.1 s |
| Sustained decode | 16K | 8 | Valid | 3,113.7 | 396.5 | 135 ms | 2.5 ms | 7.9 | 24 | 93,411 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 4,962.1 | 316.6 | 213 ms | 3.2 ms | 16.0 | 32 | 148,934 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 6,986.7 | 224.5 | 400 ms | 4.5 ms | 30.8 | 32 | 209,572 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 8,966.3 | 147.0 | 729 ms | 6.8 ms | 61.3 | 64 | 269,720 | 30.1 s |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 458.7 | 462.0 | 66 ms | 2.2 ms | 1.0 | 3 | 13,762 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 872.2 | 440.7 | 101 ms | 2.3 ms | 2.0 | 6 | 26,167 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 2,869.3 | 369.8 | 202 ms | 2.7 ms | 7.7 | 24 | 86,078 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 6,026.0 | 195.6 | 703 ms | 5.1 ms | 31.3 | 32 | 180,779 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 7,452.8 | 120.6 | 1,394 ms | 8.3 ms | 61.9 | 52 | 223,560 | 30.0 s |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 2 | Valid | 774.0 | 394.9 | 173 ms | 2.5 ms | 1.9 | 6 | 23,218 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,475.2 | 378.1 | 211 ms | 2.6 ms | 3.9 | 12 | 44,267 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,538.6 | 325.3 | 318 ms | 3.1 ms | 7.7 | 16 | 76,153 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,707.4 | 237.3 | 555 ms | 4.2 ms | 15.8 | 16 | 111,253 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,518.3 | 150.0 | 957 ms | 6.7 ms | 31.1 | 32 | 135,550 | 30.0 s |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 330.2 | 335.6 | 233 ms | 3.0 ms | 1.0 | 2 | 9,904 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 634.7 | 324.8 | 314 ms | 3.1 ms | 2.0 | 4 | 19,042 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,219.0 | 313.3 | 412 ms | 3.2 ms | 3.9 | 8 | 36,571 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,944.1 | 252.2 | 506 ms | 4.0 ms | 7.8 | 16 | 58,275 | 30.0 s |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 242.6 | 247.1 | 510 ms | 4.0 ms | 1.0 | 1 | 7,276 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 469.0 | 239.7 | 629 ms | 4.2 ms | 2.0 | 2 | 14,069 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 901.2 | 230.7 | 782 ms | 4.3 ms | 3.9 | 4 | 27,034 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,380.0 | 176.2 | 758 ms | 5.7 ms | 7.8 | 8 | 41,402 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 1,749.6 | 110.1 | 2,818 ms | 9.1 ms | 15.9 | 5 | 52,488 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.