Speed test matrix
Qwen3.8-27B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev172+ge55909cad · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev172+ge55909cad
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e55909cad00f
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-z-lab-vllm-pr-e55909cad00f/bin/vllm" serve Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --min-kv-cache-group-layers 1 --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context6,5161,264 ms TTFT
16K context6,4222,534 ms TTFT
32K context6,0315,363 ms TTFT
64K context5,42611,883 ms TTFT
128K context4,45928,874 ms TTFT
251K context3,30276,405 ms TTFT
256K context3,26678,777 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 47.0294 ms TTFT | 95.61,589 ms TTFT | 171.22,641 ms TTFT | 293.42,312 ms TTFT | 393.1Requests queued3,161 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 44.2271 ms TTFT | 86.0470 ms TTFT | 157.7933 ms TTFT | 255.41,725 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 40.8386 ms TTFT | 75.6653 ms TTFT | 138.31,262 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 28.6426 ms TTFT | 55.1688 ms TTFT | 114.61,232 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 18.0685 ms TTFT | 41.01,116 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 12.81,094 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,516.0 | — | 1,264 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,422.0 | — | 2,534 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,031.0 | — | 5,363 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,426.0 | — | 11,883 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,459.0 | — | 28,874 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,302.0 | — | 76,405 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,266.0 | — | 78,777 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 47.0 | 49.2 | 294 ms | 20.3 ms | 1.0 | 0 | 1,410 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 95.6 | 47.7 | 1,589 ms | 21.0 ms | 2.0 | 0 | 2,866 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 171.2 | 44.3 | 2,641 ms | 22.6 ms | 4.0 | 0 | 5,133 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 293.4 | 38.0 | 2,312 ms | 26.3 ms | 8.0 | 0 | 8,790 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 393.1 | 31.9 | 3,161 ms | 31.3 ms | 12.0 | 0 | 11,777 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 44.2 | 46.1 | 271 ms | 21.7 ms | 1.0 | 0 | 1,324 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 86.0 | 45.0 | 470 ms | 22.2 ms | 2.0 | 0 | 2,579 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 157.7 | 41.3 | 933 ms | 24.2 ms | 4.0 | 0 | 4,730 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 255.4 | 33.4 | 1,725 ms | 29.9 ms | 8.0 | 0 | 7,650 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 40.8 | 42.2 | 386 ms | 23.7 ms | 1.0 | 0 | 1,222 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 75.6 | 39.2 | 653 ms | 25.5 ms | 2.0 | 0 | 2,265 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 138.3 | 35.8 | 1,262 ms | 27.9 ms | 4.0 | 0 | 4,141 | 29.9 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 28.6 | 29.9 | 426 ms | 33.4 ms | 1.0 | 0 | 857 | 29.9 s |
| Sustained decode | 64K | 2 | Valid | 55.1 | 29.1 | 688 ms | 34.4 ms | 2.0 | 0 | 1,650 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 114.6 | 30.0 | 1,232 ms | 33.3 ms | 4.0 | 0 | 3,434 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 18.0 | 19.7 | 685 ms | 50.7 ms | 1.0 | 0 | 541 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 41.0 | 21.9 | 1,116 ms | 45.6 ms | 2.0 | 0 | 1,230 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 12.8 | 14.1 | 1,094 ms | 70.7 ms | 1.0 | 0 | 383 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.