Speed test matrix
Qwen3.5-27B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-z-lab-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.5-27B-DFlash","num_speculative_tokens":7}'Prefill
prompt tok/s by context length8K context6,6591,230 ms TTFT
16K context6,4242,525 ms TTFT
32K context6,0035,379 ms TTFT
64K context5,39811,935 ms TTFT
128K context4,44328,966 ms TTFT
251K context3,29976,458 ms TTFT
256K context3,25878,952 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 70.9256 ms TTFT | 144.1507 ms TTFT | 262.71,506 ms TTFT | 432.71,964 ms TTFT | 450.1Requests queued2,928 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 61.2249 ms TTFT | 131.4434 ms TTFT | 234.9871 ms TTFT | 383.71,647 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 53.8348 ms TTFT | 114.7612 ms TTFT | 202.81,198 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 42.3408 ms TTFT | 77.7658 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 24.8647 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 14.51,001 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,659.0 | — | 1,230 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,424.0 | — | 2,525 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,003.0 | — | 5,379 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,398.0 | — | 11,935 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,443.0 | — | 28,966 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,299.0 | — | 76,458 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,258.0 | — | 78,952 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 70.9 | 72.5 | 256 ms | 13.8 ms | 1.0 | 0 | 2,127 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 144.1 | 73.0 | 507 ms | 13.7 ms | 2.0 | 0 | 4,319 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 262.7 | 67.5 | 1,506 ms | 14.8 ms | 4.0 | 0 | 7,872 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 432.7 | 55.5 | 1,964 ms | 18.0 ms | 8.0 | 0 | 12,970 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 450.1 | 46.8 | 2,928 ms | 21.4 ms | 10.3 | 6 | 13,445 | 29.9 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 61.2 | 64.0 | 249 ms | 15.6 ms | 1.0 | 0 | 1,834 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 131.4 | 66.6 | 434 ms | 15.0 ms | 2.0 | 0 | 3,939 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 234.9 | 60.3 | 871 ms | 16.6 ms | 4.0 | 0 | 7,046 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 383.7 | 49.1 | 1,647 ms | 20.4 ms | 8.0 | 0 | 11,494 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 53.8 | 55.4 | 348 ms | 18.0 ms | 1.0 | 0 | 1,612 | 29.9 s |
| Sustained decode | 32K | 2 | Valid | 114.7 | 57.7 | 612 ms | 17.3 ms | 2.0 | 0 | 3,433 | 29.9 s |
| Sustained decode | 32K | 4 | Valid | 202.8 | 52.3 | 1,198 ms | 19.1 ms | 4.0 | 0 | 6,074 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 42.3 | 43.8 | 408 ms | 22.8 ms | 1.0 | 0 | 1,268 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 77.7 | 39.9 | 658 ms | 25.1 ms | 2.0 | 0 | 2,327 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 24.8 | 26.6 | 647 ms | 37.5 ms | 1.0 | 0 | 742 | 29.9 s |
| Sustained decode | 128K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 14.5 | 15.8 | 1,001 ms | 63.2 ms | 1.0 | 0 | 434 | 29.9 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.