Speed test matrix
Qwen3.8-27B on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context8,560962 ms TTFT
16K context8,5911,893 ms TTFT
32K context8,2363,926 ms TTFT
64K context7,6508,426 ms TTFT
128K context6,65219,354 ms TTFT
251K context5,33347,298 ms TTFT
256K context5,27748,751 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 47.289 ms TTFT | 85.9137 ms TTFT | 168.6271 ms TTFT | 323.4450 ms TTFT | 625.2552 ms TTFT | 1,078.61,618 ms TTFT | 1,761.51,689 ms TTFT | 2,447.83,542 ms TTFT |
| 16K | 46.9120 ms TTFT | 85.0193 ms TTFT | 166.6372 ms TTFT | 317.4586 ms TTFT | 603.1975 ms TTFT | 1,014.01,656 ms TTFT | 1,595.82,622 ms TTFT | Over capacity |
| 32K | 45.9117 ms TTFT | 83.5174 ms TTFT | 162.8235 ms TTFT | 305.4368 ms TTFT | 562.6557 ms TTFT | 904.81,038 ms TTFT | Over capacity | Over capacity |
| 64K | 44.2189 ms TTFT | 80.4253 ms TTFT | 155.5348 ms TTFT | 284.6545 ms TTFT | 496.0836 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 41.5355 ms TTFT | 71.4452 ms TTFT | 138.6649 ms TTFT | 250.4934 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 37.1733 ms TTFT | 60.1924 ms TTFT | 116.41,395 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,560.0 | — | 962 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,591.0 | — | 1,893 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,236.0 | — | 3,926 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,650.0 | — | 8,426 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,652.0 | — | 19,354 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,333.0 | — | 47,298 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,277.0 | — | 48,751 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 47.2 | 47.2 | 89 ms | 21.2 ms | 1.0 | 0 | 1,416 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 85.9 | 42.9 | 137 ms | 23.3 ms | 2.0 | 0 | 2,576 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 168.6 | 42.2 | 271 ms | 23.7 ms | 4.0 | 0 | 5,056 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 323.4 | 40.5 | 450 ms | 24.7 ms | 8.0 | 0 | 9,696 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 625.2 | 38.8 | 552 ms | 25.8 ms | 16.0 | 0 | 18,752 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,078.6 | 33.8 | 1,618 ms | 29.6 ms | 32.0 | 0 | 32,352 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,761.5 | 26.5 | 1,689 ms | 37.7 ms | 64.0 | 0 | 52,800 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,447.8 | 17.9 | 3,542 ms | 56.0 ms | 128.0 | 0 | 73,344 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 46.9 | 46.8 | 120 ms | 21.4 ms | 1.0 | 0 | 1,405 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 85.0 | 42.4 | 193 ms | 23.6 ms | 2.0 | 0 | 2,548 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 166.6 | 41.6 | 372 ms | 24.0 ms | 4.0 | 0 | 4,996 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 317.4 | 39.6 | 586 ms | 25.3 ms | 8.0 | 0 | 9,520 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 603.1 | 37.4 | 975 ms | 26.7 ms | 16.0 | 0 | 18,080 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,014.0 | 31.0 | 1,656 ms | 32.2 ms | 32.0 | 0 | 30,400 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,595.8 | 23.5 | 2,622 ms | 42.5 ms | 64.0 | 0 | 47,872 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 45.9 | 45.9 | 117 ms | 21.8 ms | 1.0 | 0 | 1,376 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 83.5 | 41.7 | 174 ms | 24.0 ms | 2.0 | 0 | 2,504 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 162.8 | 40.7 | 235 ms | 24.6 ms | 4.0 | 0 | 4,880 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 305.4 | 38.2 | 368 ms | 26.2 ms | 8.0 | 0 | 9,163 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 562.6 | 35.0 | 557 ms | 28.6 ms | 16.0 | 0 | 16,864 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 904.8 | 28.0 | 1,038 ms | 35.7 ms | 32.0 | 0 | 27,145 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 44.2 | 44.2 | 189 ms | 22.6 ms | 1.0 | 0 | 1,327 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 80.4 | 40.2 | 253 ms | 24.9 ms | 2.0 | 0 | 2,412 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 155.5 | 38.8 | 348 ms | 25.8 ms | 4.0 | 0 | 4,664 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 284.6 | 35.5 | 545 ms | 28.2 ms | 8.0 | 0 | 8,537 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 496.0 | 30.7 | 836 ms | 32.6 ms | 16.0 | 0 | 14,880 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 41.5 | 41.5 | 355 ms | 24.1 ms | 1.0 | 0 | 1,244 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 71.4 | 35.7 | 452 ms | 28.0 ms | 2.0 | 0 | 2,140 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 138.6 | 34.6 | 649 ms | 28.9 ms | 4.0 | 0 | 4,156 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 250.4 | 31.0 | 934 ms | 32.2 ms | 8.0 | 0 | 7,504 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 37.1 | 37.2 | 733 ms | 26.9 ms | 1.0 | 0 | 1,113 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 60.1 | 30.0 | 924 ms | 33.3 ms | 2.0 | 0 | 1,800 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 116.4 | 28.9 | 1,395 ms | 34.6 ms | 4.0 | 0 | 3,492 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.