Speed test matrix
Qwen3.8-27B on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.8-27B --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --speculative-config '{"method":"mtp","num_speculative_tokens":5}'Prefill
prompt tok/s by context length8K context8,440976 ms TTFT
16K context8,4201,932 ms TTFT
32K context8,1213,982 ms TTFT
64K context7,4858,613 ms TTFT
128K context6,40320,107 ms TTFT
251K context5,02050,245 ms TTFT
256K context4,94851,993 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 72.1139 ms TTFT | 135.3224 ms TTFT | 245.0500 ms TTFT | 470.0966 ms TTFT | 705.91,115 ms TTFT | 1,019.62,078 ms TTFT | 1,311.74,036 ms TTFT | 1,322.3Requests queued5,999 ms TTFT |
| 16K | 56.3236 ms TTFT | 123.9364 ms TTFT | 205.2868 ms TTFT | 406.71,242 ms TTFT | 580.62,214 ms TTFT | 860.53,691 ms TTFT | 1,070.27,213 ms TTFT | Over capacity |
| 32K | 46.8250 ms TTFT | 95.7399 ms TTFT | 186.7863 ms TTFT | 341.71,378 ms TTFT | 441.81,779 ms TTFT | 620.83,971 ms TTFT | Over capacity | Over capacity |
| 64K | 31.88,640 ms TTFT | 68.38,932 ms TTFT | 133.21,109 ms TTFT | 248.51,898 ms TTFT | 294.33,318 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 23.4547 ms TTFT | 41.9795 ms TTFT | 79.31,279 ms TTFT | 164.32,085 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 14.0760 ms TTFT | 25.41,188 ms TTFT | 49.22,047 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,440.0 | — | 976 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,420.0 | — | 1,932 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,121.0 | — | 3,982 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,485.0 | — | 8,613 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,403.0 | — | 20,107 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,020.0 | — | 50,245 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,948.0 | — | 51,993 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 72.1 | 73.3 | 139 ms | 13.6 ms | 1.0 | 0 | 2,162 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 135.3 | 68.1 | 224 ms | 14.7 ms | 2.0 | 0 | 4,055 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 245.0 | 62.1 | 500 ms | 16.1 ms | 4.0 | 0 | 7,344 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 470.0 | 59.5 | 966 ms | 16.8 ms | 8.0 | 0 | 14,088 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 705.9 | 45.0 | 1,115 ms | 22.2 ms | 16.0 | 0 | 21,171 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,019.6 | 31.9 | 2,078 ms | 31.4 ms | 32.0 | 0 | 30,540 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,311.7 | 20.2 | 4,036 ms | 49.6 ms | 64.0 | 0 | 39,288 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,322.3 | 13.2 | 5,999 ms | 76.0 ms | 99.2 | 0 | 39,584 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 56.3 | 58.1 | 236 ms | 17.2 ms | 1.0 | 0 | 1,690 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 123.9 | 62.7 | 364 ms | 15.9 ms | 2.0 | 0 | 3,713 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 205.2 | 52.3 | 868 ms | 19.1 ms | 4.0 | 0 | 6,150 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 406.7 | 52.0 | 1,242 ms | 19.2 ms | 8.0 | 0 | 12,182 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 580.6 | 36.4 | 2,214 ms | 27.5 ms | 16.0 | 0 | 17,401 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 860.5 | 26.3 | 3,691 ms | 38.1 ms | 32.0 | 0 | 25,780 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,070.2 | 15.5 | 7,213 ms | 64.7 ms | 64.0 | 0 | 32,006 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 46.8 | 48.4 | 250 ms | 20.7 ms | 1.0 | 0 | 1,403 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 95.7 | 49.1 | 399 ms | 20.4 ms | 2.0 | 0 | 2,868 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 186.7 | 47.2 | 863 ms | 21.2 ms | 4.0 | 0 | 5,598 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 341.7 | 43.2 | 1,378 ms | 23.1 ms | 8.0 | 0 | 10,241 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 441.8 | 27.8 | 1,779 ms | 36.0 ms | 16.0 | 0 | 13,218 | 29.9 s |
| Sustained decode | 32K | 32 | Valid | 620.8 | 19.1 | 3,971 ms | 52.2 ms | 32.0 | 0 | 18,618 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 31.8 | 32.8 | 8,640 ms | 30.5 ms | 1.0 | 0 | 951 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 68.3 | 35.3 | 8,932 ms | 28.3 ms | 2.0 | 0 | 2,045 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 133.2 | 33.9 | 1,109 ms | 29.5 ms | 4.0 | 0 | 3,986 | 29.9 s |
| Sustained decode | 64K | 8 | Valid | 248.5 | 32.4 | 1,898 ms | 30.9 ms | 8.0 | 0 | 7,452 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 294.3 | 18.9 | 3,318 ms | 52.8 ms | 16.0 | 0 | 8,809 | 29.9 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 23.4 | 24.5 | 547 ms | 40.8 ms | 1.0 | 0 | 702 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 41.9 | 22.2 | 795 ms | 45.1 ms | 2.0 | 0 | 1,253 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 79.3 | 20.8 | 1,279 ms | 48.0 ms | 4.0 | 0 | 2,377 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 164.3 | 21.4 | 2,085 ms | 46.6 ms | 8.0 | 0 | 4,927 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 14.0 | 15.2 | 760 ms | 65.7 ms | 1.0 | 0 | 419 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 25.4 | 13.9 | 1,188 ms | 71.7 ms | 2.0 | 0 | 762 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 49.2 | 13.1 | 2,047 ms | 76.1 ms | 4.0 | 0 | 1,469 | 29.9 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.