Speed test matrix
Qwen3.5-27B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.92 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context9,849832 ms TTFT
16K context9,4551,716 ms TTFT
32K context8,5903,760 ms TTFT
64K context7,2408,898 ms TTFT
128K context5,50523,377 ms TTFT
251K context3,76866,934 ms TTFT
256K context3,73468,885 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 75.1128 ms TTFT | 150.6651 ms TTFT | 288.31,289 ms TTFT | 479.01,344 ms TTFT | 776.92,155 ms TTFT | 1,105.22,955 ms TTFT | 1,426.14,717 ms TTFT | Over capacity |
| 16K | 63.7153 ms TTFT | 131.8278 ms TTFT | 253.5519 ms TTFT | 385.2904 ms TTFT | 600.01,766 ms TTFT | 825.13,028 ms TTFT | Over capacity | Over capacity |
| 32K | 51.2213 ms TTFT | 102.6363 ms TTFT | 202.0642 ms TTFT | 278.31,300 ms TTFT | 416.72,173 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 36.0337 ms TTFT | 72.8552 ms TTFT | 143.7999 ms TTFT | 178.11,711 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 22.9653 ms TTFT | 45.51,061 ms TTFT | 89.71,825 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 13.71,002 ms TTFT | 27.71,455 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,849.0 | — | 832 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,455.0 | — | 1,716 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,590.0 | — | 3,760 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,240.0 | — | 8,898 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,505.0 | — | 23,377 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,768.0 | — | 66,934 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,734.0 | — | 68,885 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 75.1 | 75.6 | 128 ms | 13.2 ms | 1.0 | 0 | 2,252 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 150.6 | 75.5 | 651 ms | 13.3 ms | 2.0 | 0 | 4,518 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 288.3 | 72.5 | 1,289 ms | 13.8 ms | 4.0 | 0 | 8,645 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 479.0 | 59.3 | 1,344 ms | 16.9 ms | 8.0 | 0 | 14,360 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 776.9 | 48.8 | 2,155 ms | 20.5 ms | 16.0 | 0 | 23,276 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,105.2 | 34.4 | 2,955 ms | 29.1 ms | 32.0 | 0 | 33,144 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,426.1 | 21.5 | 4,717 ms | 46.4 ms | 64.0 | 0 | 42,685 | 29.9 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 63.7 | 64.3 | 153 ms | 15.6 ms | 1.0 | 0 | 1,910 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 131.8 | 66.6 | 278 ms | 15.0 ms | 2.0 | 0 | 3,950 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 253.5 | 63.9 | 519 ms | 15.6 ms | 4.0 | 0 | 7,597 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 385.2 | 48.4 | 904 ms | 20.7 ms | 8.0 | 0 | 11,537 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 600.0 | 37.7 | 1,766 ms | 26.5 ms | 16.0 | 0 | 17,996 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 825.1 | 25.4 | 3,028 ms | 39.3 ms | 32.0 | 0 | 24,752 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 51.2 | 51.8 | 213 ms | 19.3 ms | 1.0 | 0 | 1,533 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 102.6 | 51.8 | 363 ms | 19.3 ms | 2.0 | 0 | 3,077 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 202.0 | 50.9 | 642 ms | 19.7 ms | 4.0 | 0 | 6,057 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 278.3 | 35.0 | 1,300 ms | 28.5 ms | 8.0 | 0 | 8,346 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 416.7 | 26.1 | 2,173 ms | 38.2 ms | 16.0 | 0 | 12,471 | 29.9 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 36.0 | 36.4 | 337 ms | 27.4 ms | 1.0 | 0 | 1,079 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 72.8 | 36.8 | 552 ms | 27.2 ms | 2.0 | 0 | 2,179 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 143.7 | 36.1 | 999 ms | 27.7 ms | 4.0 | 0 | 4,309 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 178.1 | 22.2 | 1,711 ms | 45.0 ms | 8.0 | 0 | 5,338 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 22.9 | 23.1 | 653 ms | 43.2 ms | 1.0 | 0 | 686 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 45.5 | 23.1 | 1,061 ms | 43.2 ms | 2.0 | 0 | 1,362 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 89.7 | 22.6 | 1,825 ms | 44.3 ms | 4.0 | 0 | 2,687 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 13.7 | 13.9 | 1,002 ms | 71.9 ms | 1.0 | 0 | 410 | 29.8 s |
| Sustained decode | 251K | 2 | Valid | 27.7 | 13.9 | 1,455 ms | 71.8 ms | 2.0 | 0 | 830 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.