Speed test matrix
Qwen3.6-27B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.6-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.92 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context9,831833 ms TTFT
16K context9,4211,722 ms TTFT
32K context8,5573,774 ms TTFT
64K context7,2308,911 ms TTFT
128K context5,49423,427 ms TTFT
251K context3,76267,038 ms TTFT
256K context3,72069,149 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 78.1129 ms TTFT | 151.3251 ms TTFT | 292.3508 ms TTFT | 483.1943 ms TTFT | 770.41,765 ms TTFT | 1,103.02,959 ms TTFT | 1,424.64,751 ms TTFT | Over capacity |
| 16K | 65.3155 ms TTFT | 125.9282 ms TTFT | 250.0528 ms TTFT | 390.0899 ms TTFT | 605.11,761 ms TTFT | 845.72,949 ms TTFT | Over capacity | Over capacity |
| 32K | 52.1215 ms TTFT | 107.8366 ms TTFT | 199.4645 ms TTFT | 283.81,305 ms TTFT | 422.72,182 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 42.1343 ms TTFT | 74.9552 ms TTFT | 150.6975 ms TTFT | 185.21,975 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 25.0658 ms TTFT | 48.91,045 ms TTFT | 96.11,809 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 15.31,017 ms TTFT | 28.91,424 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,831.0 | — | 833 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,421.0 | — | 1,722 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,557.0 | — | 3,774 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,230.0 | — | 8,911 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,494.0 | — | 23,427 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,762.0 | — | 67,038 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,720.0 | — | 69,149 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 78.1 | 79.1 | 129 ms | 12.6 ms | 1.0 | 0 | 2,342 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 151.3 | 77.3 | 251 ms | 12.9 ms | 2.0 | 0 | 4,535 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 292.3 | 74.4 | 508 ms | 13.4 ms | 4.0 | 0 | 8,761 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 483.1 | 61.8 | 943 ms | 16.2 ms | 8.0 | 0 | 14,479 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 770.4 | 49.1 | 1,765 ms | 20.4 ms | 16.0 | 0 | 23,072 | 29.9 s |
| Sustained decode | 8K | 32 | Valid | 1,103.0 | 34.6 | 2,959 ms | 28.9 ms | 32.0 | 0 | 33,075 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,424.6 | 21.8 | 4,751 ms | 46.0 ms | 64.0 | 0 | 42,736 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 65.3 | 66.9 | 155 ms | 15.0 ms | 1.0 | 0 | 1,958 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 125.9 | 64.0 | 282 ms | 15.6 ms | 2.0 | 0 | 3,773 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 250.0 | 63.3 | 528 ms | 15.8 ms | 4.0 | 0 | 7,496 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 390.0 | 49.6 | 899 ms | 20.2 ms | 8.0 | 0 | 11,688 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 605.1 | 38.8 | 1,761 ms | 25.8 ms | 16.0 | 0 | 18,120 | 29.9 s |
| Sustained decode | 16K | 32 | Valid | 845.7 | 26.4 | 2,949 ms | 37.9 ms | 32.0 | 0 | 25,319 | 29.9 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 52.1 | 52.9 | 215 ms | 18.9 ms | 1.0 | 0 | 1,561 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 107.8 | 54.6 | 366 ms | 18.3 ms | 2.0 | 0 | 3,231 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 199.4 | 51.3 | 645 ms | 19.5 ms | 4.0 | 0 | 5,973 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 283.8 | 36.3 | 1,305 ms | 27.5 ms | 8.0 | 0 | 8,504 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 422.7 | 27.1 | 2,182 ms | 36.9 ms | 16.0 | 0 | 12,674 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 42.1 | 42.6 | 343 ms | 23.5 ms | 1.0 | 0 | 1,260 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 74.9 | 38.5 | 552 ms | 26.0 ms | 2.0 | 0 | 2,243 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 150.6 | 38.6 | 975 ms | 25.9 ms | 4.0 | 0 | 4,515 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 185.2 | 23.8 | 1,975 ms | 42.0 ms | 8.0 | 0 | 5,536 | 29.9 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 25.0 | 25.4 | 658 ms | 39.3 ms | 1.0 | 0 | 747 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 48.9 | 25.1 | 1,045 ms | 39.9 ms | 2.0 | 0 | 1,462 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 96.1 | 24.4 | 1,809 ms | 41.0 ms | 4.0 | 0 | 2,876 | 29.9 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 15.3 | 15.6 | 1,017 ms | 64.3 ms | 1.0 | 0 | 458 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 28.9 | 14.8 | 1,424 ms | 67.4 ms | 2.0 | 0 | 863 | 29.9 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.