Speed test matrix
Qwen3.8-27B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.92 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass
Prefill
prompt tok/s by context length8K context9,925830 ms TTFT
16K context9,5711,700 ms TTFT
32K context8,7203,709 ms TTFT
64K context7,4158,694 ms TTFT
128K context5,68322,656 ms TTFT
251K context3,91064,521 ms TTFT
256K context3,86966,493 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 45.679 ms TTFT | 84.2116 ms TTFT | 163.0215 ms TTFT | 323.4344 ms TTFT | 581.7719 ms TTFT | 970.61,239 ms TTFT | 1,449.22,017 ms TTFT | Over capacity |
| 16K | 44.8125 ms TTFT | 82.4201 ms TTFT | 159.1328 ms TTFT | 311.6615 ms TTFT | 543.8995 ms TTFT | 870.21,991 ms TTFT | Over capacity | Over capacity |
| 32K | 43.5108 ms TTFT | 79.5150 ms TTFT | 152.4210 ms TTFT | 289.9392 ms TTFT | 482.0607 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 41.1177 ms TTFT | 70.9230 ms TTFT | 136.2342 ms TTFT | 254.4520 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 36.9322 ms TTFT | 60.1437 ms TTFT | 114.8624 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 30.5845 ms TTFT | 46.41,086 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,925.0 | — | 830 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,571.0 | — | 1,700 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,720.0 | — | 3,709 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,415.0 | — | 8,694 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,683.0 | — | 22,656 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,910.0 | — | 64,521 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,869.0 | — | 66,493 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 45.6 | 45.6 | 79 ms | 21.9 ms | 1.0 | 0 | 1,366 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 84.2 | 42.0 | 116 ms | 23.8 ms | 2.0 | 0 | 2,524 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 163.0 | 40.8 | 215 ms | 24.5 ms | 4.0 | 0 | 4,888 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 323.4 | 40.4 | 344 ms | 24.7 ms | 8.0 | 0 | 9,696 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 581.7 | 36.4 | 719 ms | 27.5 ms | 16.0 | 0 | 17,440 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 970.6 | 30.2 | 1,239 ms | 33.1 ms | 32.0 | 0 | 29,088 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,449.2 | 22.2 | 2,017 ms | 45.1 ms | 64.0 | 0 | 43,456 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 44.8 | 44.8 | 125 ms | 22.3 ms | 1.0 | 0 | 1,343 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 82.4 | 41.2 | 201 ms | 24.3 ms | 2.0 | 0 | 2,470 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 159.1 | 39.8 | 328 ms | 25.1 ms | 4.0 | 0 | 4,772 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 311.6 | 39.0 | 615 ms | 25.7 ms | 8.0 | 0 | 9,344 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 543.8 | 33.9 | 995 ms | 29.5 ms | 16.0 | 0 | 16,304 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 870.2 | 27.0 | 1,991 ms | 37.0 ms | 32.0 | 0 | 26,080 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 43.5 | 43.5 | 108 ms | 23.0 ms | 1.0 | 0 | 1,305 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 79.5 | 39.7 | 150 ms | 25.2 ms | 2.0 | 0 | 2,384 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 152.4 | 38.1 | 210 ms | 26.2 ms | 4.0 | 0 | 4,572 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 289.9 | 36.2 | 392 ms | 27.6 ms | 8.0 | 0 | 8,696 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 482.0 | 30.0 | 607 ms | 33.4 ms | 16.0 | 0 | 14,448 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 41.1 | 41.1 | 177 ms | 24.3 ms | 1.0 | 0 | 1,232 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 70.9 | 35.4 | 230 ms | 28.2 ms | 2.0 | 0 | 2,128 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 136.2 | 34.0 | 342 ms | 29.4 ms | 4.0 | 0 | 4,084 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 254.4 | 31.6 | 520 ms | 31.6 ms | 8.0 | 0 | 7,632 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 36.9 | 36.9 | 322 ms | 27.1 ms | 1.0 | 0 | 1,107 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 60.1 | 30.1 | 437 ms | 33.3 ms | 2.0 | 0 | 1,802 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 114.8 | 28.6 | 624 ms | 34.9 ms | 4.0 | 0 | 3,440 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 30.5 | 30.6 | 845 ms | 32.7 ms | 1.0 | 0 | 915 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 46.4 | 23.1 | 1,086 ms | 43.2 ms | 2.0 | 0 | 1,392 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.