Speed test matrix
Qwen3.8-27B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.8-27B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.92 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context9,767843 ms TTFT
16K context9,4191,727 ms TTFT
32K context8,5893,765 ms TTFT
64K context7,2338,913 ms TTFT
128K context5,49323,437 ms TTFT
251K context3,76067,097 ms TTFT
256K context3,72269,114 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 58.6137 ms TTFT | 127.3270 ms TTFT | 238.0530 ms TTFT | 391.6993 ms TTFT | 638.21,636 ms TTFT | 909.03,081 ms TTFT | 1,150.04,791 ms TTFT | Over capacity |
| 16K | 52.8165 ms TTFT | 108.3291 ms TTFT | 219.6541 ms TTFT | 326.3893 ms TTFT | 505.11,854 ms TTFT | 686.42,970 ms TTFT | Over capacity | Over capacity |
| 32K | 44.3219 ms TTFT | 87.1367 ms TTFT | 168.7649 ms TTFT | 231.11,332 ms TTFT | 350.82,228 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 30.7342 ms TTFT | 65.9553 ms TTFT | 126.5983 ms TTFT | 155.42,005 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 18.2663 ms TTFT | 36.91,066 ms TTFT | 78.81,849 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 10.71,024 ms TTFT | 22.81,443 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,767.0 | — | 843 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,419.0 | — | 1,727 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,589.0 | — | 3,765 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,233.0 | — | 8,913 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,493.0 | — | 23,437 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,760.0 | — | 67,097 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,722.0 | — | 69,114 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 58.6 | 59.2 | 137 ms | 16.9 ms | 1.0 | 0 | 1,755 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 127.3 | 64.2 | 270 ms | 15.6 ms | 2.0 | 0 | 3,820 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 238.0 | 60.2 | 530 ms | 16.6 ms | 4.0 | 0 | 7,134 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 391.6 | 49.6 | 993 ms | 20.1 ms | 8.0 | 0 | 11,747 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 638.2 | 40.5 | 1,636 ms | 24.7 ms | 16.0 | 0 | 19,141 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 909.0 | 28.7 | 3,081 ms | 34.8 ms | 32.0 | 0 | 27,267 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,150.0 | 17.6 | 4,791 ms | 56.9 ms | 64.0 | 0 | 34,363 | 29.9 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 52.8 | 53.8 | 165 ms | 18.6 ms | 1.0 | 0 | 1,585 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 108.3 | 55.0 | 291 ms | 18.2 ms | 2.0 | 0 | 3,247 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 219.6 | 54.6 | 541 ms | 18.3 ms | 4.0 | 0 | 6,583 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 326.3 | 41.0 | 893 ms | 24.4 ms | 8.0 | 0 | 9,773 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 505.1 | 32.5 | 1,854 ms | 30.8 ms | 16.0 | 0 | 15,119 | 29.9 s |
| Sustained decode | 16K | 32 | Valid | 686.4 | 21.7 | 2,970 ms | 46.0 ms | 32.0 | 0 | 20,530 | 29.9 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 44.3 | 45.6 | 219 ms | 22.0 ms | 1.0 | 0 | 1,329 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 87.1 | 44.1 | 367 ms | 22.7 ms | 2.0 | 0 | 2,610 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 168.7 | 42.5 | 649 ms | 23.6 ms | 4.0 | 0 | 5,056 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 231.1 | 29.9 | 1,332 ms | 33.4 ms | 8.0 | 0 | 6,925 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 350.8 | 22.3 | 2,228 ms | 44.8 ms | 16.0 | 0 | 10,517 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 30.7 | 31.7 | 342 ms | 31.5 ms | 1.0 | 0 | 921 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 65.9 | 33.5 | 553 ms | 29.9 ms | 2.0 | 0 | 1,972 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 126.5 | 32.3 | 983 ms | 31.0 ms | 4.0 | 0 | 3,791 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 155.4 | 20.3 | 2,005 ms | 49.2 ms | 8.0 | 0 | 4,658 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 18.2 | 19.0 | 663 ms | 52.5 ms | 1.0 | 0 | 544 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 36.9 | 19.6 | 1,066 ms | 51.1 ms | 2.0 | 0 | 1,106 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 78.8 | 20.3 | 1,849 ms | 49.2 ms | 4.0 | 0 | 2,356 | 29.9 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 10.7 | 11.6 | 1,024 ms | 85.9 ms | 1.0 | 0 | 320 | 29.8 s |
| Sustained decode | 251K | 2 | Valid | 22.8 | 12.3 | 1,443 ms | 81.4 ms | 2.0 | 0 | 683 | 29.9 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.