Speed test matrix
Qwen3.6-35B-A3B-FP8-dynamic on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context34.3K238 ms TTFT
16K context33.2K488 ms TTFT
32K context31.2K1,034 ms TTFT
64K context25.3K2,546 ms TTFT
128K context17.9K7,178 ms TTFT
251K context11.4K22,158 ms TTFT
256K context11.2K22,893 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 190.973 ms TTFT | 290.7127 ms TTFT | 503.3462 ms TTFT | 801.4246 ms TTFT | 1,270.1363 ms TTFT | 1,863.2548 ms TTFT | 2,715.71,004 ms TTFT | 3,705.91,690 ms TTFT |
| 16K | 186.671 ms TTFT | 284.490 ms TTFT | 499.6143 ms TTFT | 776.5227 ms TTFT | 1,208.7443 ms TTFT | 1,760.5483 ms TTFT | 2,472.6852 ms TTFT | Over capacity |
| 32K | 177.7110 ms TTFT | 272.7170 ms TTFT | 473.2249 ms TTFT | 748.6366 ms TTFT | 1,129.1483 ms TTFT | 1,561.9844 ms TTFT | Over capacity | Over capacity |
| 64K | 162.2139 ms TTFT | 250.9187 ms TTFT | 432.5259 ms TTFT | 677.6451 ms TTFT | 953.7765 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 139.3354 ms TTFT | 199.3452 ms TTFT | 357.1616 ms TTFT | 565.9957 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 112.6704 ms TTFT | 152.5909 ms TTFT | 275.81,296 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.3K | — | 238 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.2K | — | 488 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.2K | — | 1,034 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.3K | — | 2,546 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.9K | — | 7,178 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 11.4K | — | 22,158 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.2K | — | 22,893 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 190.9 | 191.5 | 73 ms | 5.2 ms | 1.0 | 1 | 5,725 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 290.7 | 146.7 | 127 ms | 6.8 ms | 2.0 | 2 | 8,722 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 503.3 | 129.6 | 462 ms | 7.7 ms | 4.0 | 4 | 15,096 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 801.4 | 101.0 | 246 ms | 9.9 ms | 8.0 | 0 | 24,041 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,270.1 | 80.1 | 363 ms | 12.5 ms | 16.0 | 0 | 38,096 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,863.2 | 58.9 | 548 ms | 17.0 ms | 32.0 | 0 | 55,872 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,715.7 | 42.7 | 1,004 ms | 23.4 ms | 64.0 | 0 | 81,472 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,705.9 | 28.6 | 1,690 ms | 35.0 ms | 128.0 | 0 | 111,104 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 186.6 | 187.4 | 71 ms | 5.3 ms | 1.0 | 1 | 5,599 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 284.4 | 143.2 | 90 ms | 7.0 ms | 2.0 | 2 | 8,530 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 499.6 | 125.5 | 143 ms | 8.0 ms | 4.0 | 4 | 14,988 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 776.5 | 97.8 | 227 ms | 10.2 ms | 8.0 | 0 | 23,296 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,208.7 | 76.1 | 443 ms | 13.1 ms | 16.0 | 0 | 36,256 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,760.5 | 55.4 | 483 ms | 18.1 ms | 32.0 | 0 | 52,800 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,472.6 | 38.8 | 852 ms | 25.8 ms | 64.0 | 0 | 74,176 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 177.7 | 178.3 | 110 ms | 5.6 ms | 1.0 | 1 | 5,330 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 272.7 | 137.6 | 170 ms | 7.3 ms | 2.0 | 2 | 8,180 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 473.2 | 118.7 | 249 ms | 8.4 ms | 4.0 | 0 | 14,192 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 748.6 | 93.8 | 366 ms | 10.7 ms | 8.0 | 0 | 22,456 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,129.1 | 70.5 | 483 ms | 14.2 ms | 16.0 | 0 | 33,872 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,561.9 | 48.7 | 844 ms | 20.5 ms | 32.0 | 0 | 46,848 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 162.2 | 163.1 | 139 ms | 6.1 ms | 1.0 | 1 | 4,866 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 250.9 | 126.3 | 187 ms | 7.9 ms | 2.0 | 2 | 7,526 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 432.5 | 108.3 | 259 ms | 9.2 ms | 4.0 | 0 | 12,972 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 677.6 | 84.5 | 451 ms | 11.8 ms | 8.0 | 0 | 20,328 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 953.7 | 59.3 | 765 ms | 16.9 ms | 16.0 | 0 | 28,608 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 139.3 | 141.3 | 354 ms | 7.1 ms | 1.0 | 1 | 4,178 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 199.3 | 99.4 | 452 ms | 10.1 ms | 2.0 | 0 | 5,978 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 357.1 | 88.9 | 616 ms | 11.2 ms | 4.0 | 0 | 10,712 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 565.9 | 69.8 | 957 ms | 14.3 ms | 8.0 | 0 | 16,976 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 112.6 | 112.8 | 704 ms | 8.9 ms | 1.0 | 0 | 3,378 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 152.5 | 76.1 | 909 ms | 13.1 ms | 2.0 | 0 | 4,574 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 275.8 | 68.4 | 1,296 ms | 14.6 ms | 4.0 | 0 | 8,272 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.