Speed test matrix
Qwen3.6-35B-A3B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/Qwen3.6-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context35.5K231 ms TTFT
16K context34.6K468 ms TTFT
32K context33.1K976 ms TTFT
64K context26.7K2,416 ms TTFT
128K context18.6K6,921 ms TTFT
251K context11.7K21,623 ms TTFT
256K context11.5K22,372 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 189.979 ms TTFT | 278.1138 ms TTFT | 455.1160 ms TTFT | 712.6245 ms TTFT | 1,084.9346 ms TTFT | 1,523.8552 ms TTFT | 2,221.7962 ms TTFT | 3,177.61,671 ms TTFT |
| 16K | 185.877 ms TTFT | 272.9110 ms TTFT | 445.9185 ms TTFT | 676.1247 ms TTFT | 1,054.1336 ms TTFT | 1,453.1505 ms TTFT | 2,070.1825 ms TTFT | Over capacity |
| 32K | 176.0111 ms TTFT | 259.5189 ms TTFT | 431.7257 ms TTFT | 670.1347 ms TTFT | 980.1513 ms TTFT | 1,313.5842 ms TTFT | Over capacity | Over capacity |
| 64K | 160.1138 ms TTFT | 240.9177 ms TTFT | 395.4273 ms TTFT | 608.6446 ms TTFT | 861.1773 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 137.7357 ms TTFT | 191.5476 ms TTFT | 330.3646 ms TTFT | 508.5997 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 111.4705 ms TTFT | 148.3922 ms TTFT | 258.51,304 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 35.5K | — | 231 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.6K | — | 468 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33.1K | — | 976 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.7K | — | 2,416 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 18.6K | — | 6,921 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 11.7K | — | 21,623 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.5K | — | 22,372 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 189.9 | 190.5 | 79 ms | 5.2 ms | 1.0 | 1 | 5,696 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 278.1 | 140.7 | 138 ms | 7.1 ms | 2.0 | 2 | 8,343 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 455.1 | 114.3 | 160 ms | 8.7 ms | 4.0 | 0 | 13,648 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 712.6 | 89.9 | 245 ms | 11.1 ms | 8.0 | 0 | 21,376 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,084.9 | 68.4 | 346 ms | 14.6 ms | 16.0 | 0 | 32,544 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,523.8 | 48.3 | 552 ms | 20.7 ms | 32.0 | 0 | 45,696 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,221.7 | 35.1 | 962 ms | 28.5 ms | 64.0 | 0 | 66,624 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,177.6 | 24.8 | 1,671 ms | 40.4 ms | 128.0 | 0 | 95,232 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 185.8 | 186.6 | 77 ms | 5.4 ms | 1.0 | 1 | 5,574 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 272.9 | 137.3 | 110 ms | 7.3 ms | 2.0 | 2 | 8,186 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 445.9 | 111.9 | 185 ms | 8.9 ms | 4.0 | 0 | 13,377 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 676.1 | 85.3 | 247 ms | 11.7 ms | 8.0 | 0 | 20,280 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,054.1 | 66.3 | 336 ms | 15.1 ms | 16.0 | 0 | 31,616 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,453.1 | 45.8 | 505 ms | 21.8 ms | 32.0 | 0 | 43,584 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,070.1 | 32.6 | 825 ms | 30.7 ms | 64.0 | 0 | 62,080 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 176.0 | 176.7 | 111 ms | 5.7 ms | 1.0 | 1 | 5,280 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 259.5 | 131.5 | 189 ms | 7.6 ms | 2.0 | 2 | 7,784 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 431.7 | 108.2 | 257 ms | 9.2 ms | 4.0 | 0 | 12,952 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 670.1 | 83.8 | 347 ms | 11.9 ms | 8.0 | 0 | 20,096 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 980.1 | 61.3 | 513 ms | 16.3 ms | 16.0 | 0 | 29,392 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,313.5 | 41.0 | 842 ms | 24.4 ms | 32.0 | 0 | 39,392 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 160.1 | 160.9 | 138 ms | 6.2 ms | 1.0 | 1 | 4,803 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 240.9 | 121.2 | 177 ms | 8.3 ms | 2.0 | 2 | 7,226 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 395.4 | 98.9 | 273 ms | 10.1 ms | 4.0 | 0 | 11,860 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 608.6 | 76.0 | 446 ms | 13.2 ms | 8.0 | 0 | 18,256 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 861.1 | 53.4 | 773 ms | 18.7 ms | 16.0 | 0 | 25,824 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 137.7 | 139.7 | 357 ms | 7.2 ms | 1.0 | 1 | 4,131 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 191.5 | 95.6 | 476 ms | 10.5 ms | 2.0 | 0 | 5,742 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 330.3 | 82.2 | 646 ms | 12.2 ms | 4.0 | 0 | 9,908 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 508.5 | 62.9 | 997 ms | 15.9 ms | 8.0 | 0 | 15,248 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 111.4 | 111.6 | 705 ms | 9.0 ms | 1.0 | 0 | 3,342 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 148.3 | 74.0 | 922 ms | 13.5 ms | 2.0 | 0 | 4,450 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 258.5 | 64.1 | 1,304 ms | 15.6 ms | 4.0 | 0 | 7,752 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.