Speed test matrix
Qwen3.5-35B-A3B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --linear-backend cutlass --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context32K256 ms TTFT
16K context37.1K438 ms TTFT
32K context31.8K1,014 ms TTFT
64K context25.5K2,524 ms TTFT
128K context17.5K7,350 ms TTFT
251K context10.8K23,460 ms TTFT
256K context10.6K24,244 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 265.382 ms TTFT | 475.0121 ms TTFT | 756.6142 ms TTFT | 1,196.8168 ms TTFT | Looped | 2,680.4968 ms TTFT | 3,802.31,687 ms TTFT | 4,907.23,049 ms TTFT |
| 16K | 260.790 ms TTFT | 467.3137 ms TTFT | 750.3169 ms TTFT | 1,179.2189 ms TTFT | 1,772.2520 ms TTFT | 2,612.7963 ms TTFT | 3,613.01,483 ms TTFT | Over capacity |
| 32K | 248.8125 ms TTFT | 451.6205 ms TTFT | 736.0170 ms TTFT | 1,132.1176 ms TTFT | 1,731.5722 ms TTFT | 2,417.81,181 ms TTFT | Over capacity | Over capacity |
| 64K | 238.4199 ms TTFT | 434.2247 ms TTFT | 700.5375 ms TTFT | 1,061.1266 ms TTFT | 1,567.61,074 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 210.5361 ms TTFT | 388.3388 ms TTFT | 634.5483 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 173.5728 ms TTFT | 314.8831 ms TTFT | 537.41,176 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32K | — | 256 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 37.1K | — | 438 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.8K | — | 1,014 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.5K | — | 2,524 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.5K | — | 7,350 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.8K | — | 23,460 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.6K | — | 24,244 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 265.3 | 265.4 | 82 ms | 3.8 ms | 1.0 | 2 | 7,956 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 475.0 | 238.8 | 121 ms | 4.2 ms | 2.0 | 2 | 14,248 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 756.6 | 189.1 | 142 ms | 5.3 ms | 4.0 | 4 | 22,697 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,196.8 | 149.1 | 168 ms | 6.7 ms | 8.0 | 8 | 35,890 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Valid | 2,680.4 | 84.7 | 968 ms | 11.8 ms | 32.0 | 0 | 80,398 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,802.3 | 59.0 | 1,687 ms | 16.9 ms | 64.0 | 0 | 113,974 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,907.2 | 37.2 | 3,049 ms | 26.8 ms | 128.0 | 0 | 147,120 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 260.7 | 262.4 | 90 ms | 3.8 ms | 1.0 | 2 | 7,820 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 467.3 | 235.9 | 137 ms | 4.2 ms | 2.0 | 3 | 14,019 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 750.3 | 191.5 | 169 ms | 5.2 ms | 4.0 | 4 | 22,505 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,179.2 | 150.8 | 189 ms | 6.6 ms | 8.0 | 8 | 35,368 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,772.2 | 112.9 | 520 ms | 8.9 ms | 16.0 | 0 | 53,161 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,612.7 | 82.7 | 963 ms | 12.1 ms | 32.0 | 0 | 78,350 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,613.0 | 56.6 | 1,483 ms | 17.7 ms | 64.0 | 0 | 108,306 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 248.8 | 252.4 | 125 ms | 4.0 ms | 1.0 | 2 | 7,465 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 451.6 | 228.2 | 205 ms | 4.4 ms | 2.0 | 2 | 13,544 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 736.0 | 186.5 | 170 ms | 5.4 ms | 4.0 | 4 | 22,084 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,132.1 | 144.8 | 176 ms | 6.9 ms | 8.0 | 8 | 33,964 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,731.5 | 109.6 | 722 ms | 9.1 ms | 16.0 | 0 | 51,927 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,417.8 | 76.2 | 1,181 ms | 13.1 ms | 32.0 | 0 | 72,467 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 238.4 | 239.9 | 199 ms | 4.2 ms | 1.0 | 1 | 7,151 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 434.2 | 219.4 | 247 ms | 4.6 ms | 2.0 | 2 | 13,030 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 700.5 | 178.3 | 375 ms | 5.6 ms | 3.9 | 4 | 21,014 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,061.1 | 134.5 | 266 ms | 7.4 ms | 8.0 | 8 | 31,817 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,567.6 | 98.3 | 1,074 ms | 10.2 ms | 16.0 | 0 | 47,014 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 210.5 | 214.5 | 361 ms | 4.7 ms | 1.0 | 1 | 6,315 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 388.3 | 197.6 | 388 ms | 5.1 ms | 2.0 | 2 | 11,646 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 634.5 | 162.4 | 483 ms | 6.2 ms | 4.0 | 4 | 19,033 | 30.0 s |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 173.5 | 177.0 | 728 ms | 5.7 ms | 1.0 | 1 | 5,204 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 314.8 | 162.1 | 831 ms | 6.2 ms | 2.0 | 2 | 9,444 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 537.4 | 138.2 | 1,176 ms | 7.2 ms | 3.9 | 4 | 16,120 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.