Speed test matrix
Qwen3.6-35B-A3B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.6-35B-A3B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --linear-backend cutlass --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context31.9K257 ms TTFT
16K context37.1K438 ms TTFT
32K context31.7K1,017 ms TTFT
64K context25.5K2,527 ms TTFT
128K context17.5K7,350 ms TTFT
251K context10.7K23,518 ms TTFT
256K context10.6K24,185 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 266.680 ms TTFT | 471.4100 ms TTFT | 789.6146 ms TTFT | 1,195.8148 ms TTFT | 1,794.9572 ms TTFT | 2,687.9970 ms TTFT | 3,698.21,664 ms TTFT | 3,641.6Requests queued2,885 ms TTFT |
| 16K | 263.886 ms TTFT | 453.7144 ms TTFT | 765.4190 ms TTFT | 1,215.0158 ms TTFT | 1,794.5570 ms TTFT | 2,609.5919 ms TTFT | 3,546.91,539 ms TTFT | Over capacity |
| 32K | 267.0120 ms TTFT | 449.8167 ms TTFT | 733.5170 ms TTFT | 1,143.2213 ms TTFT | 1,715.7703 ms TTFT | 2,427.41,294 ms TTFT | Over capacity | Over capacity |
| 64K | 246.9193 ms TTFT | 423.1281 ms TTFT | 724.8274 ms TTFT | 1,069.0399 ms TTFT | 1,567.61,049 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 219.1357 ms TTFT | 369.9379 ms TTFT | 643.9413 ms TTFT | 956.9642 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 179.5721 ms TTFT | 313.6874 ms TTFT | 546.1954 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.9K | — | 257 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 37.1K | — | 438 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.7K | — | 1,017 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.5K | — | 2,527 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.5K | — | 7,350 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.7K | — | 23,518 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.6K | — | 24,185 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 266.6 | 272.4 | 80 ms | 3.7 ms | 1.0 | 2 | 7,994 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 471.4 | 244.2 | 100 ms | 4.1 ms | 2.0 | 3 | 14,142 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 789.6 | 198.9 | 146 ms | 5.0 ms | 4.0 | 4 | 23,680 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,195.8 | 154.0 | 148 ms | 6.5 ms | 8.0 | 8 | 35,862 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,794.9 | 114.9 | 572 ms | 8.7 ms | 16.0 | 0 | 53,833 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,687.9 | 84.6 | 970 ms | 11.8 ms | 32.0 | 0 | 80,600 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,698.2 | 57.5 | 1,664 ms | 17.4 ms | 64.0 | 0 | 110,929 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 3,641.6 | 37.9 | 2,885 ms | 26.4 ms | 97.8 | 0 | 109,132 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 263.8 | 265.6 | 86 ms | 3.8 ms | 1.0 | 2 | 7,914 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 453.7 | 230.1 | 144 ms | 4.3 ms | 2.0 | 2 | 13,611 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 765.4 | 194.5 | 190 ms | 5.1 ms | 4.0 | 4 | 22,962 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,215.0 | 156.2 | 158 ms | 6.4 ms | 7.9 | 8 | 36,441 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,794.5 | 114.1 | 570 ms | 8.8 ms | 16.0 | 0 | 53,851 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,609.5 | 82.4 | 919 ms | 12.1 ms | 32.0 | 0 | 78,267 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,546.9 | 55.6 | 1,539 ms | 18.0 ms | 64.0 | 0 | 106,388 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 267.0 | 273.2 | 120 ms | 3.7 ms | 1.0 | 2 | 8,013 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 449.8 | 228.1 | 167 ms | 4.4 ms | 2.0 | 2 | 13,494 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 733.5 | 187.7 | 170 ms | 5.3 ms | 4.0 | 4 | 21,999 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,143.2 | 148.3 | 213 ms | 6.7 ms | 8.0 | 8 | 34,284 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,715.7 | 109.2 | 703 ms | 9.2 ms | 16.0 | 0 | 51,469 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,427.4 | 76.9 | 1,294 ms | 13.0 ms | 32.0 | 0 | 72,803 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 246.9 | 248.6 | 193 ms | 4.0 ms | 1.0 | 2 | 7,365 | 29.8 s |
| Sustained decode | 64K | 2 | Valid | 423.1 | 216.0 | 281 ms | 4.6 ms | 2.0 | 2 | 12,689 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 724.8 | 181.8 | 274 ms | 5.5 ms | 4.0 | 4 | 21,739 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,069.0 | 137.2 | 399 ms | 7.3 ms | 7.9 | 8 | 32,062 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,567.6 | 98.8 | 1,049 ms | 10.1 ms | 16.0 | 0 | 47,012 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 219.1 | 222.7 | 357 ms | 4.5 ms | 1.0 | 1 | 6,571 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 369.9 | 190.7 | 379 ms | 5.2 ms | 2.0 | 2 | 11,096 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 643.9 | 162.3 | 413 ms | 6.2 ms | 4.0 | 4 | 19,307 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 956.9 | 118.7 | 642 ms | 8.4 ms | 7.9 | 5 | 28,640 | 29.9 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 179.5 | 187.2 | 721 ms | 5.3 ms | 1.0 | 1 | 5,385 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 313.6 | 159.5 | 874 ms | 6.3 ms | 1.9 | 2 | 9,407 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 546.1 | 139.6 | 954 ms | 7.2 ms | 3.9 | 4 | 16,389 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.