Speed test matrix
Qwen3.6-35B-A3B on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.6-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --moe-backend triton --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context30.9K265 ms TTFT
16K context32K506 ms TTFT
32K context30K1,075 ms TTFT
64K context26.4K2,439 ms TTFT
128K context20.9K6,151 ms TTFT
251K context14.7K17,199 ms TTFT
256K context14.4K17,801 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 196.987 ms TTFT | 354.9107 ms TTFT | 616.0102 ms TTFT | 1,000.2137 ms TTFT | 1,677.6554 ms TTFT | 2,377.91,018 ms TTFT | 3,417.81,869 ms TTFT | 3,960.13,465 ms TTFT |
| 16K | 153.590 ms TTFT | 286.9114 ms TTFT | 507.6119 ms TTFT | 873.7289 ms TTFT | 1,503.8599 ms TTFT | 1,972.6858 ms TTFT | 2,793.11,536 ms TTFT | 3,205.82,873 ms TTFT |
| 32K | 105.3118 ms TTFT | 209.4196 ms TTFT | 369.9256 ms TTFT | 665.5402 ms TTFT | 1,172.3592 ms TTFT | 1,465.91,099 ms TTFT | 2,061.91,867 ms TTFT | Over capacity |
| 64K | 65.8182 ms TTFT | 131.8264 ms TTFT | 244.6364 ms TTFT | 460.8553 ms TTFT | 832.4895 ms TTFT | 964.51,660 ms TTFT | Over capacity | Over capacity |
| 128K | 36.5320 ms TTFT | 74.7466 ms TTFT | 145.9650 ms TTFT | 281.41,000 ms TTFT | 526.81,794 ms TTFT | 575.83,102 ms TTFT | Over capacity | Over capacity |
| 251K | 21.2631 ms TTFT | 41.4925 ms TTFT | 82.51,394 ms TTFT | 162.92,229 ms TTFT | 307.63,741 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.9K | — | 265 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32K | — | 506 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30K | — | 1,075 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.4K | — | 2,439 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.9K | — | 6,151 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 14.7K | — | 17,199 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 14.4K | — | 17,801 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 196.9 | 200.8 | 87 ms | 5.0 ms | 1.0 | 1 | 5,908 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 354.9 | 181.3 | 107 ms | 5.5 ms | 2.0 | 2 | 10,646 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 616.0 | 158.0 | 102 ms | 6.3 ms | 4.0 | 4 | 18,480 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,000.2 | 128.6 | 137 ms | 7.8 ms | 8.0 | 8 | 30,002 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,677.6 | 106.3 | 554 ms | 9.4 ms | 16.0 | 0 | 50,313 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,377.9 | 74.4 | 1,018 ms | 13.4 ms | 32.0 | 0 | 71,290 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,417.8 | 52.5 | 1,869 ms | 19.1 ms | 64.0 | 0 | 102,426 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,960.1 | 29.7 | 3,465 ms | 33.7 ms | 128.0 | 0 | 118,725 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 153.5 | 159.2 | 90 ms | 6.3 ms | 1.0 | 1 | 4,605 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 286.9 | 149.9 | 114 ms | 6.7 ms | 2.0 | 2 | 8,605 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 507.6 | 130.9 | 119 ms | 7.6 ms | 4.0 | 4 | 15,222 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 873.7 | 110.9 | 289 ms | 9.0 ms | 8.0 | 0 | 26,203 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,503.8 | 94.8 | 599 ms | 10.5 ms | 16.0 | 0 | 45,090 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,972.6 | 62.2 | 858 ms | 16.1 ms | 32.0 | 0 | 59,120 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,793.1 | 43.5 | 1,536 ms | 23.0 ms | 64.0 | 0 | 83,694 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,205.8 | 24.5 | 2,873 ms | 40.8 ms | 128.0 | 0 | 95,915 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 105.3 | 106.2 | 118 ms | 9.4 ms | 1.0 | 0 | 3,157 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 209.4 | 105.2 | 196 ms | 9.5 ms | 2.0 | 0 | 6,283 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 369.9 | 93.7 | 256 ms | 10.7 ms | 4.0 | 0 | 11,098 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 665.5 | 83.9 | 402 ms | 11.9 ms | 8.0 | 0 | 19,950 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,172.3 | 73.8 | 592 ms | 13.5 ms | 16.0 | 0 | 35,166 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,465.9 | 46.0 | 1,099 ms | 21.8 ms | 32.0 | 0 | 43,918 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,061.9 | 32.0 | 1,867 ms | 31.2 ms | 64.0 | 0 | 61,786 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 65.8 | 66.5 | 182 ms | 15.0 ms | 1.0 | 0 | 1,972 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 131.8 | 66.3 | 264 ms | 15.1 ms | 2.0 | 0 | 3,950 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 244.6 | 61.6 | 364 ms | 16.2 ms | 4.0 | 0 | 7,338 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 460.8 | 58.0 | 553 ms | 17.3 ms | 8.0 | 0 | 13,809 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 832.4 | 52.4 | 895 ms | 19.1 ms | 16.0 | 0 | 24,949 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 964.5 | 30.2 | 1,660 ms | 33.1 ms | 32.0 | 0 | 28,928 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 36.5 | 37.3 | 320 ms | 26.8 ms | 1.0 | 0 | 1,095 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 74.7 | 37.9 | 466 ms | 26.4 ms | 2.0 | 0 | 2,238 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 145.9 | 36.9 | 650 ms | 27.1 ms | 4.0 | 0 | 4,369 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 281.4 | 35.7 | 1,000 ms | 28.0 ms | 8.0 | 0 | 8,437 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 526.8 | 32.9 | 1,794 ms | 30.4 ms | 16.0 | 0 | 15,803 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 575.8 | 18.2 | 3,102 ms | 54.9 ms | 32.0 | 0 | 17,237 | 29.9 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 21.2 | 21.7 | 631 ms | 46.1 ms | 1.0 | 0 | 636 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 41.4 | 21.2 | 925 ms | 47.2 ms | 2.0 | 0 | 1,238 | 29.9 s |
| Sustained decode | 251K | 4 | Valid | 82.5 | 21.0 | 1,394 ms | 47.7 ms | 4.0 | 0 | 2,475 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 162.9 | 20.3 | 2,229 ms | 49.3 ms | 8.0 | 0 | 4,886 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 307.6 | 19.2 | 3,741 ms | 52.1 ms | 16.0 | 0 | 9,210 | 29.9 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.