Speed test matrix
Qwen3.6-35B-A3B on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.6-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --moe-backend triton
Prefill
prompt tok/s by context length8K context31.4K261 ms TTFT
16K context31.4K516 ms TTFT
32K context31.1K1,038 ms TTFT
64K context28.4K2,266 ms TTFT
128K context23.2K5,557 ms TTFT
251K context17.1K14,748 ms TTFT
256K context16.8K15,307 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 229.770 ms TTFT | 328.994 ms TTFT | 565.0148 ms TTFT | 892.2250 ms TTFT | 1,406.5354 ms TTFT | 2,123.0520 ms TTFT | 3,199.71,002 ms TTFT | 4,551.01,656 ms TTFT |
| 16K | 227.171 ms TTFT | 326.4100 ms TTFT | 560.5137 ms TTFT | 878.3220 ms TTFT | 1,336.6298 ms TTFT | 2,034.6484 ms TTFT | 3,050.2793 ms TTFT | 4,224.91,432 ms TTFT |
| 32K | 220.4109 ms TTFT | 316.5151 ms TTFT | 540.2201 ms TTFT | 846.5295 ms TTFT | 1,286.7479 ms TTFT | 1,852.7803 ms TTFT | 2,764.71,396 ms TTFT | 3,660.12,616 ms TTFT |
| 64K | 210.4149 ms TTFT | 298.7216 ms TTFT | 519.9231 ms TTFT | 804.7469 ms TTFT | 1,200.2807 ms TTFT | 1,658.01,633 ms TTFT | 2,285.73,135 ms TTFT | Over capacity |
| 128K | 191.5329 ms TTFT | 270.7460 ms TTFT | 465.3627 ms TTFT | 712.21,002 ms TTFT | 1,007.21,775 ms TTFT | 1,330.63,236 ms TTFT | Over capacity | Over capacity |
| 251K | 162.9645 ms TTFT | 234.3861 ms TTFT | 379.21,247 ms TTFT | 590.31,987 ms TTFT | 780.53,571 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 31.4K | — | 261 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.4K | — | 516 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.1K | — | 1,038 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.4K | — | 2,266 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.2K | — | 5,557 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.1K | — | 14,748 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.8K | — | 15,307 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 229.7 | 230.2 | 70 ms | 4.3 ms | 1.0 | 1 | 6,889 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 328.9 | 165.1 | 94 ms | 6.1 ms | 2.0 | 2 | 9,866 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 565.0 | 142.5 | 148 ms | 7.0 ms | 4.0 | 4 | 16,948 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 892.2 | 112.1 | 250 ms | 8.9 ms | 8.0 | 0 | 26,760 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,406.5 | 88.4 | 354 ms | 11.3 ms | 16.0 | 0 | 42,192 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,123.0 | 66.7 | 520 ms | 15.0 ms | 32.0 | 0 | 63,680 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,199.7 | 50.3 | 1,002 ms | 19.9 ms | 64.0 | 0 | 95,936 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,551.0 | 35.0 | 1,656 ms | 28.5 ms | 128.0 | 0 | 136,448 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 227.1 | 227.7 | 71 ms | 4.4 ms | 1.0 | 1 | 6,812 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 326.4 | 164.2 | 100 ms | 6.1 ms | 2.0 | 2 | 9,792 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 560.5 | 140.9 | 137 ms | 7.1 ms | 4.0 | 4 | 16,815 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 878.3 | 110.2 | 220 ms | 9.1 ms | 8.0 | 0 | 26,344 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,336.6 | 84.1 | 298 ms | 11.9 ms | 16.0 | 0 | 40,096 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,034.6 | 63.9 | 484 ms | 15.6 ms | 32.0 | 0 | 61,024 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,050.2 | 47.7 | 793 ms | 21.0 ms | 64.0 | 0 | 91,509 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,224.9 | 32.7 | 1,432 ms | 30.6 ms | 128.0 | 0 | 126,720 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 220.4 | 221.3 | 109 ms | 4.5 ms | 1.0 | 1 | 6,611 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 316.5 | 158.9 | 151 ms | 6.3 ms | 2.0 | 2 | 9,494 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 540.2 | 136.0 | 201 ms | 7.4 ms | 4.0 | 4 | 16,203 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 846.5 | 106.1 | 295 ms | 9.4 ms | 8.0 | 0 | 25,392 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,286.7 | 80.4 | 479 ms | 12.4 ms | 16.0 | 0 | 38,592 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,852.7 | 57.7 | 803 ms | 17.3 ms | 32.0 | 0 | 55,552 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,764.7 | 42.5 | 1,396 ms | 23.5 ms | 64.0 | 0 | 82,944 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,660.1 | 27.4 | 2,616 ms | 36.4 ms | 128.0 | 0 | 109,696 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 210.4 | 211.5 | 149 ms | 4.7 ms | 1.0 | 1 | 6,311 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 298.7 | 150.5 | 216 ms | 6.6 ms | 2.0 | 2 | 8,961 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 519.9 | 130.7 | 231 ms | 7.6 ms | 3.9 | 4 | 15,593 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 804.7 | 100.4 | 469 ms | 10.0 ms | 8.0 | 0 | 24,136 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,200.2 | 74.5 | 807 ms | 13.4 ms | 16.0 | 0 | 36,000 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,658.0 | 51.3 | 1,633 ms | 19.5 ms | 32.0 | 0 | 49,728 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,285.7 | 34.6 | 3,135 ms | 28.9 ms | 64.0 | 0 | 68,544 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 191.5 | 193.9 | 329 ms | 5.2 ms | 1.0 | 1 | 5,743 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 270.7 | 137.2 | 460 ms | 7.3 ms | 2.0 | 2 | 8,120 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 465.3 | 116.2 | 627 ms | 8.6 ms | 4.0 | 0 | 13,956 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 712.2 | 88.2 | 1,002 ms | 11.3 ms | 8.0 | 0 | 21,360 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,007.2 | 61.2 | 1,775 ms | 16.4 ms | 16.0 | 0 | 30,208 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,330.6 | 39.4 | 3,236 ms | 25.4 ms | 32.0 | 0 | 39,904 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 162.9 | 167.1 | 645 ms | 6.0 ms | 1.0 | 1 | 4,887 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 234.3 | 117.2 | 861 ms | 8.5 ms | 2.0 | 0 | 7,028 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 379.2 | 94.4 | 1,247 ms | 10.6 ms | 4.0 | 0 | 11,376 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 590.3 | 72.8 | 1,987 ms | 13.7 ms | 8.0 | 0 | 17,704 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 780.5 | 46.8 | 3,571 ms | 21.4 ms | 16.0 | 0 | 23,408 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.