Speed test matrix
Qwen3.5-35B-A3B-Base on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-Base --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --max-model-len 262144 --gpu-memory-utilization 0.98 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --moe-backend triton
Prefill
prompt tok/s by context length8K context32K256 ms TTFT
16K context31.3K518 ms TTFT
32K context31.1K1,039 ms TTFT
64K context28.5K2,258 ms TTFT
128K context23.2K5,553 ms TTFT
251K context17.1K14,736 ms TTFT
256K context16.6K15,490 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 236.169 ms TTFT | 337.194 ms TTFT | 569.3146 ms TTFT | 891.5245 ms TTFT | 1,327.0352 ms TTFT | 1,937.3506 ms TTFT | 2,909.7986 ms TTFT | 4,240.31,939 ms TTFT |
| 16K | 233.472 ms TTFT | 334.292 ms TTFT | 564.9134 ms TTFT | 853.1197 ms TTFT | 1,285.5269 ms TTFT | 1,857.3439 ms TTFT | 2,794.9775 ms TTFT | 3,951.51,409 ms TTFT |
| 32K | 226.5109 ms TTFT | 323.4137 ms TTFT | 551.9193 ms TTFT | 834.3292 ms TTFT | 1,221.1473 ms TTFT | 1,759.4870 ms TTFT | 2,528.81,383 ms TTFT | 3,436.72,583 ms TTFT |
| 64K | 216.1139 ms TTFT | 302.3164 ms TTFT | 523.2212 ms TTFT | 795.5430 ms TTFT | 1,129.5745 ms TTFT | 1,565.91,319 ms TTFT | 2,127.82,492 ms TTFT | Over capacity |
| 128K | 195.8319 ms TTFT | 274.0387 ms TTFT | 475.3598 ms TTFT | 703.0902 ms TTFT | 959.91,576 ms TTFT | 1,266.93,109 ms TTFT | Over capacity | Over capacity |
| 251K | 166.1627 ms TTFT | 237.4806 ms TTFT | 382.01,151 ms TTFT | 580.21,886 ms TTFT | 755.23,316 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32K | — | 256 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.3K | — | 518 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.1K | — | 1,039 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 28.5K | — | 2,258 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.2K | — | 5,553 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.1K | — | 14,736 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.6K | — | 15,490 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 236.1 | 236.7 | 69 ms | 4.2 ms | 1.0 | 1 | 7,084 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 337.1 | 169.2 | 94 ms | 5.9 ms | 2.0 | 2 | 10,112 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 569.3 | 143.4 | 146 ms | 7.0 ms | 4.0 | 4 | 17,076 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 891.5 | 111.6 | 245 ms | 9.0 ms | 8.0 | 0 | 26,744 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,327.0 | 83.0 | 352 ms | 12.0 ms | 16.0 | 0 | 39,808 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,937.3 | 60.4 | 506 ms | 16.5 ms | 32.0 | 0 | 58,112 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,909.7 | 45.3 | 986 ms | 22.1 ms | 64.0 | 0 | 87,296 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,240.3 | 32.3 | 1,939 ms | 30.9 ms | 128.0 | 0 | 127,104 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 233.4 | 233.9 | 72 ms | 4.3 ms | 1.0 | 1 | 7,001 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 334.2 | 167.7 | 92 ms | 6.0 ms | 2.0 | 2 | 10,024 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 564.9 | 141.9 | 134 ms | 7.0 ms | 4.0 | 4 | 16,947 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 853.1 | 106.9 | 197 ms | 9.4 ms | 8.0 | 0 | 25,592 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,285.5 | 80.4 | 269 ms | 12.4 ms | 16.0 | 0 | 38,560 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,857.3 | 58.1 | 439 ms | 17.2 ms | 32.0 | 0 | 55,712 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,794.9 | 43.5 | 775 ms | 23.0 ms | 64.0 | 0 | 83,840 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,951.5 | 30.4 | 1,409 ms | 32.8 ms | 128.0 | 0 | 118,528 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 226.5 | 227.3 | 109 ms | 4.4 ms | 1.0 | 1 | 6,795 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 323.4 | 162.3 | 137 ms | 6.2 ms | 2.0 | 2 | 9,700 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 551.9 | 138.6 | 193 ms | 7.2 ms | 3.9 | 4 | 16,555 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 834.3 | 104.4 | 292 ms | 9.6 ms | 8.0 | 0 | 25,024 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,221.1 | 76.1 | 473 ms | 13.1 ms | 16.0 | 0 | 36,624 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,759.4 | 54.7 | 870 ms | 18.3 ms | 32.0 | 0 | 52,768 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,528.8 | 38.9 | 1,383 ms | 25.7 ms | 64.0 | 0 | 75,840 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,436.7 | 25.8 | 2,583 ms | 38.7 ms | 128.0 | 0 | 103,110 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 216.1 | 217.2 | 139 ms | 4.6 ms | 1.0 | 1 | 6,484 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 302.3 | 152.5 | 164 ms | 6.6 ms | 2.0 | 2 | 9,068 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 523.2 | 131.5 | 212 ms | 7.6 ms | 4.0 | 4 | 15,695 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 795.5 | 99.3 | 430 ms | 10.1 ms | 8.0 | 0 | 23,864 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,129.5 | 70.3 | 745 ms | 14.2 ms | 16.0 | 0 | 33,872 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,565.9 | 48.5 | 1,319 ms | 20.6 ms | 32.0 | 0 | 46,977 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,127.8 | 32.4 | 2,492 ms | 30.9 ms | 64.0 | 0 | 63,808 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 195.8 | 198.1 | 319 ms | 5.0 ms | 1.0 | 1 | 5,873 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 274.0 | 138.7 | 387 ms | 7.2 ms | 2.0 | 2 | 8,220 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 475.3 | 118.5 | 598 ms | 8.4 ms | 4.0 | 0 | 14,255 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 703.0 | 87.1 | 902 ms | 11.5 ms | 8.0 | 0 | 21,088 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 959.9 | 58.5 | 1,576 ms | 17.1 ms | 16.0 | 0 | 28,784 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,266.9 | 37.6 | 3,109 ms | 26.6 ms | 32.0 | 0 | 37,984 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 166.1 | 170.1 | 627 ms | 5.9 ms | 1.0 | 1 | 4,984 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 237.4 | 118.7 | 806 ms | 8.4 ms | 2.0 | 0 | 7,123 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 382.0 | 95.0 | 1,151 ms | 10.5 ms | 4.0 | 0 | 11,456 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 580.2 | 71.6 | 1,886 ms | 14.0 ms | 8.0 | 0 | 17,400 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 755.2 | 45.3 | 3,316 ms | 22.1 ms | 16.0 | 0 | 22,656 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.