Speed test matrix
Qwen3.5-35B-A3B on RTX PRO 6000 Blackwell Server Edition 96GB ×2
vllm · 0.30.1rc1.dev452+g73a583112 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 2
- VRAM
- 192 GB (2 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-35B-A3B --served-model-name qwen --trust-remote-code --tensor-parallel-size 2 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --moe-backend triton --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context30.9K265 ms TTFT
16K context32K508 ms TTFT
32K context29.9K1,082 ms TTFT
64K context26.4K2,442 ms TTFT
128K context20.9K6,157 ms TTFT
251K context14.6K17,216 ms TTFT
256K context14.4K17,805 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 198.287 ms TTFT | 367.7107 ms TTFT | 618.0136 ms TTFT | 994.8163 ms TTFT | 1,660.8557 ms TTFT | 2,381.21,016 ms TTFT | 3,409.61,869 ms TTFT | 3,972.83,480 ms TTFT |
| 16K | 151.590 ms TTFT | 288.2120 ms TTFT | 497.8120 ms TTFT | 859.1275 ms TTFT | 1,466.6575 ms TTFT | 1,973.8829 ms TTFT | 2,809.71,551 ms TTFT | 3,227.02,809 ms TTFT |
| 32K | 104.6146 ms TTFT | 203.3199 ms TTFT | 375.8256 ms TTFT | 671.1407 ms TTFT | 1,176.4599 ms TTFT | 1,470.91,007 ms TTFT | 2,070.61,872 ms TTFT | Over capacity |
| 64K | 65.3183 ms TTFT | 130.1268 ms TTFT | 245.6369 ms TTFT | 451.6558 ms TTFT | 833.2959 ms TTFT | 972.61,612 ms TTFT | Over capacity | Over capacity |
| 128K | 35.0325 ms TTFT | 77.8466 ms TTFT | 144.6653 ms TTFT | 280.11,019 ms TTFT | 521.91,788 ms TTFT | 581.43,117 ms TTFT | Over capacity | Over capacity |
| 251K | 21.6633 ms TTFT | 42.1943 ms TTFT | 79.71,415 ms TTFT | 156.52,232 ms TTFT | 307.33,753 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 30.9K | — | 265 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32K | — | 508 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 29.9K | — | 1,082 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 26.4K | — | 2,442 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 20.9K | — | 6,157 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 14.6K | — | 17,216 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 14.4K | — | 17,805 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 198.2 | 202.0 | 87 ms | 5.0 ms | 1.0 | 1 | 5,944 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 367.7 | 187.6 | 107 ms | 5.3 ms | 2.0 | 2 | 11,029 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 618.0 | 160.3 | 136 ms | 6.2 ms | 4.0 | 4 | 18,540 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 994.8 | 127.9 | 163 ms | 7.8 ms | 8.0 | 8 | 29,828 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,660.8 | 104.9 | 557 ms | 9.5 ms | 16.0 | 0 | 49,800 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,381.2 | 74.6 | 1,016 ms | 13.4 ms | 32.0 | 0 | 71,403 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,409.6 | 52.4 | 1,869 ms | 19.1 ms | 64.0 | 0 | 102,279 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,972.8 | 29.8 | 3,480 ms | 33.6 ms | 128.0 | 0 | 119,046 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 151.5 | 158.3 | 90 ms | 6.3 ms | 1.0 | 1 | 4,544 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 288.2 | 150.6 | 120 ms | 6.6 ms | 2.0 | 2 | 8,643 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 497.8 | 132.1 | 120 ms | 7.6 ms | 4.0 | 4 | 14,931 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 859.1 | 108.9 | 275 ms | 9.2 ms | 8.0 | 0 | 25,758 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,466.6 | 92.9 | 575 ms | 10.8 ms | 16.0 | 0 | 43,995 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,973.8 | 62.2 | 829 ms | 16.1 ms | 32.0 | 0 | 59,185 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,809.7 | 43.9 | 1,551 ms | 22.8 ms | 64.0 | 0 | 84,170 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,227.0 | 24.8 | 2,809 ms | 40.3 ms | 128.0 | 0 | 96,579 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 104.6 | 105.3 | 146 ms | 9.5 ms | 1.0 | 0 | 3,137 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 203.3 | 102.8 | 199 ms | 9.7 ms | 2.0 | 0 | 6,095 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 375.8 | 95.4 | 256 ms | 10.5 ms | 4.0 | 0 | 11,274 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 671.1 | 85.2 | 407 ms | 11.7 ms | 8.0 | 0 | 20,132 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,176.4 | 74.3 | 599 ms | 13.5 ms | 16.0 | 0 | 35,269 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,470.9 | 46.4 | 1,007 ms | 21.5 ms | 32.0 | 0 | 44,058 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,070.6 | 32.2 | 1,872 ms | 31.1 ms | 64.0 | 0 | 61,975 | 29.9 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 65.3 | 66.3 | 183 ms | 15.1 ms | 1.0 | 0 | 1,958 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 130.1 | 65.8 | 268 ms | 15.2 ms | 2.0 | 0 | 3,903 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 245.6 | 62.6 | 369 ms | 16.0 ms | 4.0 | 0 | 7,366 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 451.6 | 57.4 | 558 ms | 17.4 ms | 8.0 | 0 | 13,541 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 833.2 | 52.4 | 959 ms | 19.1 ms | 16.0 | 0 | 24,987 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 972.6 | 30.5 | 1,612 ms | 32.8 ms | 32.0 | 0 | 29,178 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 35.0 | 35.9 | 325 ms | 27.8 ms | 1.0 | 0 | 1,048 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 77.8 | 39.7 | 466 ms | 25.2 ms | 2.0 | 0 | 2,332 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 144.6 | 36.7 | 653 ms | 27.2 ms | 4.0 | 0 | 4,338 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 280.1 | 35.5 | 1,019 ms | 28.2 ms | 8.0 | 0 | 8,400 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 521.9 | 32.8 | 1,788 ms | 30.5 ms | 16.0 | 0 | 15,627 | 29.9 s |
| Sustained decode | 128K | 32 | Valid | 581.4 | 18.5 | 3,117 ms | 54.0 ms | 32.0 | 0 | 17,430 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 21.6 | 22.0 | 633 ms | 45.5 ms | 1.0 | 0 | 645 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 42.1 | 21.6 | 943 ms | 46.3 ms | 2.0 | 0 | 1,263 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 79.7 | 20.5 | 1,415 ms | 48.7 ms | 4.0 | 0 | 2,389 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 156.5 | 19.9 | 2,232 ms | 50.2 ms | 8.0 | 0 | 4,692 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 307.3 | 19.2 | 3,753 ms | 52.0 ms | 16.0 | 0 | 9,195 | 29.9 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.