Speed test matrix
Qwen3.5-35B-A3B-GPTQ-Int4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --moe-backend triton --quantization moe_wna16
Prefill
prompt tok/s by context length8K context13.2K623 ms TTFT
16K context13.1K1,241 ms TTFT
32K context13.2K2,444 ms TTFT
64K context12.4K5,207 ms TTFT
128K context10.3K12,498 ms TTFT
251K context7,85932,094 ms TTFT
256K context7,77333,092 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 117.5136 ms TTFT | 157.9219 ms TTFT | 249.5337 ms TTFT | 371.4532 ms TTFT | 527.8992 ms TTFT | 778.01,244 ms TTFT | 1,159.42,322 ms TTFT | 1,682.03,953 ms TTFT |
| 16K | 116.1129 ms TTFT | 156.8193 ms TTFT | 246.9308 ms TTFT | 364.9416 ms TTFT | 511.2707 ms TTFT | 764.8881 ms TTFT | 1,146.31,863 ms TTFT | 1,637.42,604 ms TTFT |
| 32K | 113.2171 ms TTFT | 154.2274 ms TTFT | 248.0388 ms TTFT | 359.8592 ms TTFT | 504.7915 ms TTFT | 736.01,321 ms TTFT | 1,072.92,399 ms TTFT | Over capacity |
| 64K | 107.8127 ms TTFT | 148.9195 ms TTFT | 240.8252 ms TTFT | 352.1414 ms TTFT | 486.8770 ms TTFT | 693.71,440 ms TTFT | Over capacity | Over capacity |
| 128K | 97.8443 ms TTFT | 139.5581 ms TTFT | 224.5865 ms TTFT | 326.61,302 ms TTFT | 445.42,179 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 83.2766 ms TTFT | 123.91,048 ms TTFT | 205.21,466 ms TTFT | 289.82,230 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.2K | — | 623 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.1K | — | 1,241 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.2K | — | 2,444 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.4K | — | 5,207 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.3K | — | 12,498 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 7,859.0 | — | 32,094 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,773.0 | — | 33,092 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 117.5 | 117.6 | 136 ms | 8.5 ms | 1.0 | 0 | 3,525 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 157.9 | 79.0 | 219 ms | 12.7 ms | 2.0 | 0 | 4,738 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 249.5 | 63.2 | 337 ms | 15.8 ms | 4.0 | 0 | 7,484 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 371.4 | 47.3 | 532 ms | 21.1 ms | 8.0 | 0 | 11,136 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 527.8 | 34.5 | 992 ms | 29.0 ms | 16.0 | 0 | 15,824 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 778.0 | 25.3 | 1,244 ms | 39.6 ms | 32.0 | 0 | 23,328 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,159.4 | 18.4 | 2,322 ms | 54.3 ms | 64.0 | 0 | 34,752 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,682.0 | 12.6 | 3,953 ms | 79.1 ms | 128.0 | 0 | 50,432 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 116.1 | 116.2 | 129 ms | 8.6 ms | 1.0 | 0 | 3,484 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 156.8 | 78.4 | 193 ms | 12.8 ms | 2.0 | 0 | 4,702 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 246.9 | 62.3 | 308 ms | 16.0 ms | 4.0 | 0 | 7,408 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 364.9 | 46.3 | 416 ms | 21.6 ms | 8.0 | 0 | 10,944 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 511.2 | 33.2 | 707 ms | 30.1 ms | 16.0 | 0 | 15,328 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 764.8 | 24.9 | 881 ms | 40.1 ms | 32.0 | 0 | 22,944 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,146.3 | 18.7 | 1,863 ms | 53.5 ms | 64.0 | 0 | 34,368 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,637.4 | 12.8 | 2,604 ms | 77.9 ms | 128.0 | 0 | 49,024 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 113.2 | 113.2 | 171 ms | 8.8 ms | 1.0 | 0 | 3,394 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 154.2 | 77.1 | 274 ms | 13.0 ms | 2.0 | 0 | 4,624 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 248.0 | 62.2 | 388 ms | 16.1 ms | 4.0 | 0 | 7,436 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 359.8 | 45.7 | 592 ms | 21.9 ms | 8.0 | 0 | 10,792 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 504.7 | 32.5 | 915 ms | 30.8 ms | 16.0 | 0 | 15,136 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 736.0 | 23.8 | 1,321 ms | 42.1 ms | 32.0 | 0 | 22,048 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,072.9 | 17.0 | 2,399 ms | 58.7 ms | 64.0 | 0 | 32,128 | 29.9 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 107.8 | 107.8 | 127 ms | 9.3 ms | 1.0 | 0 | 3,233 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 148.9 | 74.5 | 195 ms | 13.4 ms | 2.0 | 0 | 4,464 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 240.8 | 60.4 | 252 ms | 16.6 ms | 4.0 | 0 | 7,220 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 352.1 | 44.1 | 414 ms | 22.7 ms | 8.0 | 0 | 10,560 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 486.8 | 30.6 | 770 ms | 32.7 ms | 16.0 | 0 | 14,592 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 693.7 | 21.9 | 1,440 ms | 45.7 ms | 32.0 | 0 | 20,800 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 97.8 | 97.9 | 443 ms | 10.2 ms | 1.0 | 0 | 2,933 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 139.5 | 69.6 | 581 ms | 14.4 ms | 2.0 | 0 | 4,184 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 224.5 | 56.2 | 865 ms | 17.8 ms | 4.0 | 0 | 6,736 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 326.6 | 40.5 | 1,302 ms | 24.7 ms | 8.0 | 0 | 9,792 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 445.4 | 27.6 | 2,179 ms | 36.3 ms | 16.0 | 0 | 13,360 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 83.2 | 83.3 | 766 ms | 12.0 ms | 1.0 | 0 | 2,495 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 123.9 | 61.8 | 1,048 ms | 16.2 ms | 2.0 | 0 | 3,718 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 205.2 | 50.7 | 1,466 ms | 19.7 ms | 4.0 | 0 | 6,152 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 289.8 | 35.4 | 2,230 ms | 28.3 ms | 8.0 | 0 | 8,688 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.