Speed test matrix
Qwen3.5-122B-A10B-GPTQ-Int4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.30.1rc1.dev452+g73a583112 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend marlin --moe-backend triton --quantization moe_wna16
Prefill
prompt tok/s by context length8K context9,274883 ms TTFT
16K context9,2291,758 ms TTFT
32K context9,4233,427 ms TTFT
64K context9,2366,975 ms TTFT
128K context8,46815,197 ms TTFT
251K context7,34134,354 ms TTFT
256K context7,25035,476 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 91.7163 ms TTFT | 128.8262 ms TTFT | 185.5453 ms TTFT | 304.1758 ms TTFT | 448.61,016 ms TTFT | 650.41,792 ms TTFT | 943.53,390 ms TTFT | 1,477.95,790 ms TTFT |
| 16K | 90.7147 ms TTFT | 128.5233 ms TTFT | 185.2340 ms TTFT | 302.2508 ms TTFT | 442.6575 ms TTFT | 642.01,490 ms TTFT | 924.91,844 ms TTFT | 1,431.73,478 ms TTFT |
| 32K | 90.1196 ms TTFT | 126.6301 ms TTFT | 181.6487 ms TTFT | 300.0559 ms TTFT | 433.01,409 ms TTFT | 627.21,582 ms TTFT | 886.02,674 ms TTFT | 1,344.35,318 ms TTFT |
| 64K | 87.8162 ms TTFT | 123.0255 ms TTFT | 179.5285 ms TTFT | 289.4444 ms TTFT | 410.3761 ms TTFT | 588.91,421 ms TTFT | 819.22,738 ms TTFT | Over capacity |
| 128K | 83.6441 ms TTFT | 117.4615 ms TTFT | 170.8806 ms TTFT | 275.21,350 ms TTFT | 384.82,164 ms TTFT | 536.83,750 ms TTFT | Over capacity | Over capacity |
| 251K | 77.0808 ms TTFT | 107.91,078 ms TTFT | 154.91,458 ms TTFT | 247.62,326 ms TTFT | 341.44,055 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,274.0 | — | 883 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,229.0 | — | 1,758 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,423.0 | — | 3,427 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,236.0 | — | 6,975 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,468.0 | — | 15,197 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 7,341.0 | — | 34,354 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,250.0 | — | 35,476 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 91.7 | 91.7 | 163 ms | 10.9 ms | 1.0 | 0 | 2,750 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 128.8 | 64.3 | 262 ms | 15.6 ms | 2.0 | 0 | 3,864 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 185.5 | 47.0 | 453 ms | 21.3 ms | 4.0 | 0 | 5,564 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 304.1 | 39.0 | 758 ms | 25.7 ms | 8.0 | 0 | 9,120 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 448.6 | 28.9 | 1,016 ms | 34.6 ms | 16.0 | 0 | 13,456 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 650.4 | 21.3 | 1,792 ms | 47.0 ms | 32.0 | 0 | 19,488 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 943.5 | 14.8 | 3,390 ms | 67.7 ms | 64.0 | 0 | 28,288 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,477.9 | 10.6 | 5,790 ms | 94.1 ms | 128.0 | 0 | 44,288 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 90.7 | 90.7 | 147 ms | 11.0 ms | 1.0 | 0 | 2,721 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 128.5 | 64.2 | 233 ms | 15.6 ms | 2.0 | 0 | 3,854 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 185.2 | 47.1 | 340 ms | 21.2 ms | 4.0 | 0 | 5,556 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 302.2 | 38.6 | 508 ms | 25.9 ms | 8.0 | 0 | 9,064 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 442.6 | 28.6 | 575 ms | 34.9 ms | 16.0 | 0 | 13,264 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 642.0 | 21.3 | 1,490 ms | 47.0 ms | 32.0 | 0 | 19,232 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 924.9 | 14.9 | 1,844 ms | 67.2 ms | 64.0 | 0 | 27,712 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,431.7 | 11.0 | 3,478 ms | 91.3 ms | 128.0 | 0 | 42,880 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 90.1 | 90.1 | 196 ms | 11.1 ms | 1.0 | 0 | 2,703 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 126.6 | 63.2 | 301 ms | 15.8 ms | 2.0 | 0 | 3,798 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 181.6 | 45.8 | 487 ms | 21.8 ms | 4.0 | 0 | 5,444 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 300.0 | 37.6 | 559 ms | 26.6 ms | 8.0 | 0 | 8,992 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 433.0 | 28.0 | 1,409 ms | 35.8 ms | 16.0 | 0 | 12,976 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 627.2 | 20.0 | 1,582 ms | 50.0 ms | 32.0 | 0 | 18,816 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 886.0 | 13.8 | 2,674 ms | 72.5 ms | 64.0 | 0 | 26,560 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 1,344.3 | 9.8 | 5,318 ms | 102.4 ms | 128.0 | 0 | 40,320 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 87.8 | 87.8 | 162 ms | 11.4 ms | 1.0 | 0 | 2,633 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 123.0 | 61.5 | 255 ms | 16.2 ms | 2.0 | 0 | 3,688 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 179.5 | 45.0 | 285 ms | 22.2 ms | 4.0 | 0 | 5,384 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 289.4 | 36.3 | 444 ms | 27.6 ms | 8.0 | 0 | 8,680 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 410.3 | 25.8 | 761 ms | 38.8 ms | 16.0 | 0 | 12,304 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 588.9 | 18.5 | 1,421 ms | 54.2 ms | 32.0 | 0 | 17,664 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 819.2 | 12.7 | 2,738 ms | 79.0 ms | 64.0 | 0 | 24,576 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 83.6 | 83.7 | 441 ms | 12.0 ms | 1.0 | 0 | 2,508 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 117.4 | 58.6 | 615 ms | 17.1 ms | 2.0 | 0 | 3,520 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 170.8 | 42.6 | 806 ms | 23.5 ms | 4.0 | 0 | 5,124 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 275.2 | 34.2 | 1,350 ms | 29.2 ms | 8.0 | 0 | 8,248 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 384.8 | 23.7 | 2,164 ms | 42.2 ms | 16.0 | 0 | 11,536 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 536.8 | 16.1 | 3,750 ms | 62.1 ms | 32.0 | 0 | 16,096 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 77.0 | 77.1 | 808 ms | 13.0 ms | 1.0 | 0 | 2,310 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 107.9 | 53.9 | 1,078 ms | 18.6 ms | 2.0 | 0 | 3,238 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 154.9 | 38.6 | 1,458 ms | 25.9 ms | 4.0 | 0 | 4,648 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 247.6 | 30.4 | 2,326 ms | 32.9 ms | 8.0 | 0 | 7,424 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 341.4 | 20.2 | 4,055 ms | 49.6 ms | 16.0 | 0 | 10,240 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.