Speed test matrix
Qwen3.5-122B-A10B-GPTQ-Int4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.30.1rc1.dev452+g73a583112 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend marlin --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":2}' --quantization moe_wna16Prefill
prompt tok/s by context length8K context9,619851 ms TTFT
16K context9,9531,630 ms TTFT
32K context9,6893,333 ms TTFT
64K context9,4146,843 ms TTFT
128K context8,57215,013 ms TTFT
251K context7,31034,499 ms TTFT
256K context7,22835,586 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 115.0239 ms TTFT | 184.6366 ms TTFT | 283.3882 ms TTFT | 424.2970 ms TTFT | 625.91,690 ms TTFT | 928.03,129 ms TTFT | 1,378.26,043 ms TTFT | 2,131.612,000 ms TTFT |
| 16K | 97.9221 ms TTFT | 162.5355 ms TTFT | 257.9696 ms TTFT | 389.5960 ms TTFT | 592.61,686 ms TTFT | 860.42,783 ms TTFT | 1,257.05,337 ms TTFT | 1,866.19,875 ms TTFT |
| 32K | 73.3286 ms TTFT | 129.8454 ms TTFT | 214.9872 ms TTFT | 345.61,138 ms TTFT | 530.01,768 ms TTFT | 726.73,374 ms TTFT | 1,061.66,626 ms TTFT | Over capacity |
| 64K | 49.6294 ms TTFT | 90.7441 ms TTFT | 158.3666 ms TTFT | 270.21,131 ms TTFT | 434.21,590 ms TTFT | 565.12,808 ms TTFT | Over capacity | Over capacity |
| 128K | 30.0584 ms TTFT | 58.0832 ms TTFT | 105.11,289 ms TTFT | 187.72,254 ms TTFT | 326.73,757 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 16.91,012 ms TTFT | 34.31,322 ms TTFT | 63.11,934 ms TTFT | 118.83,226 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,619.0 | — | 851 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,953.0 | — | 1,630 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,689.0 | — | 3,333 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,414.0 | — | 6,843 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,572.0 | — | 15,013 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 7,310.0 | — | 34,499 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,228.0 | — | 35,586 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 115.0 | 116.0 | 239 ms | 8.6 ms | 1.0 | 0 | 3,449 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 184.6 | 90.7 | 366 ms | 11.0 ms | 2.0 | 0 | 5,539 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 283.3 | 70.1 | 882 ms | 14.3 ms | 4.0 | 0 | 8,497 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 424.2 | 52.6 | 970 ms | 19.0 ms | 8.0 | 0 | 12,711 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 625.9 | 38.6 | 1,690 ms | 25.9 ms | 16.0 | 0 | 18,744 | 29.9 s |
| Sustained decode | 8K | 32 | Valid | 928.0 | 28.4 | 3,129 ms | 35.2 ms | 32.0 | 0 | 27,821 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,378.2 | 19.5 | 6,043 ms | 51.3 ms | 64.0 | 0 | 41,302 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,131.6 | 13.5 | 12,000 ms | 73.8 ms | 128.0 | 0 | 63,691 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 97.9 | 98.6 | 221 ms | 10.1 ms | 1.0 | 0 | 2,936 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 162.5 | 82.1 | 355 ms | 12.2 ms | 2.0 | 0 | 4,875 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 257.9 | 65.3 | 696 ms | 15.3 ms | 4.0 | 0 | 7,737 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 389.5 | 49.4 | 960 ms | 20.2 ms | 8.0 | 0 | 11,682 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 592.6 | 37.2 | 1,686 ms | 26.9 ms | 16.0 | 0 | 17,747 | 29.9 s |
| Sustained decode | 16K | 32 | Valid | 860.4 | 26.3 | 2,783 ms | 38.1 ms | 32.0 | 0 | 25,772 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,257.0 | 18.4 | 5,337 ms | 54.4 ms | 64.0 | 0 | 37,564 | 29.9 s |
| Sustained decode | 16K | 128 | Valid | 1,866.1 | 12.3 | 9,875 ms | 81.1 ms | 128.0 | 0 | 55,791 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 73.3 | 74.4 | 286 ms | 13.4 ms | 1.0 | 0 | 2,198 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 129.8 | 65.3 | 454 ms | 15.3 ms | 2.0 | 0 | 3,890 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 214.9 | 54.1 | 872 ms | 18.5 ms | 4.0 | 0 | 6,444 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 345.6 | 43.6 | 1,138 ms | 22.9 ms | 8.0 | 0 | 10,365 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 530.0 | 33.0 | 1,768 ms | 30.3 ms | 16.0 | 0 | 15,860 | 29.9 s |
| Sustained decode | 32K | 32 | Valid | 726.7 | 22.4 | 3,374 ms | 44.6 ms | 32.0 | 0 | 21,790 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,061.6 | 15.1 | 6,626 ms | 66.1 ms | 64.0 | 0 | 31,827 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 49.6 | 49.8 | 294 ms | 20.1 ms | 1.0 | 0 | 1,488 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 90.7 | 45.8 | 441 ms | 21.8 ms | 2.0 | 0 | 2,718 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 158.3 | 40.0 | 666 ms | 25.0 ms | 4.0 | 0 | 4,744 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 270.2 | 34.2 | 1,131 ms | 29.3 ms | 8.0 | 0 | 8,091 | 29.9 s |
| Sustained decode | 64K | 16 | Valid | 434.2 | 27.5 | 1,590 ms | 36.4 ms | 16.0 | 0 | 12,993 | 29.9 s |
| Sustained decode | 64K | 32 | Valid | 565.1 | 17.6 | 2,808 ms | 56.8 ms | 32.0 | 0 | 16,919 | 29.9 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 30.0 | 30.5 | 584 ms | 32.7 ms | 1.0 | 0 | 899 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 58.0 | 29.2 | 832 ms | 34.2 ms | 2.0 | 0 | 1,741 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 105.1 | 26.4 | 1,289 ms | 37.9 ms | 4.0 | 0 | 3,146 | 29.9 s |
| Sustained decode | 128K | 8 | Valid | 187.7 | 23.5 | 2,254 ms | 42.6 ms | 8.0 | 0 | 5,625 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 326.7 | 19.9 | 3,757 ms | 50.3 ms | 16.0 | 0 | 9,773 | 29.9 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 16.9 | 17.3 | 1,012 ms | 57.8 ms | 1.0 | 0 | 505 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 34.3 | 17.5 | 1,322 ms | 57.3 ms | 2.0 | 0 | 1,027 | 29.9 s |
| Sustained decode | 251K | 4 | Valid | 63.1 | 16.1 | 1,934 ms | 62.0 ms | 4.0 | 0 | 1,885 | 29.9 s |
| Sustained decode | 251K | 8 | Valid | 118.8 | 15.0 | 3,226 ms | 66.7 ms | 8.0 | 0 | 3,550 | 29.9 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.