Speed test matrix
Qwen3.5-122B-A10B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-122B-A10B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass --moe-backend triton
Prefill
prompt tok/s by context length8K context15.6K525 ms TTFT
16K context15.5K1,045 ms TTFT
32K context15.4K2,093 ms TTFT
64K context14.6K4,413 ms TTFT
128K context12.8K10,048 ms TTFT
251K context10.4K24,331 ms TTFT
256K context10.2K25,275 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 149.299 ms TTFT | 226.7144 ms TTFT | 419.4282 ms TTFT | 693.4712 ms TTFT | 1,051.5680 ms TTFT | 1,541.61,185 ms TTFT | 2,115.71,915 ms TTFT | 3,193.03,671 ms TTFT |
| 16K | 147.496 ms TTFT | 225.9153 ms TTFT | 418.4213 ms TTFT | 687.5258 ms TTFT | 1,015.5537 ms TTFT | 1,489.51,024 ms TTFT | 2,014.61,223 ms TTFT | 2,981.52,602 ms TTFT |
| 32K | 144.2139 ms TTFT | 220.8223 ms TTFT | 409.1281 ms TTFT | 663.8487 ms TTFT | 973.1770 ms TTFT | 1,389.61,364 ms TTFT | 1,856.52,369 ms TTFT | Over capacity |
| 64K | 139.0158 ms TTFT | 211.0230 ms TTFT | 389.8293 ms TTFT | 622.6455 ms TTFT | 896.1777 ms TTFT | 1,231.71,417 ms TTFT | Over capacity | Over capacity |
| 128K | 128.6380 ms TTFT | 193.8524 ms TTFT | 353.3693 ms TTFT | 556.51,092 ms TTFT | 767.51,697 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 115.0749 ms TTFT | 169.8970 ms TTFT | 291.11,362 ms TTFT | 464.92,196 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.6K | — | 525 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.5K | — | 1,045 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.4K | — | 2,093 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.6K | — | 4,413 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.8K | — | 10,048 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.4K | — | 24,331 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.2K | — | 25,275 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 149.2 | 149.9 | 99 ms | 6.7 ms | 1.0 | 1 | 4,475 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 226.7 | 113.4 | 144 ms | 8.8 ms | 2.0 | 0 | 6,798 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 419.4 | 105.2 | 282 ms | 9.5 ms | 4.0 | 0 | 12,580 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 693.4 | 87.1 | 712 ms | 11.5 ms | 8.0 | 0 | 20,800 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,051.5 | 66.2 | 680 ms | 15.1 ms | 16.0 | 0 | 31,536 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,541.6 | 48.3 | 1,185 ms | 20.7 ms | 32.0 | 0 | 46,240 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,115.7 | 32.6 | 1,915 ms | 30.7 ms | 64.0 | 0 | 63,424 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,193.0 | 23.6 | 3,671 ms | 42.4 ms | 128.0 | 0 | 95,744 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 147.4 | 148.0 | 96 ms | 6.8 ms | 1.0 | 1 | 4,420 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 225.9 | 113.1 | 153 ms | 8.8 ms | 2.0 | 0 | 6,776 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 418.4 | 104.9 | 213 ms | 9.5 ms | 4.0 | 0 | 12,552 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 687.5 | 86.2 | 258 ms | 11.6 ms | 8.0 | 0 | 20,624 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,015.5 | 64.1 | 537 ms | 15.6 ms | 16.0 | 0 | 30,459 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,489.5 | 47.1 | 1,024 ms | 21.2 ms | 32.0 | 0 | 44,672 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,014.6 | 31.5 | 1,223 ms | 31.7 ms | 64.0 | 0 | 60,416 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,981.5 | 23.1 | 2,602 ms | 43.3 ms | 128.0 | 0 | 89,472 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 144.2 | 145.1 | 139 ms | 6.9 ms | 1.0 | 1 | 4,325 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 220.8 | 110.3 | 223 ms | 9.1 ms | 2.0 | 0 | 6,622 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 409.1 | 102.3 | 281 ms | 9.8 ms | 4.0 | 0 | 12,272 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 663.8 | 82.9 | 487 ms | 12.1 ms | 8.0 | 0 | 19,912 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 973.1 | 61.0 | 770 ms | 16.4 ms | 16.0 | 0 | 29,184 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,389.6 | 43.6 | 1,364 ms | 23.0 ms | 32.0 | 0 | 41,664 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,856.5 | 28.7 | 2,369 ms | 34.9 ms | 64.0 | 0 | 55,697 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 139.0 | 140.1 | 158 ms | 7.1 ms | 1.0 | 1 | 4,168 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 211.0 | 105.5 | 230 ms | 9.5 ms | 2.0 | 0 | 6,328 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 389.8 | 97.5 | 293 ms | 10.3 ms | 4.0 | 0 | 11,696 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 622.6 | 77.8 | 455 ms | 12.9 ms | 8.0 | 0 | 18,680 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 896.1 | 55.7 | 777 ms | 17.9 ms | 16.0 | 0 | 26,880 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,231.7 | 38.0 | 1,417 ms | 26.3 ms | 32.0 | 0 | 36,928 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 128.6 | 131.7 | 380 ms | 7.6 ms | 1.0 | 1 | 3,858 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 193.8 | 96.8 | 524 ms | 10.3 ms | 2.0 | 0 | 5,814 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 353.3 | 87.9 | 693 ms | 11.4 ms | 4.0 | 0 | 10,600 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 556.5 | 68.7 | 1,092 ms | 14.5 ms | 8.0 | 0 | 16,688 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 767.5 | 46.7 | 1,697 ms | 21.4 ms | 16.0 | 0 | 23,024 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 115.0 | 115.2 | 749 ms | 8.7 ms | 1.0 | 0 | 3,449 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 169.8 | 84.8 | 970 ms | 11.8 ms | 2.0 | 0 | 5,094 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 291.1 | 72.3 | 1,362 ms | 13.8 ms | 4.0 | 0 | 8,732 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 464.9 | 57.1 | 2,196 ms | 17.5 ms | 8.0 | 0 | 13,944 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.