Speed test matrix
Qwen3.5-397B-A17B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-397B-A17B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context9,947823 ms TTFT
16K context10.3K1,580 ms TTFT
32K context10.2K3,156 ms TTFT
64K context9,8456,543 ms TTFT
128K context9,28113,866 ms TTFT
251K context8,25430,554 ms TTFT
256K context8,10831,723 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 129.5144 ms TTFT | 235.1216 ms TTFT | 423.2417 ms TTFT | 702.0830 ms TTFT | 1,033.6919 ms TTFT | 1,375.61,632 ms TTFT | 2,267.53,141 ms TTFT | 2,942.35,509 ms TTFT |
| 16K | 102.1166 ms TTFT | 198.0267 ms TTFT | 354.7475 ms TTFT | 603.2829 ms TTFT | 911.21,002 ms TTFT | 1,284.01,972 ms TTFT | 1,991.53,496 ms TTFT | Over capacity |
| 32K | 72.3191 ms TTFT | 142.1278 ms TTFT | 265.8450 ms TTFT | 472.1519 ms TTFT | 762.81,193 ms TTFT | 1,119.91,512 ms TTFT | Over capacity | Over capacity |
| 64K | 46.0247 ms TTFT | 93.6360 ms TTFT | 170.9500 ms TTFT | 328.1796 ms TTFT | 559.01,239 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 26.3444 ms TTFT | 53.1629 ms TTFT | 103.3874 ms TTFT | 204.71,365 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 14.9832 ms TTFT | 29.01,141 ms TTFT | 56.41,684 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,947.0 | — | 823 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.3K | — | 1,580 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.2K | — | 3,156 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,845.0 | — | 6,543 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,281.0 | — | 13,866 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,254.0 | — | 30,554 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,108.0 | — | 31,723 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 129.5 | 139.5 | 144 ms | 7.2 ms | 1.0 | 1 | 3,883 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 235.1 | 114.8 | 216 ms | 8.7 ms | 2.0 | 0 | 7,050 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 423.2 | 102.8 | 417 ms | 9.7 ms | 4.0 | 0 | 12,694 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 702.0 | 87.2 | 830 ms | 11.5 ms | 8.0 | 0 | 21,044 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,033.6 | 63.1 | 919 ms | 15.9 ms | 16.0 | 0 | 30,974 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,375.6 | 43.1 | 1,632 ms | 23.2 ms | 32.0 | 0 | 41,221 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,267.5 | 34.6 | 3,141 ms | 28.9 ms | 64.0 | 0 | 67,895 | 29.9 s |
| Sustained decode | 8K | 128 | Valid | 2,942.3 | 21.1 | 5,509 ms | 47.4 ms | 128.0 | 0 | 87,991 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 102.1 | 103.2 | 166 ms | 9.7 ms | 1.0 | 0 | 3,060 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 198.0 | 100.1 | 267 ms | 10.0 ms | 2.0 | 0 | 5,939 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 354.7 | 89.5 | 475 ms | 11.2 ms | 4.0 | 0 | 10,636 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 603.2 | 76.6 | 829 ms | 13.1 ms | 8.0 | 0 | 18,096 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 911.2 | 57.4 | 1,002 ms | 17.4 ms | 16.0 | 0 | 27,300 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,284.0 | 39.9 | 1,972 ms | 25.1 ms | 32.0 | 0 | 38,502 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,991.5 | 30.0 | 3,496 ms | 33.3 ms | 64.0 | 0 | 59,679 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 72.3 | 73.1 | 191 ms | 13.7 ms | 1.0 | 0 | 2,166 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 142.1 | 72.0 | 278 ms | 13.9 ms | 2.0 | 0 | 4,264 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 265.8 | 67.1 | 450 ms | 14.9 ms | 4.0 | 0 | 7,971 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 472.1 | 59.5 | 519 ms | 16.8 ms | 8.0 | 0 | 14,150 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 762.8 | 48.1 | 1,193 ms | 20.8 ms | 16.0 | 0 | 22,864 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,119.9 | 35.1 | 1,512 ms | 28.5 ms | 32.0 | 0 | 33,575 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 46.0 | 46.7 | 247 ms | 21.4 ms | 1.0 | 0 | 1,379 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 93.6 | 47.6 | 360 ms | 21.0 ms | 2.0 | 0 | 2,805 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 170.9 | 43.0 | 500 ms | 23.3 ms | 4.0 | 0 | 5,120 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 328.1 | 41.3 | 796 ms | 24.2 ms | 8.0 | 0 | 9,825 | 29.9 s |
| Sustained decode | 64K | 16 | Valid | 559.0 | 35.1 | 1,239 ms | 28.5 ms | 16.0 | 0 | 16,764 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 26.3 | 26.8 | 444 ms | 37.4 ms | 1.0 | 0 | 786 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 53.1 | 27.1 | 629 ms | 36.9 ms | 2.0 | 0 | 1,590 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 103.3 | 25.7 | 874 ms | 38.9 ms | 4.0 | 0 | 3,091 | 29.9 s |
| Sustained decode | 128K | 8 | Valid | 204.7 | 25.9 | 1,365 ms | 38.6 ms | 8.0 | 0 | 6,126 | 29.9 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 14.9 | 15.1 | 832 ms | 66.1 ms | 1.0 | 0 | 445 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 29.0 | 14.7 | 1,141 ms | 67.9 ms | 2.0 | 0 | 865 | 29.8 s |
| Sustained decode | 251K | 4 | Valid | 56.4 | 14.4 | 1,684 ms | 69.7 ms | 4.0 | 0 | 1,691 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.