Speed test matrix
Qwen3.5-397B-A17B-FP8 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-397B-A17B-FP8 --served-model-name qwen --trust-remote-code --tensor-parallel-size 8 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend cutlass --moe-backend triton
Prefill
prompt tok/s by context length8K context10K817 ms TTFT
16K context10.4K1,567 ms TTFT
32K context10.2K3,165 ms TTFT
64K context10.1K6,403 ms TTFT
128K context9,45713,609 ms TTFT
251K context8,42929,922 ms TTFT
256K context8,31930,917 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 114.398 ms TTFT | 186.4139 ms TTFT | 329.7821 ms TTFT | 562.6333 ms TTFT | 810.4423 ms TTFT | 1,197.11,060 ms TTFT | 1,615.51,329 ms TTFT | 2,496.92,266 ms TTFT |
| 16K | 113.0126 ms TTFT | 186.1198 ms TTFT | 326.8303 ms TTFT | 554.3415 ms TTFT | 793.9800 ms TTFT | 1,162.71,160 ms TTFT | 1,549.01,877 ms TTFT | Over capacity |
| 32K | 111.3125 ms TTFT | 181.1182 ms TTFT | 319.7227 ms TTFT | 538.7337 ms TTFT | 762.8491 ms TTFT | 1,094.2792 ms TTFT | 1,429.11,366 ms TTFT | Over capacity |
| 64K | 107.4190 ms TTFT | 173.7242 ms TTFT | 305.9300 ms TTFT | 504.8456 ms TTFT | 699.4792 ms TTFT | 973.41,424 ms TTFT | Over capacity | Over capacity |
| 128K | 100.3413 ms TTFT | 159.9544 ms TTFT | 283.1720 ms TTFT | 455.81,099 ms TTFT | 601.41,886 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 89.2790 ms TTFT | 139.61,059 ms TTFT | 236.11,479 ms TTFT | 379.42,362 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10K | — | 817 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.4K | — | 1,567 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.2K | — | 3,165 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.1K | — | 6,403 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,457.0 | — | 13,609 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,429.0 | — | 29,922 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,319.0 | — | 30,917 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 114.3 | 114.2 | 98 ms | 8.8 ms | 1.0 | 0 | 3,427 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 186.4 | 93.3 | 139 ms | 10.7 ms | 2.0 | 0 | 5,590 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 329.7 | 82.6 | 821 ms | 12.1 ms | 4.0 | 0 | 9,888 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 562.6 | 70.8 | 333 ms | 14.1 ms | 8.0 | 0 | 16,872 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 810.4 | 51.3 | 423 ms | 19.5 ms | 16.0 | 0 | 24,304 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,197.1 | 38.4 | 1,060 ms | 26.0 ms | 32.0 | 0 | 35,904 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,615.5 | 25.7 | 1,329 ms | 38.9 ms | 64.0 | 0 | 48,448 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,496.9 | 19.5 | 2,266 ms | 51.2 ms | 128.0 | 0 | 74,880 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 113.0 | 113.0 | 126 ms | 8.9 ms | 1.0 | 0 | 3,391 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 186.1 | 93.2 | 198 ms | 10.7 ms | 2.0 | 0 | 5,582 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 326.8 | 81.9 | 303 ms | 12.2 ms | 4.0 | 0 | 9,804 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 554.3 | 69.7 | 415 ms | 14.4 ms | 8.0 | 0 | 16,624 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 793.9 | 50.5 | 800 ms | 19.8 ms | 16.0 | 0 | 23,808 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,162.7 | 37.0 | 1,160 ms | 27.0 ms | 32.0 | 0 | 34,881 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,549.0 | 24.4 | 1,877 ms | 41.0 ms | 64.0 | 0 | 46,464 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 111.3 | 111.2 | 125 ms | 9.0 ms | 1.0 | 0 | 3,339 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 181.1 | 90.6 | 182 ms | 11.0 ms | 2.0 | 0 | 5,432 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 319.7 | 79.9 | 227 ms | 12.5 ms | 4.0 | 0 | 9,588 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 538.7 | 67.8 | 337 ms | 14.8 ms | 8.0 | 0 | 16,160 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 762.8 | 47.9 | 491 ms | 20.9 ms | 16.0 | 0 | 22,880 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,094.2 | 34.4 | 792 ms | 29.1 ms | 32.0 | 0 | 32,800 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,429.1 | 22.2 | 1,366 ms | 45.1 ms | 64.0 | 0 | 42,816 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 107.4 | 107.5 | 190 ms | 9.3 ms | 1.0 | 0 | 3,222 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 173.7 | 86.9 | 242 ms | 11.5 ms | 2.0 | 0 | 5,208 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 305.9 | 76.4 | 300 ms | 13.1 ms | 4.0 | 0 | 9,176 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 504.8 | 63.0 | 456 ms | 15.9 ms | 8.0 | 0 | 15,145 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 699.4 | 43.5 | 792 ms | 23.0 ms | 16.0 | 0 | 20,976 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 973.4 | 30.2 | 1,424 ms | 33.2 ms | 32.0 | 0 | 29,184 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 100.3 | 100.4 | 413 ms | 10.0 ms | 1.0 | 0 | 3,008 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 159.9 | 79.9 | 544 ms | 12.5 ms | 2.0 | 0 | 4,796 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 283.1 | 70.4 | 720 ms | 14.2 ms | 4.0 | 0 | 8,492 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 455.8 | 56.2 | 1,099 ms | 17.8 ms | 8.0 | 0 | 13,672 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 601.4 | 36.7 | 1,886 ms | 27.3 ms | 16.0 | 0 | 18,032 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 89.2 | 89.4 | 790 ms | 11.2 ms | 1.0 | 0 | 2,676 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 139.6 | 69.7 | 1,059 ms | 14.3 ms | 2.0 | 0 | 4,188 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 236.1 | 58.5 | 1,479 ms | 17.1 ms | 4.0 | 0 | 7,080 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 379.4 | 46.4 | 2,362 ms | 21.5 ms | 8.0 | 0 | 11,376 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.