Speed test matrix
Qwen3.5-397B-A17B-GPTQ-Int4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.30.1rc1.dev452+g73a583112 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-397B-A17B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend marlin --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":2}' --quantization moe_wna16Prefill
prompt tok/s by context length8K context5,1441,592 ms TTFT
16K context5,4522,975 ms TTFT
32K context5,3776,006 ms TTFT
64K context5,36212,013 ms TTFT
128K context5,04625,504 ms TTFT
251K context4,50655,975 ms TTFT
256K context4,46757,575 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 73.0471 ms TTFT | 116.6717 ms TTFT | 170.31,503 ms TTFT | 237.11,777 ms TTFT | 352.53,069 ms TTFT | 543.45,665 ms TTFT | 817.510,884 ms TTFT | 1,259.421,500 ms TTFT |
| 16K | 65.2431 ms TTFT | 105.7679 ms TTFT | 154.51,295 ms TTFT | 226.61,740 ms TTFT | 334.63,031 ms TTFT | 498.14,984 ms TTFT | 789.69,539 ms TTFT | Over capacity |
| 32K | 52.2509 ms TTFT | 88.6781 ms TTFT | 133.61,536 ms TTFT | 206.91,979 ms TTFT | 318.33,352 ms TTFT | 463.06,157 ms TTFT | Over capacity | Over capacity |
| 64K | 37.8487 ms TTFT | 67.1738 ms TTFT | 111.91,322 ms TTFT | 183.11,694 ms TTFT | 293.73,585 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 24.4833 ms TTFT | 44.31,212 ms TTFT | 80.02,310 ms TTFT | 138.82,626 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 14.11,271 ms TTFT | 28.41,754 ms TTFT | 49.82,599 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,144.0 | — | 1,592 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,452.0 | — | 2,975 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,377.0 | — | 6,006 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,362.0 | — | 12,013 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,046.0 | — | 25,504 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,506.0 | — | 55,975 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,467.0 | — | 57,575 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 73.0 | 73.9 | 471 ms | 13.5 ms | 1.0 | 0 | 2,191 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 116.6 | 59.2 | 717 ms | 16.9 ms | 2.0 | 0 | 3,496 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 170.3 | 44.2 | 1,503 ms | 22.6 ms | 4.0 | 0 | 5,106 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 237.1 | 31.4 | 1,777 ms | 31.8 ms | 8.0 | 0 | 7,100 | 29.9 s |
| Sustained decode | 8K | 16 | Valid | 352.5 | 22.1 | 3,069 ms | 45.2 ms | 16.0 | 0 | 10,556 | 29.9 s |
| Sustained decode | 8K | 32 | Valid | 543.4 | 15.7 | 5,665 ms | 63.5 ms | 32.0 | 0 | 16,217 | 29.8 s |
| Sustained decode | 8K | 64 | Valid | 817.5 | 10.6 | 10,884 ms | 94.4 ms | 64.0 | 0 | 24,513 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 1,259.4 | 7.0 | 21,500 ms | 142.3 ms | 128.0 | 0 | 37,683 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 65.2 | 66.0 | 431 ms | 15.2 ms | 1.0 | 0 | 1,955 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 105.7 | 53.3 | 679 ms | 18.8 ms | 2.0 | 0 | 3,167 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 154.5 | 39.5 | 1,295 ms | 25.3 ms | 4.0 | 0 | 4,626 | 29.9 s |
| Sustained decode | 16K | 8 | Valid | 226.6 | 29.7 | 1,740 ms | 33.7 ms | 8.0 | 0 | 6,789 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 334.6 | 21.7 | 3,031 ms | 46.2 ms | 16.0 | 0 | 10,018 | 29.9 s |
| Sustained decode | 16K | 32 | Valid | 498.1 | 15.2 | 4,984 ms | 65.7 ms | 32.0 | 0 | 14,861 | 29.8 s |
| Sustained decode | 16K | 64 | Valid | 789.6 | 10.8 | 9,539 ms | 92.9 ms | 64.0 | 0 | 23,629 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 52.2 | 53.0 | 509 ms | 18.9 ms | 1.0 | 0 | 1,563 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 88.6 | 44.8 | 781 ms | 22.3 ms | 2.0 | 0 | 2,655 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 133.6 | 35.2 | 1,536 ms | 28.4 ms | 4.0 | 0 | 3,998 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 206.9 | 27.0 | 1,979 ms | 37.0 ms | 8.0 | 0 | 6,189 | 29.9 s |
| Sustained decode | 32K | 16 | Valid | 318.3 | 20.2 | 3,352 ms | 49.4 ms | 16.0 | 0 | 9,515 | 29.9 s |
| Sustained decode | 32K | 32 | Valid | 463.0 | 13.8 | 6,157 ms | 72.6 ms | 32.0 | 0 | 13,804 | 29.8 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 37.8 | 38.4 | 487 ms | 26.1 ms | 1.0 | 0 | 1,134 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 67.1 | 33.9 | 738 ms | 29.5 ms | 2.0 | 0 | 2,011 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 111.9 | 28.8 | 1,322 ms | 34.7 ms | 4.0 | 0 | 3,350 | 29.9 s |
| Sustained decode | 64K | 8 | Valid | 183.1 | 23.2 | 1,694 ms | 43.0 ms | 8.0 | 0 | 5,488 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 293.7 | 18.7 | 3,585 ms | 53.5 ms | 16.0 | 0 | 8,783 | 29.9 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 24.4 | 24.8 | 833 ms | 40.3 ms | 1.0 | 0 | 732 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 44.3 | 22.5 | 1,212 ms | 44.4 ms | 2.0 | 0 | 1,329 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 80.0 | 20.4 | 2,310 ms | 49.1 ms | 4.0 | 0 | 2,398 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 138.8 | 17.2 | 2,626 ms | 58.1 ms | 8.0 | 0 | 4,151 | 29.9 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 14.1 | 14.4 | 1,271 ms | 69.5 ms | 1.0 | 0 | 424 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 28.4 | 14.2 | 1,754 ms | 70.2 ms | 2.0 | 0 | 849 | 29.9 s |
| Sustained decode | 251K | 4 | Valid | 49.8 | 12.3 | 2,599 ms | 81.1 ms | 4.0 | 0 | 1,491 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.