Speed test matrix
Qwen3.5-35B-A3B-GPTQ-Int4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-35B-A3B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --moe-backend triton --speculative-config '{"method":"mtp","num_speculative_tokens":2}' --quantization moe_wna16Prefill
prompt tok/s by context length8K context8,734938 ms TTFT
16K context13.6K1,197 ms TTFT
32K context12.8K2,514 ms TTFT
64K context11.9K5,436 ms TTFT
128K context9,97912,897 ms TTFT
251K context7,44433,880 ms TTFT
256K context7,38534,832 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 152.9187 ms TTFT | 237.1339 ms TTFT | 347.8628 ms TTFT | 521.1890 ms TTFT | 785.21,522 ms TTFT | 1,149.32,621 ms TTFT | 1,633.44,591 ms TTFT | 2,739.28,387 ms TTFT |
| 16K | 149.0173 ms TTFT | 232.1310 ms TTFT | 344.8474 ms TTFT | 517.4758 ms TTFT | 761.21,294 ms TTFT | 1,131.82,110 ms TTFT | 1,586.03,504 ms TTFT | Over capacity |
| 32K | 146.4208 ms TTFT | 234.5346 ms TTFT | 330.6554 ms TTFT | 498.9987 ms TTFT | 770.71,492 ms TTFT | 1,100.62,578 ms TTFT | Over capacity | Over capacity |
| 64K | Looped | 223.8444 ms TTFT | 328.7689 ms TTFT | Looped | 718.31,906 ms TTFT | 1,022.22,921 ms TTFT | Over capacity | Over capacity |
| 128K | 133.3441 ms TTFT | 214.5595 ms TTFT | 323.1968 ms TTFT | Looped | 668.02,492 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 118.0799 ms TTFT | 191.51,085 ms TTFT | 297.11,439 ms TTFT | 425.72,291 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,734.0 | — | 938 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.6K | — | 1,197 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.8K | — | 2,514 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.9K | — | 5,436 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,979.0 | — | 12,897 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 7,444.0 | — | 33,880 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,385.0 | — | 34,832 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 152.9 | 153.7 | 187 ms | 6.5 ms | 1.0 | 1 | 4,587 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 237.1 | 119.5 | 339 ms | 8.4 ms | 2.0 | 1 | 7,096 | 29.9 s |
| Sustained decode | 8K | 4 | Valid | 347.8 | 87.7 | 628 ms | 11.4 ms | 4.0 | 0 | 10,430 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 521.1 | 66.2 | 890 ms | 15.1 ms | 8.0 | 0 | 15,623 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 785.2 | 50.1 | 1,522 ms | 20.0 ms | 16.0 | 0 | 23,533 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,149.3 | 36.0 | 2,621 ms | 27.7 ms | 32.0 | 0 | 34,455 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,633.4 | 24.4 | 4,591 ms | 41.0 ms | 64.0 | 0 | 48,925 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,739.2 | 18.8 | 8,387 ms | 53.1 ms | 128.0 | 0 | 82,033 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 149.0 | 151.4 | 173 ms | 6.6 ms | 1.0 | 1 | 4,471 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 232.1 | 117.2 | 310 ms | 8.5 ms | 2.0 | 0 | 6,960 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 344.8 | 87.9 | 474 ms | 11.4 ms | 4.0 | 0 | 10,344 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 517.4 | 65.8 | 758 ms | 15.2 ms | 8.0 | 0 | 15,516 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 761.2 | 48.9 | 1,294 ms | 20.5 ms | 16.0 | 0 | 22,806 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,131.8 | 35.8 | 2,110 ms | 27.9 ms | 32.0 | 0 | 33,921 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,586.0 | 24.2 | 3,504 ms | 41.3 ms | 64.0 | 0 | 47,464 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 146.4 | 151.3 | 208 ms | 6.6 ms | 1.0 | 1 | 4,392 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 234.5 | 118.1 | 346 ms | 8.5 ms | 2.0 | 0 | 7,033 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 330.6 | 84.6 | 554 ms | 11.8 ms | 4.0 | 0 | 9,912 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 498.9 | 63.7 | 987 ms | 15.7 ms | 8.0 | 0 | 14,966 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 770.7 | 49.1 | 1,492 ms | 20.4 ms | 16.0 | 0 | 23,114 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,100.6 | 34.5 | 2,578 ms | 29.0 ms | 32.0 | 0 | 32,988 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 2 | Valid | 223.8 | 113.1 | 444 ms | 8.8 ms | 2.0 | 0 | 6,712 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 328.7 | 83.5 | 689 ms | 12.0 ms | 4.0 | 0 | 9,859 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 718.3 | 45.7 | 1,906 ms | 21.9 ms | 16.0 | 0 | 21,522 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,022.2 | 32.4 | 2,921 ms | 30.9 ms | 32.0 | 0 | 30,640 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 133.3 | 142.0 | 441 ms | 7.1 ms | 1.0 | 1 | 3,997 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 214.5 | 108.0 | 595 ms | 9.3 ms | 2.0 | 0 | 6,432 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 323.1 | 81.1 | 968 ms | 12.3 ms | 4.0 | 0 | 9,687 | 30.0 s |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Valid | 668.0 | 41.4 | 2,492 ms | 24.1 ms | 16.0 | 0 | 20,038 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 118.0 | 117.9 | 799 ms | 8.5 ms | 1.0 | 0 | 3,539 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 191.5 | 95.9 | 1,085 ms | 10.4 ms | 2.0 | 0 | 5,743 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 297.1 | 74.3 | 1,439 ms | 13.5 ms | 4.0 | 0 | 8,905 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 425.7 | 51.9 | 2,291 ms | 19.3 ms | 8.0 | 0 | 12,761 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.