Speed test matrix
Qwen3.5-27B-GPTQ-Int4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- qwen
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-qwen-vllm-73a5831127a9/bin/vllm" serve Qwen/Qwen3.5-27B-GPTQ-Int4 --served-model-name qwen --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.92 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend marlin --speculative-config '{"method":"mtp","num_speculative_tokens":2}'Prefill
prompt tok/s by context length8K context6,1451,333 ms TTFT
16K context6,0282,691 ms TTFT
32K context5,6695,697 ms TTFT
64K context5,04512,769 ms TTFT
128K context4,13631,115 ms TTFT
251K context3,07182,117 ms TTFT
256K context3,04384,534 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 75.3188 ms TTFT | 154.1758 ms TTFT | 298.41,536 ms TTFT | 470.52,691 ms TTFT | 772.22,263 ms TTFT | 1,052.73,755 ms TTFT | 1,297.96,482 ms TTFT | Over capacity |
| 16K | 64.1216 ms TTFT | 131.5389 ms TTFT | 259.0758 ms TTFT | 385.11,356 ms TTFT | 602.42,813 ms TTFT | 791.75,056 ms TTFT | Over capacity | Over capacity |
| 32K | 52.1282 ms TTFT | 103.9492 ms TTFT | 204.9949 ms TTFT | 277.01,680 ms TTFT | 414.13,327 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 37.1408 ms TTFT | 72.8683 ms TTFT | 147.71,256 ms TTFT | 177.42,577 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 23.3747 ms TTFT | 47.01,225 ms TTFT | 90.72,145 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 13.61,071 ms TTFT | 26.11,521 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,145.0 | — | 1,333 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,028.0 | — | 2,691 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,669.0 | — | 5,697 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,045.0 | — | 12,769 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,136.0 | — | 31,115 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,071.0 | — | 82,117 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,043.0 | — | 84,534 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 75.3 | 75.4 | 188 ms | 13.3 ms | 1.0 | 0 | 2,257 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 154.1 | 76.7 | 758 ms | 13.1 ms | 2.0 | 0 | 4,621 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 298.4 | 75.2 | 1,536 ms | 13.3 ms | 4.0 | 0 | 8,944 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 470.5 | 59.1 | 2,691 ms | 16.9 ms | 8.0 | 0 | 14,101 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 772.2 | 47.8 | 2,263 ms | 20.9 ms | 16.0 | 0 | 23,161 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,052.7 | 32.5 | 3,755 ms | 30.8 ms | 32.0 | 0 | 31,557 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,297.9 | 18.8 | 6,482 ms | 53.2 ms | 64.0 | 0 | 38,803 | 29.9 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 64.1 | 65.3 | 216 ms | 15.3 ms | 1.0 | 0 | 1,924 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 131.5 | 66.5 | 389 ms | 15.0 ms | 2.0 | 0 | 3,942 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 259.0 | 65.1 | 758 ms | 15.4 ms | 4.0 | 0 | 7,769 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 385.1 | 48.6 | 1,356 ms | 20.6 ms | 8.0 | 0 | 11,550 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 602.4 | 37.6 | 2,813 ms | 26.6 ms | 16.0 | 0 | 18,061 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 791.7 | 24.3 | 5,056 ms | 41.1 ms | 32.0 | 0 | 23,693 | 29.9 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 52.1 | 52.6 | 282 ms | 19.0 ms | 1.0 | 0 | 1,560 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 103.9 | 52.8 | 492 ms | 19.0 ms | 2.0 | 0 | 3,114 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 204.9 | 51.5 | 949 ms | 19.4 ms | 4.0 | 0 | 6,147 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 277.0 | 35.0 | 1,680 ms | 28.6 ms | 8.0 | 0 | 8,311 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 414.1 | 25.9 | 3,327 ms | 38.6 ms | 16.0 | 0 | 12,398 | 29.9 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 37.1 | 37.4 | 408 ms | 26.8 ms | 1.0 | 0 | 1,112 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 72.8 | 36.5 | 683 ms | 27.4 ms | 2.0 | 0 | 2,181 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 147.7 | 37.0 | 1,256 ms | 27.0 ms | 4.0 | 0 | 4,424 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 177.4 | 22.3 | 2,577 ms | 44.9 ms | 8.0 | 0 | 5,312 | 29.9 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 23.3 | 23.6 | 747 ms | 42.3 ms | 1.0 | 0 | 699 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 47.0 | 23.7 | 1,225 ms | 42.2 ms | 2.0 | 0 | 1,410 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 90.7 | 22.8 | 2,145 ms | 43.9 ms | 4.0 | 0 | 2,721 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 13.6 | 13.8 | 1,071 ms | 72.3 ms | 1.0 | 0 | 409 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 26.1 | 13.2 | 1,521 ms | 75.5 ms | 2.0 | 0 | 782 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.