Speed test matrix
Qwen3.5-4B-AWQ-4bit on RTX 3060 12GB
vllm · 0.32.0 · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3060
- VRAM
- 12 GB
- CPU
- Intel Xeon E5-2680 v4 (14C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04 vllm serve cyankiwi/Qwen3.5-4B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context2,3433,477 ms TTFT
16K context1,9558,244 ms TTFT
32K context1,47721,724 ms TTFT
64K context990.064,676 ms TTFT
128K context600.0213,395 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 81.581 ms TTFT | 156.4145 ms TTFT | 288.8227 ms TTFT | 503.8365 ms TTFT | 796.0611 ms TTFT | 1,054.71,015 ms TTFT | 1,279.81,901 ms TTFT | Over capacity |
| 16K | 69.01,272 ms TTFT | 121.01,888 ms TTFT | 190.83,075 ms TTFT | 276.05,465 ms TTFT | 352.010,323 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 59.81,525 ms TTFT | 98.42,246 ms TTFT | 141.63,643 ms TTFT | 189.86,447 ms TTFT | 224.912,178 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 47.31,282 ms TTFT | 71.71,708 ms TTFT | 93.72,766 ms TTFT | 116.34,745 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 30.4Requests queued3,432 ms TTFT | 37.6Requests queued5,021 ms TTFT | 55.57,946 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,343.0 | — | 3,477 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 1,955.0 | — | 8,244 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,477.0 | — | 21,724 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 990.0 | — | 64,676 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 600.0 | — | 213,395 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 81.5 | 81.8 | 81 ms | 12.2 ms | 1.0 | 0 | 2,444 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 156.4 | 78.6 | 145 ms | 12.7 ms | 2.0 | 0 | 4,690 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 288.8 | 72.7 | 227 ms | 13.7 ms | 4.0 | 0 | 8,660 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 503.8 | 63.6 | 365 ms | 15.7 ms | 8.0 | 0 | 15,112 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 796.0 | 50.4 | 611 ms | 19.8 ms | 16.0 | 0 | 23,888 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,054.7 | 33.2 | 1,015 ms | 30.1 ms | 32.0 | 0 | 31,648 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,279.8 | 20.0 | 1,901 ms | 50.0 ms | 64.0 | 0 | 38,400 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 69.0 | 69.2 | 1,272 ms | 14.4 ms | 1.0 | 0 | 2,072 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 121.0 | 59.6 | 1,888 ms | 16.8 ms | 2.0 | 0 | 3,628 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 190.8 | 45.6 | 3,075 ms | 21.9 ms | 4.0 | 0 | 5,723 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 276.0 | 30.9 | 5,465 ms | 32.4 ms | 8.0 | 0 | 8,280 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 352.0 | 17.6 | 10,323 ms | 56.7 ms | 16.0 | 0 | 10,560 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 59.8 | 60.0 | 1,525 ms | 16.7 ms | 1.0 | 0 | 1,793 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 98.4 | 48.4 | 2,246 ms | 20.7 ms | 2.0 | 0 | 2,950 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 141.6 | 33.6 | 3,643 ms | 29.8 ms | 4.0 | 0 | 4,248 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 189.8 | 20.9 | 6,447 ms | 47.8 ms | 8.0 | 0 | 5,688 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 224.9 | 11.0 | 12,178 ms | 91.0 ms | 16.0 | 0 | 6,736 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 47.3 | 47.5 | 1,282 ms | 21.0 ms | 1.0 | 0 | 1,420 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 71.7 | 35.5 | 1,708 ms | 28.2 ms | 2.0 | 0 | 2,150 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 93.7 | 22.6 | 2,766 ms | 44.3 ms | 4.0 | 0 | 2,811 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 116.3 | 13.4 | 4,745 ms | 74.8 ms | 8.0 | 0 | 3,488 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Requests queued | 30.4 | 33.7 | 3,432 ms | 29.7 ms | 0.9 | 0 | 912 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 37.6 | 22.2 | 5,021 ms | 45.3 ms | 1.8 | 0 | 1,126 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 55.5 | 12.4 | 7,946 ms | 80.8 ms | 4.0 | 0 | 1,660 | 29.9 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Over capacity | — | — | — | — | 0.7 | 0 | 0 | 30.0 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | 0.7 | 0 | 0 | 30.0 s |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.