Speed test matrix
Qwen3.5-4B-AWQ-4bit on RTX 3060 12GB
vllm · 0.32.0 · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3060
- VRAM
- 12 GB
- CPU
- Intel Xeon E5-2680 v4 (14C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: fp8 # 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04 vllm serve cyankiwi/Qwen3.5-4B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype fp8
Prefill
prompt tok/s by context length8K context2,7073,010 ms TTFT
16K context2,4906,475 ms TTFT
32K context2,18014,721 ms TTFT
64K context1,72437,135 ms TTFT
128K context1,218104,998 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 84.5108 ms TTFT | 163.0121 ms TTFT | 304.0213 ms TTFT | 529.5337 ms TTFT | 858.9555 ms TTFT | 1,152.0937 ms TTFT | Over capacity | Over capacity |
| 16K | 79.3261 ms TTFT | 147.4363 ms TTFT | 247.6532 ms TTFT | 335.6878 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 74.6454 ms TTFT | 134.0613 ms TTFT | 207.3899 ms TTFT | 244.01,495 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 66.8954 ms TTFT | 113.21,263 ms TTFT | 156.41,930 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 55.4968 ms TTFT | 85.41,294 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 39.3Requests queued2,592 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,707.0 | — | 3,010 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 2,490.0 | — | 6,475 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 2,180.0 | — | 14,721 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,724.0 | — | 37,135 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,218.0 | — | 104,998 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 84.5 | 84.6 | 108 ms | 11.8 ms | 1.0 | 0 | 2,534 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 163.0 | 81.7 | 121 ms | 12.2 ms | 2.0 | 0 | 4,889 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 304.0 | 76.3 | 213 ms | 13.1 ms | 4.0 | 0 | 9,116 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 529.5 | 66.8 | 337 ms | 15.0 ms | 8.0 | 0 | 15,887 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 858.9 | 54.2 | 555 ms | 18.5 ms | 16.0 | 0 | 25,776 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,152.0 | 36.1 | 937 ms | 27.7 ms | 32.0 | 0 | 34,560 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 79.3 | 79.4 | 261 ms | 12.6 ms | 1.0 | 0 | 2,378 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 147.4 | 73.7 | 363 ms | 13.6 ms | 2.0 | 0 | 4,423 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 247.6 | 61.7 | 532 ms | 16.2 ms | 4.0 | 0 | 7,428 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 335.6 | 41.5 | 878 ms | 24.1 ms | 8.0 | 0 | 10,064 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 74.6 | 74.8 | 454 ms | 13.4 ms | 1.0 | 0 | 2,238 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 134.0 | 67.0 | 613 ms | 14.9 ms | 2.0 | 0 | 4,020 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 207.3 | 51.4 | 899 ms | 19.4 ms | 4.0 | 0 | 6,216 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 244.0 | 29.9 | 1,495 ms | 33.5 ms | 8.0 | 0 | 7,320 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 66.8 | 67.0 | 954 ms | 14.9 ms | 1.0 | 0 | 2,002 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 113.2 | 56.3 | 1,263 ms | 17.8 ms | 2.0 | 0 | 3,394 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 156.4 | 38.3 | 1,930 ms | 26.1 ms | 4.0 | 0 | 4,688 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 55.4 | 55.6 | 968 ms | 18.0 ms | 1.0 | 0 | 1,661 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 85.4 | 42.7 | 1,294 ms | 23.4 ms | 2.0 | 0 | 2,560 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 39.3 | 42.3 | 2,592 ms | 23.7 ms | 0.9 | 0 | 1,180 | 30.0 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.