Speed test matrix
Qwen3.5-4B-AWQ-4bit on RTX 3090 24GB
vllm · 0.32.0 · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- Intel Core i7-9700K (8C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04 vllm serve cyankiwi/Qwen3.5-4B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context9,313879 ms TTFT
16K context7,3962,193 ms TTFT
32K context5,4725,902 ms TTFT
64K context3,55118,140 ms TTFT
128K context2,08461,767 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 172.939 ms TTFT | 319.940 ms TTFT | 599.172 ms TTFT | 1,036.1101 ms TTFT | 1,732.3164 ms TTFT | 2,588.0273 ms TTFT | 3,448.4498 ms TTFT | 4,063.4808 ms TTFT |
| 16K | 142.2378 ms TTFT | 259.1553 ms TTFT | 417.1941 ms TTFT | 661.31,650 ms TTFT | 953.53,069 ms TTFT | 1,181.26,013 ms TTFT | 1,317.411,980 ms TTFT | Over capacity |
| 32K | 120.1557 ms TTFT | 216.3755 ms TTFT | 312.71,210 ms TTFT | 471.62,134 ms TTFT | 641.03,970 ms TTFT | 756.97,720 ms TTFT | Over capacity | Over capacity |
| 64K | 91.6671 ms TTFT | 162.1858 ms TTFT | 211.71,339 ms TTFT | 301.42,330 ms TTFT | 388.34,265 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 61.91,843 ms TTFT | 107.92,599 ms TTFT | 128.54,155 ms TTFT | 173.97,309 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 35.4Requests queued2,917 ms TTFT | 66.14,090 ms TTFT | 74.26,455 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,313.0 | — | 879 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,396.0 | — | 2,193 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,472.0 | — | 5,902 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,551.0 | — | 18,140 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,084.0 | — | 61,767 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 172.9 | 174.1 | 39 ms | 5.7 ms | 1.0 | 0 | 5,187 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 319.9 | 161.2 | 40 ms | 6.2 ms | 2.0 | 0 | 9,596 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 599.1 | 151.1 | 72 ms | 6.6 ms | 4.0 | 0 | 17,971 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 1,036.1 | 131.2 | 101 ms | 7.6 ms | 8.0 | 0 | 31,080 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,732.3 | 110.3 | 164 ms | 9.1 ms | 16.0 | 0 | 51,968 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,588.0 | 82.5 | 273 ms | 12.1 ms | 32.0 | 0 | 77,632 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 3,448.4 | 54.7 | 498 ms | 18.3 ms | 64.0 | 0 | 103,424 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 4,063.4 | 31.9 | 808 ms | 31.4 ms | 128.0 | 0 | 121,856 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 142.2 | 142.6 | 378 ms | 7.0 ms | 1.0 | 0 | 4,266 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 259.1 | 129.4 | 553 ms | 7.7 ms | 2.0 | 0 | 7,772 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 417.1 | 103.1 | 941 ms | 9.7 ms | 4.0 | 0 | 12,512 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 661.3 | 80.3 | 1,650 ms | 12.5 ms | 8.0 | 0 | 19,840 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 953.5 | 55.9 | 3,069 ms | 17.9 ms | 16.0 | 0 | 28,608 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,181.2 | 32.4 | 6,013 ms | 30.9 ms | 32.0 | 0 | 35,424 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,317.4 | 16.3 | 11,980 ms | 61.2 ms | 64.0 | 0 | 39,488 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 120.1 | 120.5 | 557 ms | 8.3 ms | 1.0 | 0 | 3,604 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 216.3 | 107.7 | 755 ms | 9.3 ms | 2.0 | 0 | 6,488 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 312.7 | 77.0 | 1,210 ms | 13.0 ms | 4.0 | 0 | 9,380 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 471.6 | 56.6 | 2,134 ms | 17.7 ms | 8.0 | 0 | 14,144 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 641.0 | 36.8 | 3,970 ms | 27.2 ms | 16.0 | 0 | 19,216 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 756.9 | 20.0 | 7,720 ms | 50.1 ms | 32.0 | 0 | 22,688 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 91.6 | 91.7 | 671 ms | 10.9 ms | 1.0 | 0 | 2,748 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 162.1 | 80.6 | 858 ms | 12.4 ms | 2.0 | 0 | 4,863 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 211.7 | 52.1 | 1,339 ms | 19.2 ms | 4.0 | 0 | 6,348 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 301.4 | 36.2 | 2,330 ms | 27.6 ms | 8.0 | 0 | 9,041 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 388.3 | 22.2 | 4,265 ms | 45.0 ms | 16.0 | 0 | 11,648 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 61.9 | 62.0 | 1,843 ms | 16.1 ms | 1.0 | 0 | 1,855 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 107.9 | 52.9 | 2,599 ms | 18.9 ms | 2.0 | 0 | 3,236 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 128.5 | 30.2 | 4,155 ms | 33.2 ms | 4.0 | 0 | 3,856 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 173.9 | 18.9 | 7,309 ms | 52.8 ms | 8.0 | 0 | 5,216 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 35.4 | 38.5 | 2,917 ms | 26.0 ms | 0.9 | 0 | 1,062 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 66.1 | 32.0 | 4,090 ms | 31.2 ms | 2.0 | 0 | 1,982 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 74.2 | 16.9 | 6,455 ms | 59.2 ms | 4.0 | 0 | 2,224 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.