Speed test matrix
Qwen3.5-4B-AWQ-4bit on RTX 3090 24GB
vllm · 0.32.0 · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- Intel Core i7-9700K (8C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04
vllm serve cyankiwi/Qwen3.5-4B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}Prefill
prompt tok/s by context length8K context11K746 ms TTFT
16K context9,9871,624 ms TTFT
32K context8,9423,612 ms TTFT
64K context7,3458,771 ms TTFT
128K context5,32724,158 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 204.439 ms TTFT | 392.444 ms TTFT | 731.887 ms TTFT | 1,179.699 ms TTFT | 1,726.4199 ms TTFT | 2,170.5237 ms TTFT | Over capacity | Over capacity |
| 16K | 127.0145 ms TTFT | 245.5225 ms TTFT | 467.0419 ms TTFT | 638.5677 ms TTFT | 757.41,127 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 91.0195 ms TTFT | 179.2254 ms TTFT | 343.2446 ms TTFT | 406.6731 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 55.7336 ms TTFT | 109.8397 ms TTFT | 216.5622 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 32.5535 ms TTFT | 65.0723 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 17.4919 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11K | — | 746 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,987.0 | — | 1,624 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,942.0 | — | 3,612 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,345.0 | — | 8,771 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,327.0 | — | 24,158 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 204.4 | 209.1 | 39 ms | 4.8 ms | 1.0 | 0 | 6,132 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 392.4 | 201.5 | 44 ms | 5.0 ms | 2.0 | 0 | 11,768 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 731.8 | 185.3 | 87 ms | 5.4 ms | 4.0 | 0 | 21,951 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 1,179.6 | 150.6 | 99 ms | 6.6 ms | 8.0 | 0 | 35,387 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,726.4 | 113.0 | 199 ms | 8.8 ms | 16.0 | 0 | 51,793 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,170.5 | 70.4 | 237 ms | 14.2 ms | 32.0 | 0 | 65,119 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 127.0 | 129.7 | 145 ms | 7.7 ms | 1.0 | 0 | 3,811 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 245.5 | 124.9 | 225 ms | 8.0 ms | 2.0 | 0 | 7,360 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 467.0 | 117.8 | 419 ms | 8.5 ms | 4.0 | 0 | 14,005 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 638.5 | 80.4 | 677 ms | 12.4 ms | 8.0 | 0 | 19,146 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 757.4 | 47.8 | 1,127 ms | 20.9 ms | 16.0 | 0 | 22,690 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 91.0 | 92.5 | 195 ms | 10.8 ms | 1.0 | 0 | 2,730 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 179.2 | 90.9 | 254 ms | 11.0 ms | 2.0 | 0 | 5,375 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 343.2 | 85.6 | 446 ms | 11.7 ms | 4.0 | 0 | 10,293 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 406.6 | 51.8 | 731 ms | 19.3 ms | 8.0 | 0 | 12,187 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 55.7 | 57.1 | 336 ms | 17.5 ms | 1.0 | 0 | 1,670 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 109.8 | 55.8 | 397 ms | 17.9 ms | 2.0 | 0 | 3,293 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 216.5 | 54.8 | 622 ms | 18.2 ms | 4.0 | 0 | 6,492 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 32.5 | 33.7 | 535 ms | 29.7 ms | 1.0 | 0 | 976 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 65.0 | 33.4 | 723 ms | 30.0 ms | 2.0 | 0 | 1,949 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 17.4 | 18.3 | 919 ms | 54.6 ms | 1.0 | 0 | 521 | 29.9 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.