Speed test matrix
Qwen3-14B-AWQ on RTX 3090 24GB
vllm · 0.32.0 · AWQ3 contexts (0–32K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AWQ
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve Qwen/Qwen3-14B-AWQ --max-model-len 40960 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context1,9994,128 ms TTFT
16K context1,43211,420 ms TTFT
32K context924.035,209 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 71.840 ms TTFT | 139.737 ms TTFT | 262.344 ms TTFT | 472.063 ms TTFT | 788.178 ms TTFT | 1,325.8161 ms TTFT | Over capacity | Over capacity |
| 16K | 50.592 ms TTFT | 78.7144 ms TTFT | 122.7221 ms TTFT | 171.7286 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 39.0161 ms TTFT | 54.6250 ms TTFT | 79.5347 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 1,999.0 | — | 4,128 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 1,432.0 | — | 11,420 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 924.0 | — | 35,209 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 71.8 | 72.3 | 40 ms | 13.8 ms | 1.0 | 0 | 2,154 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 139.7 | 70.5 | 37 ms | 14.2 ms | 2.0 | 0 | 4,190 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 262.3 | 66.5 | 44 ms | 15.0 ms | 4.0 | 0 | 7,868 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 472.0 | 60.4 | 63 ms | 16.6 ms | 8.0 | 0 | 14,152 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 788.1 | 51.0 | 78 ms | 19.6 ms | 16.0 | 0 | 23,632 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,325.8 | 43.5 | 161 ms | 23.0 ms | 32.0 | 0 | 39,744 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 50.5 | 50.6 | 92 ms | 19.8 ms | 1.0 | 0 | 1,516 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 78.7 | 39.5 | 144 ms | 25.3 ms | 2.0 | 0 | 2,358 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 122.7 | 30.8 | 221 ms | 32.5 ms | 4.0 | 0 | 3,680 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 171.7 | 21.5 | 286 ms | 46.5 ms | 8.0 | 0 | 5,144 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 39.0 | 39.0 | 161 ms | 25.6 ms | 1.0 | 0 | 1,169 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 54.6 | 27.3 | 250 ms | 36.6 ms | 2.0 | 0 | 1,636 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 79.5 | 19.9 | 347 ms | 50.2 ms | 4.0 | 0 | 2,384 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.