Speed test matrix
gemma-4-12B-it-qat-w4a16-ct on RTX 3090 24GB
vllm · 0.32.0 · W4A166 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- W4A16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve google/gemma-4-12B-it-qat-w4a16-ct --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context2,6902,979 ms TTFT
16K context2,3756,684 ms TTFT
32K context1,94716,232 ms TTFT
64K context1,42044,392 ms TTFT
128K context940.0133,922 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 66.642 ms TTFT | 126.446 ms TTFT | 228.248 ms TTFT | 388.467 ms TTFT | 597.687 ms TTFT | 914.2157 ms TTFT | 1,326.8292 ms TTFT | Over capacity |
| 16K | 48.583 ms TTFT | Looped | Looped | Looped | 453.2327 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 42.1140 ms TTFT | 83.5181 ms TTFT | 156.4297 ms TTFT | 295.5443 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 36.6257 ms TTFT | 72.5325 ms TTFT | 133.1466 ms TTFT | 251.5725 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 28.9527 ms TTFT | 57.7689 ms TTFT | 102.2966 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 20.4Requests queued1,112 ms TTFT | 40.0Requests queued1,509 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,690.0 | — | 2,979 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 2,375.0 | — | 6,684 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,947.0 | — | 16,232 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,420.0 | — | 44,392 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 940.0 | — | 133,922 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 66.6 | 67.2 | 42 ms | 14.9 ms | 1.0 | 0 | 1,997 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 126.4 | 64.1 | 46 ms | 15.6 ms | 2.0 | 0 | 3,790 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 228.2 | 58.4 | 48 ms | 17.1 ms | 4.0 | 0 | 6,844 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 388.4 | 50.3 | 67 ms | 19.9 ms | 8.0 | 0 | 11,647 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 597.6 | 39.6 | 87 ms | 25.3 ms | 16.0 | 0 | 17,920 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 914.2 | 31.1 | 157 ms | 32.2 ms | 32.0 | 0 | 27,392 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,326.8 | 22.3 | 292 ms | 44.9 ms | 64.0 | 0 | 39,744 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 48.5 | 48.8 | 83 ms | 20.5 ms | 1.0 | 0 | 1,453 | 30.0 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Valid | 453.2 | 28.4 | 327 ms | 35.2 ms | 16.0 | 0 | 13,584 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 42.1 | 42.2 | 140 ms | 23.7 ms | 1.0 | 0 | 1,263 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 83.5 | 41.8 | 181 ms | 23.9 ms | 2.0 | 0 | 2,504 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 156.4 | 39.1 | 297 ms | 25.6 ms | 4.0 | 0 | 4,692 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 295.5 | 36.9 | 443 ms | 27.1 ms | 8.0 | 0 | 8,864 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 36.6 | 36.6 | 257 ms | 27.3 ms | 1.0 | 0 | 1,097 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 72.5 | 36.2 | 325 ms | 27.6 ms | 2.0 | 0 | 2,174 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 133.1 | 33.2 | 466 ms | 30.1 ms | 4.0 | 0 | 3,992 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 251.5 | 31.3 | 725 ms | 32.0 ms | 8.0 | 0 | 7,544 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 28.9 | 29.0 | 527 ms | 34.5 ms | 1.0 | 0 | 868 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 57.7 | 28.8 | 689 ms | 34.7 ms | 2.0 | 0 | 1,730 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 102.2 | 25.4 | 966 ms | 39.3 ms | 4.0 | 0 | 3,064 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 20.4 | 20.8 | 1,112 ms | 48.0 ms | 1.0 | 0 | 612 | 30.0 s |
| Sustained decode | 248K | 2 | Requests queued | 40.0 | 20.7 | 1,509 ms | 48.3 ms | 1.9 | 0 | 1,200 | 30.0 s |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.