Speed test matrix
Fara1.5-4B-AutoRound-W4A16-RTN on RTX 3090 24GB
vllm · 0.32.0 · AutoRound W4A166 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AutoRound W4A16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve LeaderboardModel1/Fara1.5-4B-AutoRound-W4A16-RTN --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context8,459968 ms TTFT
16K context6,9142,346 ms TTFT
32K context4,9906,471 ms TTFT
64K context3,18620,220 ms TTFT
128K context1,86768,936 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 167.425 ms TTFT | 324.543 ms TTFT | 605.754 ms TTFT | 1,058.0101 ms TTFT | 1,694.8150 ms TTFT | 2,446.5248 ms TTFT | 3,180.9491 ms TTFT | 3,794.2760 ms TTFT |
| 16K | 136.9387 ms TTFT | 256.7577 ms TTFT | 409.7956 ms TTFT | 640.21,713 ms TTFT | 890.43,238 ms TTFT | 1,084.36,343 ms TTFT | 1,226.912,859 ms TTFT | Over capacity |
| 32K | 115.7525 ms TTFT | 213.7782 ms TTFT | 304.81,280 ms TTFT | 450.32,274 ms TTFT | 591.14,290 ms TTFT | 685.68,405 ms TTFT | Over capacity | Over capacity |
| 64K | 88.3594 ms TTFT | 158.2849 ms TTFT | 204.41,368 ms TTFT | 282.62,408 ms TTFT | 353.24,524 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 59.81,846 ms TTFT | 103.32,702 ms TTFT | 122.64,414 ms TTFT | 161.67,894 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 34.3Requests queued2,950 ms TTFT | 62.64,270 ms TTFT | 69.96,824 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,459.0 | — | 968 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,914.0 | — | 2,346 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,990.0 | — | 6,471 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,186.0 | — | 20,220 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,867.0 | — | 68,936 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 167.4 | 168.7 | 25 ms | 5.9 ms | 1.0 | 0 | 5,023 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 324.5 | 163.7 | 43 ms | 6.1 ms | 2.0 | 0 | 9,735 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 605.7 | 153.4 | 54 ms | 6.5 ms | 4.0 | 0 | 18,171 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 1,058.0 | 134.6 | 101 ms | 7.4 ms | 8.0 | 0 | 31,736 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,694.8 | 108.3 | 150 ms | 9.2 ms | 16.0 | 0 | 50,832 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,446.5 | 78.2 | 248 ms | 12.8 ms | 32.0 | 0 | 73,376 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 3,180.9 | 50.4 | 491 ms | 19.8 ms | 64.0 | 0 | 95,424 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,794.2 | 29.7 | 760 ms | 33.6 ms | 128.0 | 0 | 113,792 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 136.9 | 137.4 | 387 ms | 7.3 ms | 1.0 | 0 | 4,107 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 256.7 | 128.2 | 577 ms | 7.8 ms | 2.0 | 0 | 7,702 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 409.7 | 101.3 | 956 ms | 9.9 ms | 4.0 | 0 | 12,288 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 640.2 | 77.5 | 1,713 ms | 12.9 ms | 8.0 | 0 | 19,200 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 890.4 | 51.7 | 3,238 ms | 19.3 ms | 16.0 | 0 | 26,704 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,084.3 | 29.4 | 6,343 ms | 34.1 ms | 32.0 | 0 | 32,512 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,226.9 | 14.9 | 12,859 ms | 67.0 ms | 64.0 | 0 | 36,800 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 115.7 | 116.1 | 525 ms | 8.6 ms | 1.0 | 0 | 3,472 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 213.7 | 106.4 | 782 ms | 9.4 ms | 2.0 | 0 | 6,410 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 304.8 | 74.9 | 1,280 ms | 13.4 ms | 4.0 | 0 | 9,145 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 450.3 | 53.8 | 2,274 ms | 18.6 ms | 8.0 | 0 | 13,504 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 591.1 | 33.6 | 4,290 ms | 29.8 ms | 16.0 | 0 | 17,728 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 685.6 | 17.8 | 8,405 ms | 56.2 ms | 32.0 | 0 | 20,544 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 88.3 | 88.5 | 594 ms | 11.3 ms | 1.0 | 0 | 2,648 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 158.2 | 78.7 | 849 ms | 12.7 ms | 2.0 | 0 | 4,746 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 204.4 | 50.2 | 1,368 ms | 19.9 ms | 4.0 | 0 | 6,128 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 282.6 | 33.7 | 2,408 ms | 29.6 ms | 8.0 | 0 | 8,472 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 353.2 | 20.0 | 4,524 ms | 49.9 ms | 16.0 | 0 | 10,592 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 59.8 | 59.9 | 1,846 ms | 16.7 ms | 1.0 | 0 | 1,794 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 103.3 | 50.5 | 2,702 ms | 19.8 ms | 2.0 | 0 | 3,098 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 122.6 | 28.6 | 4,414 ms | 35.0 ms | 4.0 | 0 | 3,676 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 161.6 | 17.3 | 7,894 ms | 58.0 ms | 8.0 | 0 | 4,848 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 34.3 | 37.3 | 2,950 ms | 26.8 ms | 0.9 | 0 | 1,029 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 62.6 | 30.2 | 4,270 ms | 33.1 ms | 2.0 | 0 | 1,876 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 69.9 | 15.7 | 6,824 ms | 63.6 ms | 4.0 | 0 | 2,096 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.