Speed test matrix
Fara1.5-4B-AutoRound-W4A16-RTN on RTX 3090 24GB
vllm · 0.32.0 · AutoRound W4A166 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- Intel Core i7-9700K (8C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AutoRound W4A16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04 vllm serve LeaderboardModel1/Fara1.5-4B-AutoRound-W4A16-RTN --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context9,375874 ms TTFT
16K context7,5602,146 ms TTFT
32K context5,5095,862 ms TTFT
64K context3,56718,062 ms TTFT
128K context2,08861,626 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 171.638 ms TTFT | 331.041 ms TTFT | 614.675 ms TTFT | 1,056.8102 ms TTFT | 1,714.3183 ms TTFT | 2,537.5254 ms TTFT | 3,344.4498 ms TTFT | 3,981.6717 ms TTFT |
| 16K | 141.4376 ms TTFT | 266.0566 ms TTFT | 427.2904 ms TTFT | 672.81,657 ms TTFT | 949.13,050 ms TTFT | 1,171.55,974 ms TTFT | 1,299.511,984 ms TTFT | Over capacity |
| 32K | 119.6571 ms TTFT | 221.6754 ms TTFT | 318.31,221 ms TTFT | 477.92,140 ms TTFT | 638.83,973 ms TTFT | 752.07,724 ms TTFT | Over capacity | Over capacity |
| 64K | 91.2610 ms TTFT | 165.2859 ms TTFT | 214.31,344 ms TTFT | 303.92,321 ms TTFT | 386.84,256 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 61.71,819 ms TTFT | 109.12,603 ms TTFT | 129.04,159 ms TTFT | 174.77,320 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 35.4Requests queued2,908 ms TTFT | 66.54,107 ms TTFT | 74.36,458 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,375.0 | — | 874 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,560.0 | — | 2,146 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,509.0 | — | 5,862 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,567.0 | — | 18,062 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,088.0 | — | 61,626 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 171.6 | 172.6 | 38 ms | 5.8 ms | 1.0 | 0 | 5,148 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 331.0 | 167.0 | 41 ms | 6.0 ms | 2.0 | 0 | 9,928 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 614.6 | 155.5 | 75 ms | 6.4 ms | 4.0 | 0 | 18,436 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 1,056.8 | 134.0 | 102 ms | 7.5 ms | 8.0 | 0 | 31,705 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,714.3 | 109.0 | 183 ms | 9.2 ms | 16.0 | 0 | 51,424 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,537.5 | 80.7 | 254 ms | 12.4 ms | 32.0 | 0 | 76,128 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 3,344.4 | 53.0 | 498 ms | 18.9 ms | 64.0 | 0 | 100,288 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,981.6 | 31.2 | 717 ms | 32.1 ms | 128.0 | 0 | 119,424 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 141.4 | 141.8 | 376 ms | 7.0 ms | 1.0 | 0 | 4,243 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 266.0 | 132.8 | 566 ms | 7.5 ms | 2.0 | 0 | 7,978 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 427.2 | 105.6 | 904 ms | 9.5 ms | 4.0 | 0 | 12,816 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 672.8 | 81.7 | 1,657 ms | 12.2 ms | 8.0 | 0 | 20,184 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 949.1 | 55.6 | 3,050 ms | 18.0 ms | 16.0 | 0 | 28,464 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,171.5 | 32.1 | 5,974 ms | 31.1 ms | 32.0 | 0 | 35,136 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,299.5 | 16.1 | 11,984 ms | 62.0 ms | 64.0 | 0 | 38,976 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 119.6 | 119.9 | 571 ms | 8.3 ms | 1.0 | 0 | 3,589 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 221.6 | 110.4 | 754 ms | 9.1 ms | 2.0 | 0 | 6,648 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 318.3 | 78.3 | 1,221 ms | 12.8 ms | 4.0 | 0 | 9,548 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 477.9 | 57.4 | 2,140 ms | 17.4 ms | 8.0 | 0 | 14,336 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 638.8 | 36.6 | 3,973 ms | 27.3 ms | 16.0 | 0 | 19,152 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 752.0 | 19.8 | 7,724 ms | 50.4 ms | 32.0 | 0 | 22,560 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 91.2 | 91.4 | 610 ms | 10.9 ms | 1.0 | 0 | 2,735 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 165.2 | 82.2 | 859 ms | 12.2 ms | 2.0 | 0 | 4,954 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 214.3 | 52.7 | 1,344 ms | 19.0 ms | 4.0 | 0 | 6,428 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 303.9 | 36.5 | 2,321 ms | 27.4 ms | 8.0 | 0 | 9,112 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 386.8 | 22.1 | 4,256 ms | 45.2 ms | 16.0 | 0 | 11,600 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 61.7 | 61.8 | 1,819 ms | 16.2 ms | 1.0 | 0 | 1,849 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 109.1 | 53.5 | 2,603 ms | 18.7 ms | 2.0 | 0 | 3,272 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 129.0 | 30.3 | 4,159 ms | 33.0 ms | 4.0 | 0 | 3,868 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 174.7 | 19.0 | 7,320 ms | 52.6 ms | 8.0 | 0 | 5,240 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 35.4 | 38.5 | 2,908 ms | 26.0 ms | 0.9 | 0 | 1,061 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 66.5 | 32.3 | 4,107 ms | 31.0 ms | 2.0 | 0 | 1,996 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 74.3 | 16.9 | 6,458 ms | 59.2 ms | 4.0 | 0 | 2,228 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.