Speed test matrix
LFM2.5-2.6B on RTX 3090 24GB
vllm · 0.26.0 · BF164 contexts (0–32K) × 5 concurrency levels (1–16) · prefill at 4K, 16K, 32K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 9950X 16-Core Processor
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- BF16
Run
- Served model
- LiquidAI/LFM2.5-2.6B
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve /home/lotto/models/LFM2.5-2.6B --served-model-name LiquidAI/LFM2.5-2.6B --max-model-len 65536 --gpu-memory-utilization 0.9
Prefill
prompt tok/s by context length4K context10.9K383 ms TTFT
16K context11K1,480 ms TTFT
32K context10.1K3,199 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 128.122 ms TTFT | 228.829 ms TTFT | 435.5148 ms TTFT | 837.6891 ms TTFT | 1,553.7890 ms TTFT |
| 4K | 126.3358 ms TTFT | 223.3604 ms TTFT | 420.31,016 ms TTFT | 782.11,696 ms TTFT | 1,376.23,071 ms TTFT |
| 16K | 122.01,467 ms TTFT | 210.52,218 ms TTFT | 379.63,694 ms TTFT | 655.16,699 ms TTFT | 1,028.412,822 ms TTFT |
| 32K | 116.73,191 ms TTFT | 195.84,846 ms TTFT | 336.98,157 ms TTFT | 536.114,880 ms TTFT | 766.3Requests queued28,735 ms TTFT |
Requests queued: the server could not run all requests at once, so some waited in a queue.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 4K | 1 | Valid | 10.9K | — | 383 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11K | — | 1,480 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.1K | — | 3,199 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 128.1 | 128.3 | 22 ms | 7.8 ms | 1.0 | 0 | 3,844 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 228.8 | 114.6 | 29 ms | 8.7 ms | 2.0 | 0 | 6,862 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 435.5 | 109.3 | 148 ms | 9.2 ms | 4.0 | 0 | 13,064 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 837.6 | 105.3 | 891 ms | 9.5 ms | 8.0 | 0 | 25,120 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,553.7 | 98.0 | 890 ms | 10.2 ms | 16.0 | 0 | 46,608 | 30.0 s |
| Sustained decode | 4K | 1 | Valid | 126.3 | 126.4 | 358 ms | 7.9 ms | 1.0 | 0 | 3,789 | 30.0 s |
| Sustained decode | 4K | 2 | Valid | 223.3 | 111.6 | 604 ms | 9.0 ms | 2.0 | 0 | 6,698 | 30.0 s |
| Sustained decode | 4K | 4 | Valid | 420.3 | 104.2 | 1,016 ms | 9.6 ms | 4.0 | 0 | 12,608 | 30.0 s |
| Sustained decode | 4K | 8 | Valid | 782.1 | 95.2 | 1,696 ms | 10.5 ms | 8.0 | 0 | 23,464 | 30.0 s |
| Sustained decode | 4K | 16 | Valid | 1,376.2 | 80.8 | 3,071 ms | 12.4 ms | 16.0 | 0 | 41,280 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 122.0 | 122.1 | 1,467 ms | 8.2 ms | 1.0 | 0 | 3,659 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 210.5 | 103.3 | 2,218 ms | 9.7 ms | 2.0 | 0 | 6,314 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 379.6 | 89.6 | 3,694 ms | 11.2 ms | 4.0 | 0 | 11,388 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 655.1 | 71.7 | 6,699 ms | 14.0 ms | 8.0 | 0 | 19,648 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,028.4 | 49.1 | 12,822 ms | 20.4 ms | 16.0 | 0 | 30,848 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 116.7 | 116.8 | 3,191 ms | 8.6 ms | 1.0 | 0 | 3,502 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 195.8 | 93.9 | 4,846 ms | 10.7 ms | 2.0 | 0 | 5,874 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 336.9 | 74.3 | 8,157 ms | 13.5 ms | 4.0 | 0 | 10,108 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 536.1 | 50.9 | 14,880 ms | 19.7 ms | 8.0 | 0 | 16,080 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 766.3 | 26.8 | 28,735 ms | 37.3 ms | 16.0 | 0 | 22,976 | 30.0 s |
Requests queued: the server could not run all requests at once, so some waited in a queue.