Speed test matrix
LFM3-1.7B-TT-Mixed-BFP4-BFP8-P150 on Tenstorrent P150 32GB
vllm · 0.1.dev14187+g03fa3af2e · BFP46 contexts (0–120K) × 5 concurrency levels (1–16) · prefill at 2K, 4K, 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- AI accelerator
- GPU
- Tenstorrent P150
- VRAM
- 32 GB
- CPU
- AMD Ryzen 9 9950X
- RAM
- 96 GB
- OS
- Ubuntu 26.04 LTS
Engine
- Engine
- vllm
- Version
- 0.1.dev14187+g03fa3af2e
- Quantization
- BFP4
- Backend
- tt-metal
Run
- Served model
- LiquidAI/LFM3-1.7B
- Revision
- dc39d8f9d706e626edb397306c65bdcba3cf7ebd
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
python3 launch.py --revision dc39d8f9d706e626edb397306c65bdcba3cf7ebd --max-num-seqs 16 --max-model-len 131072 --device-ownership-confirmed # Lottolabs/LFM3-1.7B-TT-Mixed-BFP4-BFP8-P150 release s12-20261030, image lottolabs/lfm3-1.7b-tt-p150:s12-20261030 (sha256:43cc3af5b72f), vLLM + TT plugin, KV pool 1572864 tokens, GPTQ BFP4 MLP w1/w3 / BFP8 rest, sampling preset liquid (T 0.1, top_k 50, rep 1.1), prompt-lookup speculation off (>1 slot), decode-ahead on, prefix caching on; base model LiquidAI/LFM3-1.7B rev 40ebe0aa
Prefill
prompt tok/s by context length2K context29.8K73 ms TTFT
4K context38.3K109 ms TTFT
8K context43.8K187 ms TTFT
16K context47.8K340 ms TTFT
32K context44.3K730 ms TTFT
64K context35.6K1,814 ms TTFT
120K context24.9K4,859 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 267.129 ms TTFT | 520.148 ms TTFT | 976.182 ms TTFT | 1,740.0145 ms TTFT | 2,892.8279 ms TTFT |
| 4K | 261.036 ms TTFT | 504.363 ms TTFT | 922.5112 ms TTFT | 1,602.5208 ms TTFT | 2,527.9404 ms TTFT |
| 16K | 248.471 ms TTFT | 465.2106 ms TTFT | 806.0256 ms TTFT | 1,276.0474 ms TTFT | 1,802.2931 ms TTFT |
| 32K | 236.585 ms TTFT | 418.2128 ms TTFT | 680.1307 ms TTFT | 993.2588 ms TTFT | 1,285.81,160 ms TTFT |
| 64K | 211.0117 ms TTFT | 350.5235 ms TTFT | 520.1411 ms TTFT | 689.9804 ms TTFT | 817.81,537 ms TTFT |
| 120K | 179.285 ms TTFT | 273.4125 ms TTFT | 369.0281 ms TTFT | 449.9490 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 2K | 1 | Valid | 29.8K | — | 73 ms | — | — | 1 | — | — |
| Prefill | 4K | 1 | Valid | 38.3K | — | 109 ms | — | — | 1 | — | — |
| Prefill | 8K | 1 | Valid | 43.8K | — | 187 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 47.8K | — | 340 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.3K | — | 730 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 35.6K | — | 1,814 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 24.9K | — | 4,859 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 267.1 | 269.3 | 29 ms | 3.7 ms | 1.0 | 1 | 8,012 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 520.1 | 264.5 | 48 ms | 3.8 ms | 2.0 | 2 | 15,602 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 976.1 | 250.5 | 82 ms | 4.0 ms | 4.0 | 4 | 29,280 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 1,740.0 | 221.1 | 145 ms | 4.5 ms | 8.0 | 0 | 52,200 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 2,892.8 | 185.5 | 279 ms | 5.4 ms | 16.0 | 0 | 86,784 | 30.0 s |
| Sustained decode | 4K | 1 | Valid | 261.0 | 263.6 | 36 ms | 3.8 ms | 1.0 | 1 | 7,830 | 30.0 s |
| Sustained decode | 4K | 2 | Valid | 504.3 | 257.1 | 63 ms | 3.9 ms | 2.0 | 2 | 15,129 | 30.0 s |
| Sustained decode | 4K | 4 | Valid | 922.5 | 232.4 | 112 ms | 4.3 ms | 4.0 | 0 | 27,676 | 30.0 s |
| Sustained decode | 4K | 8 | Valid | 1,602.5 | 202.5 | 208 ms | 4.9 ms | 8.0 | 0 | 48,072 | 30.0 s |
| Sustained decode | 4K | 16 | Valid | 2,527.9 | 160.3 | 404 ms | 6.2 ms | 16.0 | 0 | 75,824 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 248.4 | 253.5 | 71 ms | 3.9 ms | 1.0 | 1 | 7,453 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 465.2 | 233.2 | 106 ms | 4.3 ms | 2.0 | 0 | 13,956 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 806.0 | 202.7 | 256 ms | 4.9 ms | 4.0 | 0 | 24,181 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,276.0 | 160.6 | 474 ms | 6.2 ms | 8.0 | 0 | 38,281 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,802.2 | 113.4 | 931 ms | 8.8 ms | 16.0 | 0 | 54,064 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 236.5 | 236.8 | 85 ms | 4.2 ms | 1.0 | 0 | 7,095 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 418.2 | 209.4 | 128 ms | 4.8 ms | 2.0 | 0 | 12,546 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 680.1 | 170.8 | 307 ms | 5.9 ms | 4.0 | 0 | 20,400 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 993.2 | 124.7 | 588 ms | 8.0 ms | 8.0 | 0 | 29,792 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,285.8 | 80.7 | 1,160 ms | 12.4 ms | 16.0 | 0 | 38,560 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 211.0 | 211.4 | 117 ms | 4.7 ms | 1.0 | 0 | 6,330 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 350.5 | 175.8 | 235 ms | 5.7 ms | 2.0 | 0 | 10,514 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 520.1 | 130.5 | 411 ms | 7.7 ms | 4.0 | 0 | 15,604 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 689.9 | 86.6 | 804 ms | 11.5 ms | 8.0 | 0 | 20,696 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 817.8 | 51.3 | 1,537 ms | 19.5 ms | 16.0 | 0 | 24,528 | 30.0 s |
| Sustained decode | 120K | 1 | Valid | 179.2 | 179.5 | 85 ms | 5.6 ms | 1.0 | 0 | 5,376 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 273.4 | 136.9 | 125 ms | 7.3 ms | 2.0 | 0 | 8,202 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 369.0 | 92.6 | 281 ms | 10.8 ms | 4.0 | 0 | 11,068 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 449.9 | 56.6 | 490 ms | 17.7 ms | 8.0 | 0 | 13,496 | 30.0 s |
| Sustained decode | 120K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.