Speed test matrix
LFM3-1.7B-TT-Mixed-BFP4-BFP8-P150 on Tenstorrent P150 32GB
vllm · 0.1.dev14187+g03fa3af2e · BFP46 contexts (0–120K) × 5 concurrency levels (1–16) · prefill at 2K, 4K, 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- AI accelerator
- GPU
- Tenstorrent P150
- VRAM
- 32 GB
- CPU
- AMD Ryzen 9 9950X
- RAM
- 96 GB
- OS
- Ubuntu 26.04 LTS
Engine
- Engine
- vllm
- Version
- 0.1.dev14187+g03fa3af2e
- Quantization
- BFP4
- Backend
- tt-metal
Run
- Served model
- LiquidAI/LFM3-1.7B
- Revision
- dc39d8f9d706e626edb397306c65bdcba3cf7ebd
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
LFM3_MAX_NUM_SEQS=16 LFM3_MAX_MODEL_LEN=131072 tools/serve_lfm3.sh # Lottolabs/LFM3-1.7B-TT-Mixed-BFP4-BFP8-P150 checkpoint (rev dc39d8f9), runtime image lfm3-p150:dev-s13 (sha256:9b00b6a2698f; local build after release s12: shared speculation rounds + trace page-table fix), vLLM + TT plugin, KV pool 1572864 tokens, GPTQ BFP4 MLP w1/w3 / BFP8 rest (TT, format tt-lfmq lfmq-1), sampling preset liquid (T 0.1, top_k 50, rep 1.1), prompt-lookup speculation on (shared verify rounds of up to 16 rows, any number of users), decode-ahead on, prefix caching on
Prefill
prompt tok/s by context length2K context29.3K74 ms TTFT
4K context36.9K113 ms TTFT
8K context43.5K189 ms TTFT
16K context47.8K340 ms TTFT
32K context44.3K731 ms TTFT
64K context35.4K1,823 ms TTFT
120K context24.9K4,864 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 301.729 ms TTFT | 567.639 ms TTFT | 1,004.634 ms TTFT | 1,700.5129 ms TTFT | 2,883.1295 ms TTFT |
| 4K | 299.936 ms TTFT | 564.253 ms TTFT | 945.984 ms TTFT | 1,564.6231 ms TTFT | 2,512.5412 ms TTFT |
| 16K | 301.972 ms TTFT | 524.7108 ms TTFT | 791.6254 ms TTFT | 1,249.8501 ms TTFT | 1,795.6948 ms TTFT |
| 32K | 277.586 ms TTFT | 467.6156 ms TTFT | 672.1312 ms TTFT | 978.1589 ms TTFT | 1,281.21,175 ms TTFT |
| 64K | 259.3117 ms TTFT | 390.0219 ms TTFT | 513.5423 ms TTFT | 681.7807 ms TTFT | 815.91,599 ms TTFT |
| 120K | 178.984 ms TTFT | 269.9148 ms TTFT | 366.4289 ms TTFT | 447.6535 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 2K | 1 | Valid | 29.3K | — | 74 ms | — | — | 1 | — | — |
| Prefill | 4K | 1 | Valid | 36.9K | — | 113 ms | — | — | 1 | — | — |
| Prefill | 8K | 1 | Valid | 43.5K | — | 189 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 47.8K | — | 340 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.3K | — | 731 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 35.4K | — | 1,823 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 24.9K | — | 4,864 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 301.7 | 296.7 | 29 ms | 3.4 ms | 1.0 | 1 | 9,049 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 567.6 | 274.2 | 39 ms | 3.6 ms | 2.0 | 2 | 17,029 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 1,004.6 | 254.2 | 34 ms | 3.9 ms | 4.0 | 4 | 30,136 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 1,700.5 | 216.6 | 129 ms | 4.6 ms | 8.0 | 0 | 51,009 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 2,883.1 | 184.7 | 295 ms | 5.4 ms | 16.0 | 0 | 86,480 | 30.0 s |
| Sustained decode | 4K | 1 | Valid | 299.9 | 296.6 | 36 ms | 3.4 ms | 1.0 | 1 | 8,996 | 30.0 s |
| Sustained decode | 4K | 2 | Valid | 564.2 | 265.9 | 53 ms | 3.8 ms | 2.0 | 2 | 16,926 | 30.0 s |
| Sustained decode | 4K | 4 | Valid | 945.9 | 236.0 | 84 ms | 4.2 ms | 4.0 | 2 | 28,375 | 30.0 s |
| Sustained decode | 4K | 8 | Valid | 1,564.6 | 198.3 | 231 ms | 5.0 ms | 8.0 | 0 | 46,933 | 30.0 s |
| Sustained decode | 4K | 16 | Valid | 2,512.5 | 159.3 | 412 ms | 6.3 ms | 16.0 | 0 | 75,377 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 301.9 | 286.1 | 72 ms | 3.5 ms | 1.0 | 1 | 9,055 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 524.7 | 254.1 | 108 ms | 3.9 ms | 2.0 | 2 | 15,738 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 791.6 | 199.2 | 254 ms | 5.0 ms | 4.0 | 0 | 23,749 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,249.8 | 157.4 | 501 ms | 6.4 ms | 8.0 | 0 | 37,492 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,795.6 | 112.9 | 948 ms | 8.9 ms | 16.0 | 0 | 53,856 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 277.5 | 263.7 | 86 ms | 3.8 ms | 1.0 | 1 | 8,323 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 467.6 | 208.3 | 156 ms | 4.8 ms | 2.0 | 1 | 14,026 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 672.1 | 169.1 | 312 ms | 5.9 ms | 4.0 | 0 | 20,162 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 978.1 | 122.8 | 589 ms | 8.1 ms | 8.0 | 0 | 29,342 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,281.2 | 80.3 | 1,175 ms | 12.5 ms | 16.0 | 0 | 38,432 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 259.3 | 244.0 | 117 ms | 4.1 ms | 1.0 | 1 | 7,780 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 390.0 | 193.4 | 219 ms | 5.3 ms | 2.0 | 0 | 11,699 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 513.5 | 129.0 | 423 ms | 7.8 ms | 4.0 | 0 | 15,404 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 681.7 | 85.5 | 807 ms | 11.7 ms | 8.0 | 0 | 20,449 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 815.9 | 51.1 | 1,599 ms | 19.6 ms | 16.0 | 0 | 24,464 | 30.0 s |
| Sustained decode | 120K | 1 | Valid | 178.9 | 180.3 | 84 ms | 5.5 ms | 1.0 | 0 | 5,365 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 269.9 | 135.7 | 148 ms | 7.4 ms | 2.0 | 0 | 8,096 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 366.4 | 91.9 | 289 ms | 10.9 ms | 4.0 | 0 | 10,990 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 447.6 | 56.1 | 535 ms | 17.8 ms | 8.0 | 0 | 13,423 | 30.0 s |
| Sustained decode | 120K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.