Pierwsze krokiRankingDecode calculatorModeleOrganizationsReportsMatrix runsSprzętBenchmarkiMarketplaceWynajemProDokumentacja API
Język
Actual Computer — Every computer, one endpoint
Community field report

Train your own drafter: LFM3-1.7B on a P150 got 1.2–1.5× faster, output unchanged

Why local users should train a draft model for speculative decoding, and how we did it for LiquidAI/LFM3-1.7B: a 157M-parameter DFlash-style drafter trained on one RTX 3090 from the model's own answers, served on a Tenstorrent P150. Single-user prose decode 1.20–1.50× faster, every reply token-for-token identical.

TL;DR

We trained a small draft model for LiquidAI/LFM3-1.7B on one RTX 3090, using only LFM3's own answers as training data, and plugged it into speculative decoding on a single Tenstorrent P150. Single-user speed went up 1.20–1.50× on ordinary prose (chat, long-prompt chat, math) and every reply stayed token-for-token identical to the server without it. Copy-heavy work (code edits, tool calls) kept its ~1,500 tok/s because a prompt-lookup drafter still handles those.

On 242 prompts from sources never used in training (open questions, AlpacaEval, HumanEval/MBPP, long documents, translation, JSON, creative writing, role-play, agentic tool use) the gain held at 1.21× greedy / 1.18× sampled over prompt lookup (1.71× / 1.49× over no speculation). It is strongest on code and agentic work (up to 1.41×) and fades to about break-even on translation — details below.

Workload (1 user, default sampling T 0.1 / top-k 50 / rep 1.1)BeforeWith trained drafterGain
Chat285 tok/s364 tok/s1.28×
Chat, 2K-token prompt339 tok/s407 tok/s1.20×
GSM8K-style math332 tok/s500 tok/s1.50×
Code with thinking805 tok/s831 tok/s1.03×
Code edit (copies the prompt)1,497 tok/s1,481 tok/s0.99×
Tool-call file writes1,520 tok/s1,525 tok/s1.00×

Greedy decoding gains more: chat-2K 422 → 545 tok/s, GSM8K 401 → 644 tok/s. Medians of 3 runs, same day, same server build otherwise, all output texts identical.

The whole thing — data generation, training, evaluation — cost about 12 hours of hardware time on equipment we already had.

Everything is downloadable

WhatWhere
The drafter (BF16 weights, config, 65K draft vocab, acceptance calibration, all training and evaluation code, training log, eval tables)Lottolabs/LFM3-1.7B-DFlash-Drafter
The training data: 42,591 prompts with LFM3's own answers (63.3M output tokens) plus the 324-prompt held-out set, as token ids and text, with per-row source and license attributionLottolabs/LFM3-1.7B-SelfDistill-63M
The serving release (release s14-20261113): quantised checkpoint, Docker runtime image, the drafter in device format, launch.py, runtime source and the evidence for every number hereLottolabs/LFM3-1.7B-TT-Mixed-BFP4-BFP8-P150

On one P150 the release serves with the drafter on by default (python3 launch.py --device-ownership-confirmed; --drafter off to compare). A fresh anonymous download of the release reproduced chat-2K 419 tok/s and GSM8K 499 tok/s for one user.

Why you should consider training your own drafter

1. Decode on local hardware is almost always memory-bandwidth bound, so extra rows are nearly free. Every decode step reads all the weights whether it produces 1 token or 16. On our P150 a 1-row LFM3 step takes 3.52 ms and a 16-row verify step takes 4.80 ms: checking 15 guesses costs 36% more than producing one token. That spare capacity is wasted unless something proposes good guesses. The same holds on most GPUs and Macs for small and mid-size models at batch 1.

2. Speculative decoding never changes the output. A guessed token is only kept if it is exactly what the model would have produced at that position (same seeded sampler for sampled requests). A bad drafter only costs speed, never quality. That makes it one of the very few optimisations you can ship without a quality evaluation of the outputs.

3. Free drafters only help when the answer copies the prompt. Prompt lookup (n-gram matching against the prompt) is great for code edits and tool calls — we saw ~12 tokens kept per step and 5.5× speedups — but on plain chat it kept only about 1.3 tokens per step and gave ~1.0–1.3×. Ordinary prose needs a drafter that actually models the language.

4. Many models ship without one. LFM3-1.7B has no MTP heads and no official draft model. Generic small models from another family usually don't share the tokenizer (LFM3's 128K vocab doesn't match LFM2's), and even when they do, they don't imitate your model's style, your quantisation, or your thinking format.

5. Your own model is the best teacher, and it's cheap to ask it. A drafter's only job is to predict what the target will say next. Training it on the target's own outputs (self-distillation), generated by your own serving stack, gives it exactly the right distribution — including your quantised weights, chat template and reasoning style. You don't need any labelled data.

6. It's affordable. One consumer GPU and an afternoon/overnight run were enough for a 1.7B target. You don't need the 64-GPU budgets in published drafter papers to get a useful gain.

How we did it

Design: a single-pass block drafter (DFlash style)

We compared the published options against the P150's costs:

ApproachHow it draftsVerdict here
DFlash / DFlash2-style block drafterOne non-causal pass proposes a whole block of drafts, conditioned on the target's hidden statesChosen: one pass per round, fits our 8-row verify width
EAGLE-3One small autoregressive head pass per draft tokenRejected: LFM3's 128K-vocab output head costs 0.58 ms per pass on the P150, so 7 drafts would cost several ms
MTP headsExtra prediction heads trained with the modelLFM3 ships none
Prompt lookupCopy from the promptKept alongside the drafter for copy-heavy text

The drafter:

  • 3 transformer layers, hidden 2048, 16 query / 4 KV heads, SwiGLU 6144 — 157M trainable parameters.
  • Reads LFM3's hidden states after layers 2, 12 and 23. Each tap is RMS-normalised, then the three are fused by one linear layer into a context row per position; the drafter attends over the last 2,048 of these.
  • Block of 8: the current token plus 7 masked positions, all predicted in one pass. Block 16 learned slower and acceptance flattened after ~4 drafts anyway.
  • Reuses LFM3's own frozen output head, cut down to a 65,536-token draft vocabulary (the most frequent tokens in LFM3's own outputs). That covers 97.6–99.9% of output tokens depending on domain and halves the head cost.

Two changes from the stock DFlash recipe made the difference between a drafter that learned at bigram level and one that learned properly:

  1. Normalise every hidden-state tap before fusing them.
  2. A "direct" input path: every block row starts from the anchor token's embedding plus the previous position's fused features. This doubled early position-1 accuracy (14% vs 7% at step 300).

Data: let the model write its own training set

  • 42,591 prompts from public instruction sets: UltraChat, WildChat, OASST2, Aya (multilingual), Magicoder and evol-codealpaca (code), GSM8K train, MetaMathQA and NuminaMath (math), Hermes and Glaive function-calling (tool use). Rendered with LFM3's chat template, thinking on/off 50/50.
  • Answers generated by our own P150 server with 16 concurrent users: half greedy, half the default sampling preset. 63.35M output tokens in 5.6 hours (3,119 tok/s aggregate).
  • Evaluation prompts kept strictly disjoint from training (no UltraChat test, no GSM8K test, nothing overlapping our quality eval set).
  • Training targets (hidden states + next-token distributions) were computed on the RTX 3090 with the Hugging Face model loaded with our served quantised weight values, so the drafter imitates the model we actually serve. Its greedy choices matched the P150's on 99.7–100% of positions checked.

Training on one RTX 3090

  • Loss per block row: half KL divergence to the target's distribution (over the draft vocab), half cross-entropy on the token actually generated, weighted towards early positions (weight e^−(i−1)/4).
  • AdamW, lr 4e-4 → 4e-5 cosine, 300 warm-up steps, 16K target tokens per step, bf16, flex-attention block masks.
  • 11,000 steps at ~2 s/step ≈ 6 hours, 18.5 GB of GPU memory, about 2 passes over the data.

Per-position draft accuracy rose steadily the whole run:

Training stepAccuracy at draft position 1 … 7
6000.30, 0.18, 0.11, 0.08, 0.07, 0.06, 0.06
4,5400.61, 0.42, 0.29, 0.20, 0.14, 0.11, 0.09
10,5600.69, 0.52, 0.40, 0.31, 0.23, 0.18, 0.14

Evaluate offline before touching the runtime

Before writing any device code we replayed held-out LFM3 answers and counted how many tokens each drafter would have had accepted per verify round, then fed that through a cost model built from measured step times (it predicted measured prompt-lookup speeds within ~10%).

Tokens kept per round (higher is better):

Domain (default sampling)Prompt lookupTrained drafter (7 drafts)
Chat1.282.00
Code1.342.23
Math1.472.66
Tool calls1.492.56
Multilingual1.261.81
All, greedy / sampled1.60 / 1.522.34 / 2.07

We set a stop rule in advance: abandon the port if the projected gain on chat-2K and GSM8K was under ~10%, or if one drafter pass cost more than ~1 ms on the device. Neither triggered.

Running it on the P150

  • The drafter runs on the card next to the target, in the card's block-floating-point formats (BFP8 attention, BFP4 MLP and head). That cost ~1% of acceptance.
  • One draft pass: 0.67–0.78 ms for one user (0.79–1.02 ms for two).
  • The biggest serving win was deferring the pass: it's launched right after a verify step and read only when the next round is planned, so it runs while vLLM does its own host work. That cut its critical-path cost from 0.93 ms to about 0.5 ms.
  • Calibrated depth: the drafter's top-1 vs top-2 logit margin predicts acceptance well (e.g. 26% at a margin under 0.25, 99% above 8), so each round drafts only as deep as is worth verifying.
  • Combined policy: prompt lookup when the text matches 3+ prompt tokens (copying), the trained drafter otherwise.

Proving it's exact

Speed only counts if the output is unchanged, so every build went through gates:

  • Speculation on vs off: identical text, 18/18 seeded and 6/6 greedy requests.
  • 16-user server: every concurrent reply equals the same request served alone, at 2–16 users, with joins, leaves, aborts, stop strings, tool calls and mixed sampling (362/362).
  • Streamed tool calls equal non-streamed.
  • Device drafter logits match a host reference of the same weights (PCC ≥ 0.99978).

Results on the P150 server

Single user: the table at the top. On a 16-slot server the drafter helps where there are spare rows — chat-2K 1.23× at 1–2 users, GSM8K 1.48× at 1 user and 1.19× at 4 — and is neutral (within ±2.5%) at 8–16 users, where the verify step is already full of real users' tokens. Memory at 16 users × 131K context still fits with the drafter's cache.

Does it generalise to prompts it never saw?

The held-out set above was disjoint from training but drawn from the same datasets, so we also ran 242 prompts from sources that were never used for training, across 9 task types: MT-Bench / Arena-Hard-style open questions, AlpacaEval, HumanEval and MBPP code, 4–16K-token document tasks, FLORES-style translation, JSON and table output, creative writing, role-play personas and multi-turn agentic tool use. Thinking was on and off. Every prompt ran on a one-user P150 server three ways: with the drafter, with prompt lookup only, and with no speculation (1.09M output tokens per setup). All 242 greedy and 726 seeded outputs were token-identical across the three.

The gain carries over: 1.21× (greedy) and 1.18× (default sampling) over prompt lookup alone, and 1.71× / 1.49× over no speculation. Offline acceptance on these prompts equals the in-distribution held-out set: 2.30 vs 2.34 tokens kept per round greedy, 2.08 vs 2.07 sampled.

Prompt type (never trained on)Drafter (greedy / sampled tok/s)Lookup onlyNo speculationDrafter vs lookup
Code from a spec (HumanEval, MBPP)523 / 444371 / 329279 / 2721.41× / 1.35×
Agentic tool use, multi-turn485 / 421365 / 328278 / 2711.33× / 1.28×
Open questions (MT-Bench / Arena-Hard style)436 / 377341 / 303279 / 2711.28× / 1.24×
AlpacaEval449 / 371351 / 306279 / 2711.28× / 1.21×
Structured output (JSON, tables)517 / 445405 / 354278 / 2711.28× / 1.26×
Creative writing402 / 359321 / 302279 / 2711.25× / 1.19×
Role-play / persona381 / 325321 / 283279 / 2721.19× / 1.15×
Long documents (4–16K tokens)389 / 320359 / 304267 / 2601.08× / 1.05×
Translation636 / 502620 / 491278 / 2701.03× / 1.02×
All 242 prompts475 / 403392 / 341277 / 2701.21× / 1.18×

Medians of 3 seeds for sampled runs.

Where it fades, and why:

  • Role-play is the hardest text to guess (persona voices): 1.15–1.19×.
  • Long documents gain only 5–8%: prompt lookup already copies a lot from the document, and the drafter guesses less well there. A 16K-token drafter window instead of 2K did not help.
  • Translation is about break-even, for two reasons:
    • the 65K-token draft vocabulary misses 8% of non-English output tokens (16–29% for German, Polish, Hebrew, Thai and Korean);
    • with thinking on, the model copies its own draft translation from its reasoning, and our runtime pauses the drafter's context during such copy runs, which then blocks the drafter for the next 2,040 tokens (68–86% of its possible rounds lost).
    • Into English with thinking off it still gains 1.11–1.17×.
  • It never made a category slower than lookup alone (lowest 0.99×). Individual prompts dipped to 0.88–0.92×, but every output stayed identical.

Lesson: test your drafter on prompt sources you didn't train on, by task type. Ours held up on everything except multilingual output, where the fix is a draft vocabulary or training mix with more non-English text.

What we'd tell you before you try

  • Measure your verify cost curve first. The whole idea depends on extra rows being cheap. On LFM3 / P150 they are (1 → 16 rows +36%); on our Qwen3.8-27B port each extra row costs ~1.8 ms of GDN recurrence, so the economics are tighter.
  • Generate data with your real serving stack and real quantisation. The drafter should imitate what you actually serve.
  • Keep prompt lookup. The trained drafter alone keeps ~1.6 tokens per round on code edits where lookup keeps 10–12; combine them.
  • Use a reduced draft vocabulary if your model has a big vocab — the output head is the most expensive part of a small drafter.
  • Normalise the hidden-state taps and give every block row the anchor embedding. Without those our drafter learned at bigram level for the first ~500 steps.
  • Evaluate offline with a cost model and a stop rule before writing kernels.
  • Overlap the draft pass with host work — on our stack that was worth more than any kernel tuning.
  • Gate exactness, especially with concurrent users: the bugs we hit along the way (stale page tables between traces, streaming tool-call parsing with multi-token deltas) only showed up under those gates.
  • Test out of distribution, by task type. Our gains held on unseen code, chat, agentic and structured-output prompts but faded on translation and long documents; a single aggregate number would have hidden that.

Setup

  • Model: LiquidAI/LFM3-1.7B, our GPTQ mixed BFP4/BFP8 TT quant, release s14-20261113 of Lottolabs/LFM3-1.7B-TT-Mixed-BFP4-BFP8-P150.
  • Inference hardware: 1 × Tenstorrent Blackhole P150 (32 GB GDDR6), vLLM + TT plugin, custom fused decode kernels.
  • Training hardware: 1 × RTX 3090 24 GB.
  • Drafter: 3-layer DFlash-style block drafter, block 8, 157M parameters, 65,536-token draft vocab, trained 11,000 steps on 63M self-generated tokens.
Discussion

0 comments

Questions, reproduction notes, and follow-up results.

No comments yet. Start the technical discussion.