895 tok/s from a 27B dense model on one RTX 5080

This is a speed story about dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF — an uncensored, ternary-quantized 27B dense Qwen3 derivative — running fully offloaded on a single RTX 5080 16GB under WSL2, at 13.7 GB VRAM and 165–240 W.

All five benchmark runs below are submitted and linked (see Runs). Five are Verified (canonical prompt + captured evidence + batch-1 + within the physical decode ceiling), including the 854.5 tok/s pure n-gram run; the account is a Verified user.

Results (canonical localmaxxing prompts, single stream, temp 0, ctx 32768)

* The two warm runs carry a verification note from the platform's acceptance formula, which assumes a single draft source capped at specNumTokens. This stack drafts from two sources (DFlash2, capped at 5, plus an n-gram layer with a 96-token window), so accepted + drafts/5 overshoots the emitted count. Every stat on those rows is engine-verbatim; the flags are a formula artifact, not a measurement problem.

Leaderboard standing on the RTX 5080 board: #1 (894.9) and #2 (556.8) overall; the three Verified runs (151.2 / 146.4 / 142.0) are the fastest verified 20B+ results on this GPU. Next-best 20B+ on a 5080: gpt-oss-20b (MoE) at 222 tok/s. For scale: the fastest sub-2B models on this GPU reach ~450 tok/s.

The three layers of speed

1. The quant. PQ2_0, 2.13 bpw ternary (PrismML fork format, group-128). 27B dense ≈ 6.8 GB of weight traffic per token. Decode is bandwidth-bound: at 949 GB/s measured DRAM ceiling that's a ~139 tok/s vanilla wall — and profiling shows the whole matvec stack already running at ~874 GB/s average, 92% of the ceiling. You cannot kernel your way past this; the only lever left is emitting more tokens per weight pass.

2. The drafting. DFlash2 self-speculative decoding (same-family 1.1 GB Q4_K draft head, lineage: ProCreations/Ternary-Bonsai-2-27B-DFlash2) plus the fork's ngram-mod layer with a 96-token window. On fresh content DFlash2 alone lifts 99 → 151 tok/s at 60.6% acceptance (submitted as its own run; output bit-identical to the hybrid stack at temp 0). The n-gram layer costs ~6% when cold (142 vs 151) — and pays back 6× once the workload revisits content: 99.9% acceptance, 895 tok/s. Run in isolation (--spec-type ngram-mod, no draft model at all, 11.3 GB VRAM instead of 13.7), the n-gram layer alone reaches 854.5 tok/s at literally 100% acceptance (750/750) on the reasoning prompt and 625.9 on code — both Verified. A 27B with no draft weights resident, drafting purely from its own context history, outruns every other 20B+ result on this GPU.

3. The patches. Four small diffs on the PrismML llama.cpp fork (prism-v7), all in the reproduction package:

• 0001 input-layer GPU offload — the token-embedding row lookup leaves the per-token CPU path; single CUDA split, no host round-trip per token
• 0002 async graph-input staging — CUDA-graph input buffers stage on the compute stream instead of synchronously
• 0003 optional PQ2_0 SoA matvec kernel — structure-of-arrays mul_mat_vec with bias fusion, proven bitwise-identical to the stock kernel; default off (no aggregate win at this model's shapes)
• 0004 remaining CUDA dispatcher plumbing

Patches 0001+0002+0004 were worth +11% fresh decode (122 → 135 tok/s on the internal code probe at the time) and a much snappier server. CUDA graphs are load-bearing on this stack (+70% vanilla decode) — the build recipe in the package keeps them on.

What "warm" means — read this before comparing

The headline 894.9 tok/s is the warm n-gram steady state: the same generation task re-run once the n-gram map has seen the content (output bit-identical across repeats — matching sha256s in the evidence). This is not a synthetic trick; it is the actual steady state of any agent, code editor, or chat session that revisits a file or conversation — which real workloads constantly do. The cold-map companion numbers (142–151 tok/s) are submitted side by side, on the same prompts, same hardware, same evidence standard, and those are Verified.

If you benchmark this stack once on a fresh prompt and stop, you'll see ~150 tok/s and a 27B that beats most 8B Q4s. If your workload iterates — and most local workloads do — you'll see 550–900.

The kernel wall (why the patches stop there)

nsys per-kernel profiling of the full production config (draft + graphs): lm_head at 954 GB/s, fused gate_up ~950, down 857, qkv 870 — ~874 GB/s average across the ~6.8 GB/token matvec stack. Of the remaining ~10 ms token budget, ~2.4 ms is non-matvec ops (FWHT 0.76 ms, rms_norm 0.44 ms, get_rows 0.42 ms, copies 0.21 ms) and ~0.9 ms is inter-graph gap. The kernel lever on this hardware/quant combination is exhausted; future gains live in drafting (longer n-gram hits, deeper DFlash2) and in shaving the non-matvec tail.

Reproduce it

Everything is in zotowata/bonsai2-dflash2-fastfork: the four patches, the exact serve profile, the streaming benchmark harness (engine tok/s, TTFT, prefill rate, acceptance, GPU power/VRAM sampling), raw run data, and the deployment kit (systemd unit, WSL2 keepalive, Windows helpers).

# PrismML llama.cpp fork, branch prism (prism-v7) + patches/000*.patch
cmake -B build-graphs -DGGML_CUDA=ON -DGGML_CUDA_GRAPHS=ON \
      -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release
cmake --build build-graphs --config Release -j

./build-graphs/bin/llama-server \
  -m Bonsai-2-27B-PQ2_0-CRACK.gguf \
  --model-draft Bonsai-2-27B-DFlash2-Q4_K.gguf \
  --spec-type draft-dflash,ngram-mod \
  --spec-draft-n-max 5 --spec-ngram-mod-n-max 96 \
  -ngl 999 -ngld 999 -fa on -c 32768 -np 1 -t 16 -tb 32 -b 2048 -ub 512

One honest note on determinism: the patched build is bit-identical to the prebuilt fork on prose, with one known greedy near-tie flip on a code probe (GPU dequant float non-associativity flips a last-bit tie). The SoA kernel patch (0003) is proven bitwise-identical across on/off/default configs.

Credits

• Base model: dealignai/Bonsai-2-27B-Ternary-CRACK-GGUF (Apache-2.0)
• Draft head lineage: ProCreations/Ternary-Bonsai-2-27B-DFlash2
• Engine: PrismML llama.cpp fork (branch prism, DFlash2 + PQ formats)
• Reproduction package: zotowata/bonsai2-dflash2-fastfork