Setup
Custom inference engine (rainference, C++20 + HIP, no Python, no ggml runtime) for Qwen3.8-27B on a
single AMD Radeon RX 9070 XT 16 GB (gfx1201, RDNA4), Arch Linux, ROCm 7.2.4, Ryzen 9 9900X, 60 GB RAM.
- Weights: custom IQ4_S256 (dual-scale 256-value superblocks, 4.25 bpw, 13.1 GB) — a locally authored quant co-designed with the kernels.
- KV cache: q8_0 paged blocks (16 tokens/page, block-table indirection).
- Prefill: chunked WY-form GDN + WMMA flash attention (RDNA4 WMMA,
gfx12). - Speculative decoding: DFlash2 external 1.9B block-diffusion drafter, converted to the same IQ4_S256 quant (0.99 GiB), consuming target residual taps at layers 6/20/34/48/62 with a 256-rank / top-16 candidate selector; alternative configs use the in-file MTP head or no spec.
Method
Canonical LocalMaxxing prompt reasoning-v1 (sha256 9000edaadb16fc77ab78b39917245a03d4d7585c3ba770cb045a47a0e0683445)
sent verbatim, temperature 0, batch 1, cold prefill (no KV reuse), one warmup run before recording.
254 prompt tokens, 320 generated tokens. Peak VRAM sampled with hipMemGetInfo during the run.
Results (RX 9070 XT 16 GB, greedy, 254-token prompt)
| Config | Decode | Prefill | TTFT | Spec acceptance | Mean len | Peak VRAM |
|---|---|---|---|---|---|---|
| No spec (plain greedy) | 35.0 tok/s | 795 tok/s | 320 ms | — | 1.00 | 14.63 GB |
| In-file MTP head, n=3 | 84.8 tok/s | 916 tok/s | 278 ms | 0.846 | 3.54 | 14.63 GB |
| DFlash2 drafter, n=3 | 80.8 tok/s | 754 tok/s | 337 ms | 0.800 | 3.40 | 15.46 GB |
Losslessness cross-check: all three configurations produced byte-identical output text
(sha256 037f3b408643…), i.e. speculative decoding here is distribution-exact for greedy decoding.
Notes
- The DFlash2 drafter pays a fixed per-round cost (feature-fusion KV injection through a 25600→5120 projection) which the in-file MTP head does not; on this prompt MTP is ~5% ahead, on other prompts the block drafter wins by 10–20% (measured 73.9 vs 63.0 tok/s on a 189-token prompt, 0.712 vs 0.664 acceptance).
- 15.46 GB of the 16 GB card is resident with the drafter; the prefill chunk size and the draft attention window are capped to fit (128 tokens / 1024 positions).
- The engine is not yet in LocalMaxxing's engine list, so this session is published as a report; speed-test payloads (with prompt/output hashes and raw timings) are prepared for submission once the engine is registered.
Reproduction (engine tree rainference @ 3f21145 + uncommitted working tree, manifests 051/052):
RAI_TOK_GGUF=<tokenizer.gguf> RAI_SPEC_N=3 RAI_DRAFT=Qwen3.8-27B-DFlash2-iq4s256.raif \
./build/rain Qwen3.8-27B-iq4s256-cs.raif benchmarks/localmaxxing/prompt_reasoning-v1.txt bench 320
banger lfg