LocalMaxxing field report — one 64K-context stream (useful long context, as fast as it goes)

Date: 2026-09-11. Engine: rainference (C++20 + HIP), working tree with manifests 054/055. Hardware: single AMD Radeon RX 9070 XT 16 GB (gfx1201, RDNA4), Arch Linux, ROCm 7.2.4, Ryzen 9 9900X, 60 GB RAM.

Recording: run_longctx_nospec_64k_v2/ (prompt.txt, output.txt, timings.json) — the current engine. The original recording of this configuration is kept in run_longctx_nospec_64k/ (146.3 s prefill) with its prefill-chunk A/B; the difference between the two is a flash-attention change described in benchmarks/manifests/056_prefill64k_profile_flash_vpath.md. The prompt and the generated output are byte-identical between the two recordings (sha256 139fbb1d… / 4b6b0d7f…).

Setup

Method

One stream, cold prefill, greedy (temperature 0), one warmup run before recording (the standard LocalMaxxing protocol), 260 generated tokens. The prompt is composed: the canonical reasoning-v1 task statement followed by ~64.8K tokens of this project's own documentation and manifests, so the model has a real long context to attend over.

• prompt: 65,100 tokens / 195,285 chars, sha256 139fbb1dffbd3d61ce0a135dcfe24236f394eaba729346f9560a9fa831c7e0ca
• output: 260 tokens / 782 chars, sha256 4b6b0d7f763d9bc243d790a682c9c64244e016f8719e30f8fb54679e5c5a84a8
• The prompt ends mid-document, so the model first continues the trailing document text for a few tokens, then answers the reasoning task that was placed at the start of the prompt inside a <think> block — i.e. the 65K tokens in between do not drown the instruction.

Results

Prefill chunk A/B (pre-change build, still valid for the ordering): chunk 128 → 397 tok/s, chunk 256 → 445 tok/s (146.3 s); decode identical and peak VRAM 14.09 vs 14.24 GB. Chunk 256 is the recorded configuration. Kernel-profile attribution and the 1.48x flash-attention fix that took this run from 146.3 s to 99.4 s are in benchmarks/manifests/056_prefill64k_profile_flash_vpath.md.

Findings

• Speculative decoding does not run past 4,096 tokens of context in this engine — by construction, not by a flag: the MTP head / drafter need the trunk hidden state of every position ahead of them, stored in an 80 MiB/slot history capped at kHhistTokensMax = 4096. Beyond it spec_step falls back to plain decode. At 64K the decode rate is therefore the plain-decode rate (23.45 t/s), not the 83-87 t/s the same model reaches at batch 1 with a short context. Making the history a bounded window (catch-up streaming during prefill) is the single biggest remaining decode lever for long-context serving; it is not implemented.
• The 64K configuration is only possible because of this session's memory work: q4 KV halves the pool (2,240 → 1,120 MiB), the logits/snapshot/history trims remove ~310 MiB of scratch, and the embedding staging frees 521 MiB. Before them the same 65K context did not fit with the 12.9 GiB of weights on a 16 GiB card.
• Prefill is the wall clock, not decode: 99 s to read 65,100 tokens vs 11 s to generate 260. Attention over a growing context dominates: aggregate prefill falls from ~1,000+ tok/s at short prompts to 631 tok/s at 32K and 445 tok/s at 64K. Long-context TTFT is the user-visible cost here (146 s ≈ 0.44 tok/s of prompt), which is why prefix caching matters more than decode-side tuning for repeated long-context work.
• q8 KV also fits at 64K in this configuration (workspace +1.1 GiB, ≈0.6 GiB headroom left) and the measured decode rate is identical at 32K (25.2 vs 25.1 t/s q4 vs q8); q4 was chosen for headroom with the desktop compositor resident, at +0.9% perplexity (measured at 32K).
• Peak VRAM 14.24 GB leaves room for the desktop and for the KV growth of a second session, but not for a second 64K stream on this card.

Reproduction

cd rainference
cmake --preset gpu-release && cmake --build --preset gpu-release --target rain
RAI_TOK_GGUF=~/models/qwen3.8/Qwen3.8-27B-UD-Q3KX-f16.gguf \
RAI_NO_SPEC=1 RAI_EMB_HOST=1 RAI_KV_Q4=1 RAI_PF_CHUNK=256 \
RAI_BENCH_OUT=benchmarks/localmaxxing/run_longctx_nospec_64k \
./build/gpu-release/rain ~/models/qwen3.8/Qwen3.8-27B-iq4s256-cs.raif \
  /tmp/lmprompts/long64k.txt bench 260

The prompt was composed from the canonical prompt_reasoning-v1.txt plus this repo's planning/*.md, benchmarks/manifests/*.md and agents.md, tokenized to 65,100 tokens (the composing step is a throwaway script; the recorded prompt.txt is the artifact).