LocalMaxxing field report — 4 concurrent streams × 4K context (max tokens/second)

Date: 2026-09-11. Engine: rainference (C++20 + HIP, no Python, no ggml runtime), working tree with manifests 054/055 applied. Hardware: single AMD Radeon RX 9070 XT 16 GB (gfx1201, RDNA4), Arch Linux, ROCm 7.2.4, Ryzen 9 9900X, 60 GB RAM.

Recording: run_multistream_dflash_4x4k/ (4 stream outputs, timings.json with per-token arrival times, prompts + the client used).

Setup

Method

Four concurrent streams, each with 3,400 prompt tokens and 512 generated tokens, greedy (temperature 0), cold prefill (no KV reuse), one warmup request on a different prompt before the recorded burst. All four requests are released simultaneously (client barrier); the per-token arrival times come from streaming SSE deltas. VRAM sampled every 200 ms.

Prompts: the canonical LocalMaxxing reasoning-v1 task followed by a distinct ~3.1K-token section of this project's documentation per stream (so the four sequences share no prefix — identical prompts would hit the prefix cache and turn the prefill warm). sha256 of the concatenated prompts: e69e82abc92ed095667605a80283c09f62edcf89a53dd8339524f49bb46b4044; concatenated outputs: 08bc9fa890fa7ccac8035c2afca808f162a6e6a32c9f11dffb21db102c33cf30.

Each stream produced a 1.2-1.6 KB answer; the first tokens continue the trailing attached document, then the model works the task in a <think> block (same shape as the 64K run).

Three metrics are reported, because they answer different questions:

• steady window — from the last stream's first token to the last token of the run: all four streams are decoding, no prefill is outstanding. This is the serving throughput number.
• decode phase — from each stream's first token: includes contention with the other streams' remaining prefills (the scheduler interleaves chunked prefill with decode).
• wall — the whole burst including the four cold prefills.

Results (4 streams × 3,400-token prompt, 512 generated each)

2,048 output tokens per run; 1,947 of them inside the 19.8 s steady window in the recorded run.

Findings

• At four concurrent streams, speculative decoding buys nothing here. DFlash 98.3, plain decode 98.2, MTP 96.0 t/s — a 2.4% spread, i.e. run-to-run noise, while spec costs 0.8 GB (MTP) to 1.65 GB (drafter) of VRAM and slows the prefill (759 vs 871 tok/s aggregate) because the drafter also has to run during prompt processing. Spec decode is a latency lever: on the canonical 254-token prompt at batch 1 the same engine measures 86.6 t/s (MTP) / 83.0 t/s (DFlash) against 35.9 t/s plain (session of 2026-09-11, run_*_reasoning/).
• The steady-window rate is ~4× the per-stream rate (98.3 vs 24.6 t/s), which is what batched decode is for: one weight pass serves all four sequences. It sits within 10% of the raw decode_batch B=4 kernel rate measured in manifest 050 (107 t/s at a shorter context).
• Prefill dominates the wall clock at 4K-context streams: 4 × 3,400 = 13,600 prompt tokens take 15.6-17.9 s (759-871 tok/s aggregate), while 2,048 generated tokens take ~20 s. For agent-style serving with long prompts, prefill chunking and prefix reuse are worth more than any decode-side lever.
• VRAM is the binding constraint at 4 slots: with the drafter the run peaks at 15.91 GB of the 15.92 GB card — effectively full, with only 0.45 GiB free at load (the desktop compositor shares the GPU). The no-spec configuration peaks at 14.26 GB, leaving 1.66 GB for the desktop or a longer context; that headroom is also what would pay for q8 KV instead of q4 (measured neutral for throughput at these context lengths).
• The engine's compiled slot limit is 4 sequences; there is no 8-stream configuration.

Reproduction

cd rainference
cmake --preset gpu-release && cmake --build --preset gpu-release --target server
RAI_SRV_CHUNK=256 ./build/gpu-release/server \
  --model ~/models/qwen3.8/Qwen3.8-27B-iq4s256-cs.raif \
  --tokenizer ~/models/qwen3.8/Qwen3.8-27B-UD-Q3KX-f16.gguf \
  --ctx 16384 --slots 4 --emb-host --kv-quant q4 \
  --draft ~/models/qwen3.8/Qwen3.8-27B-DFlash2-iq4s256.raif
# warmup (a different prompt, so the recorded prompts stay cold)
python3 benchmarks/localmaxxing/run_multistream_dflash_4x4k/client.py --port 8167 \
  --max-tokens 16 --out /tmp/warm.json --prompt benchmarks/localmaxxing/prompt_code-v1.txt
# recorded burst
python3 benchmarks/localmaxxing/run_multistream_dflash_4x4k/client.py --port 8167 \
  --max-tokens 512 --out /tmp/ms.json --outdir /tmp/ms_out \
  --prompt benchmarks/localmaxxing/run_multistream_dflash_4x4k/prompt{1,2,3,4}.txt
# comparison runs: same command without --draft (MTP) and with --no-spec (plain)

Reproduction notes: restart the server between configurations and between the warmup and the recorded burst — the scheduler's prefix cache otherwise serves a second run from cache and the prefill measurement becomes meaningless.