Strata dual-16 GB (5080+5060 Ti): verified 98.1/102.8 tok/s, 262k context, 3 full evals — Unsloth UD-Q4_K_XL

Qwen3.8-Flash-Next Unsloth UD-Q4_K_XL on dual 16 GB (RTX 5080 + RTX 5060 Ti) — Strata expert-streaming field report

Summary: Reproducible speed + quality field report: canonical speed tests, reasoning disabled (on Strata v0.1.43: reasoning-v1 96.7 tok/s, code-v1 84.8 tok/s; on v0.1.42: 95.7 / 88.1; on config B: 98.1 / 102.8 — all verified-eligible, MTP acceptance 78–91% on the canonical short tests), verified-at-depth runs at 16k/65k/131k on both engine versions, a cold-context scaling sweep (v0.1.42: prefill plateaus ~3,300 tok/s, reproduced on v0.1.43), and three full pooled quality evals — HellaSwag 92.66%, GSM8K 98.10%, ARC-Challenge 98.43%, each 13/13 shards, APPROVED (HumanEval+ 88% / MBPP+ 86% are local dry-run probes only, not submitted). Submitted and approved: fifteen speed/depth rows spanning v0.1.42 → v0.1.43 config B, plus the three eval datasets' full shard coverage.

Setup

• Model: unsloth/Qwen3.8-Flash-Next-GGUF UD-Q4_K_XL (180B MoE, 4 shards, ~104 GB weights)
• Engine: Strata v0.1.43 (commit 7695859), https://github.com/Niko1221/Strata — expert-streaming MoE runtime, not llama.cpp. Earlier sections measured on v0.1.42 (commit 61b3fb5); the v0.1.43 re-run is below.
• Hardware: RTX 5080 16 GB + RTX 5060 Ti 16 GB (heterogeneous dual-GPU), Ryzen 7 7800X3D, 96 GB DDR5, Arch Linux
• Launch v0.1.43 (verbatim live process args, re-verified at measurement time; paths abbreviated with ... — full snippet in the run files' engineFlags.commandSnippet): user-recalibrated config: --spec-min-p 0.5 --pool-workers 4, hand-pinned --pcie-frac/--adapt-* dropped (v0.1.43 auto-measures PCIe share: 57.9 GB/s → 0.55 on the 5080, 14.5 GB/s → 0.40 on the 5060 Ti; route tail skip ON by default).

~/Projects/Strata/engine/strata --serve --pack .../Strata-data/packs/unsloth-ud-q4_k_xl --native .../unsloth-UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --expert-profile .../Strata/data/expert-profile.bin --expert-cache auto --prefill auto --spec 4 --mtp .../Strata-data/mtp/rt --max-context 262144 --kv int8 --kv-resident 32768 --resident-budget-gib 66 --vision --vram-reserve-mib 700 --layer-split 43 --split-device 1 --control-vector-scaled .../experimental-speed-projection/Qwen3.8-Flash-Next-experimental-speed-projection.gguf:1.0 --control-vector-layer-range 4 44 --cvec-mode project --cvec-dir per-layer --spec-min-p 0.5 --pool-workers 4

• Launch v0.1.42 (historical, for the runs submitted Oct 11 morning): same minus --spec-min-p 0.5 --pool-workers 4, plus --pcie-frac 0.55 --spec-min-p 0.70 --adapt-every 1 --adapt-swaps 80 --adapt-decay 0.97.

• Tooling: localmaxxing CLI v0.1.48, custom engine name + full provenance fields

Speed — canonical prompts (verified-eligible)

Version note: the table in this subsection was measured on v0.1.42 (submitted and approved). The v0.1.43 re-measurement follows in "Strata v0.1.43 re-run" below the verified-at-depth section.

Protocol: 2 warmups + 3 timed streaming iterations, median reported; temperature 0, batch 1, cache-bust nonce per request; 512 output tokens; reasoning disabled (reasoning_effort: "none" in every request) so the measurement covers answer-token decode, comparable to other users' reasoning-off runs. Measured with a protocol-matched script because lmx remote mode cannot inject reasoning_effort and this server's sidecar defaults reasoning to high.

Reasoning-ON comparison (same rig, same protocol, thinking tokens measured): reasoning-v1 86.3 tok/s, code-v1 68.3 tok/s — i.e. thinking mode costs 10–21% reported decode because the 512-token budget is consumed by reasoning text. Those earlier payloads are superseded by the reasoning-off versions; the reasoning-on figures remain in the run-file notes.

Prefill methodology note: the engine's raw prompt_n (266 / 201 tokens) is lower than the usage-reported prompt tokens (345 / 283) — one counter excludes chat-template/cvec overhead. With reasoning off, client-estimated and engine-derived prefill agree within 1%, so both are reported as equivalent. The run-file headline tokSPrefill is the client-estimated value (timingSource: client_observed_http).

Both payloads pass the server-side verified check (verified: true, zero issues): canonical prompt hashes, output hashes/samples matching the measured (reasoning-off) generation, raw engine timings, consistent MTP stats. Note: Strata's --spec-min-p 0.70 gates MTP proposals adaptively, so effective proposals/pass ≈ 2.2 despite --spec 4; specNumTokens is reported as the effective mean (2) because the consistency check assumes fixed-n speculation.

Context scaling (cold-context sweep, same server — measured on v0.1.42)

Measured with measure-context-sweep.py (this directory): per-level seeded filler + nonce-first defeats the prefix cache so every timed prefill is genuinely cold; 2 warmups + 3 timed streaming iterations per level, median; temp 0, reasoning off; prompt size calibrated against the endpoint's own usage counter (on HTTP 400 the server error states the exact token count and the script rescales). The baseline row is the canonical reasoning-v1 prompt — the same prompt as the submitted speed tests — so the short→long step-down is visible in one table. Raw data: data/cold-context-sweep.json in this directory (pass-1 n=2 data preserved as data/cold-context-sweep-pass1.json).

Peak during sweep: 15,846 + 15,894 MiB VRAM; per-GPU power maxima 261 W + 124 W (not necessarily simultaneous). Client and engine timings agree within ≈1% at every level (consistently +1.0–1.1%).

Findings:

• Prefill has no cliff. It rises from 2.9k to ~3.3k tok/s between 16k and 65k (consistent with larger prompts amortizing chunk overhead — mechanism not isolated), then plateaus flat to 131k. TTFT scales linearly at ~3.3k tok/s: a 100k-token prompt costs ≈ 30 s to first token.
• Decode steps down ~15–25% from short context to ≥16k, then stays flat to 131k. Within-level spread reaches ±10 tok/s (shared single-slot server with interactive traffic), so the individual level-to-level differences (e.g. 32k > 16k) are noise; the baseline→long-context step-down is outside that noise and appears identically in client and engine numbers.
• Plausible mechanism (not isolated): MTP draft acceptance drops from 86.5% at baseline to 74.8–81.2% at depth for most levels; speculation efficiency, not KV read bandwidth, may carry part of the step-down. The 131k row's 86.7% acceptance shows the effect is not monotonic in depth.
• Baseline reconciliation: the speed table above reports 95.7 tok/s (512-token generation, separate session); this sweep's baseline row is 90.0 tok/s (128-token generation, later session). The step-down column is computed against the same-session 90.0 baseline — the 5.7 tok/s gap between the two baselines is session-to-session variance on a shared server, and using 95.7 instead would overstate the step-down by ~6 points.
• Correction to the earlier KV-cache sweep section (this report's previous revision): the old 128K point was already a fully cold prefill (cached_tokens: 0; engine prefill 3,282 tok/s — matching this sweep's 3,311). Its prefill number was never wrong; only its TTFT and decode (57.6) were load-contaminated. The old 16K–96K rows measured warm-prefix decode (99.9% cached) and are superseded by this table for cold numbers.

Superseded warm-prefix data (kept for reference): lmx kvcache run levels 16K–128K, warm decode 65.4–83.1 tok/s, cache reuse 99.9% at 16K–96K — that tool pre-warms the prefix by design and times only the uncached suffix, which is why it cannot measure cold prefill.

Strata v0.1.43 re-run (same protocol, user-recalibrated config)

Re-measured after the user upgraded to v0.1.43 and recalibrated the config (--spec-min-p 0.5, --pool-workers 4, PCIe/adapt flags now engine-auto). Same canonical prompts, reasoning off, 2 warmups + 3 timed, idle windows (one launch-contaminated reasoning-v1 pass discarded and re-run clean; pass-1 kept as data/*-pass1-contaminated.json). Raw data: data/reasoning-off-*.json, data/verified-at-depth.json. All five payloads (runs/unsloth-Qwen3.8-Flash-Next-GGUF/v0143-*.json) dry-ran verified: true, zero issues, privacy-clean, and passed an independent review gate; submitted and APPROVED (ids cmv42bnin… reasoning-v1, cmv42bobw… code-v1, cmv42bp5s… 16k, cmv42bq05… 65k, cmv42bqvz… 131k).

Reading the deltas honestly. The v0.1.43 release's NVIDIA gains are for cards that miss experts (route tail skip) and non-Blackwell kernels; on this rig the release itself is expected near-neutral. The measured deltas mix engine and user-config changes and are within/near session-to-session variance on a shared server: reasoning-v1 +1.0 (within noise), code-v1 −3.3 (within the spread of its own samples), at-depth −2.5/+2.2/−4.6 across levels with no consistent direction. Draft acceptance dropped on the short runs (90.8→81.8%) — consistent with --spec-min-p 0.5 proposing more drafts per pass (specNumTokens 2→3, draft_n 391→433) at lower per-draft acceptance; net decode is what matters and it is unchanged within noise. Prefill is unchanged within 2% — the v0.1.43 prompt_n semantics change (engine-read tokens only, cache_n separate) did not shift the numbers here because cache_n=0 on all cold runs. Peak power draw is lower on the short runs (148+55 W vs 259+121 W) at equal decode — consistent with the adaptive-tier copy overlap doing its job, though a single sample each. Conclusion: v0.1.43 + the recalibrated config performs the same as v0.1.42 within measurement noise; no regression, no significant gain.

Second v0.1.43 re-run — tuning config B (v0143b-*, submitted APPROVED)

Same evening, after the user re-tuned: --resident-budget-gib 72, --layer-split 43, --spec-min-p 0.5, env STRATA_ROUTE_RESIDENT=1 (control vector, --split-device/--pool-workers/hand-pinned --pcie-frac all dropped from the live launch; PCIe share auto-measured 0.55/0.40; adaptive-tier copy overlap is inert on a layer split). Quality gate first: 12-question greedy probe to data/probes-20261012/probe-split43.txt — 10/12 raw; recheck with thinking budget cleared probe 08 (12:25), probe 05 remained truncated in the artifact (both failures are max_tokens truncations of the reasoning preamble, not wrong answers; probe 05 was never cleanly re-passed on record — earlier probe-post-tweak.txt shows 11/12). Zero wrong-answer failures attributable to the config. Measured in idle windows with the lss probe parked; live engine args logged at chain start and end.

*config A's code-v1 84.8 was its own re-measurement; a same-day repeat under the same args measured 98.3–102.2 — the 84.8 was measured before the day's RAM/split changes and the code-v1 improvement (~+16% over v0.1.42, tight 0.5-tok spread) is attributable to budget 72 GiB + split 43 + auto PCIe share, not to v0.1.43 itself.

Submitted after dry-run verified: true ×5, privacy scan clean, and server readback: all five APPROVED, verified=true — ids cmv5hub9d… (reasoning-v1 98.1), cmv5hudya… (code-v1 102.8), cmv5huh2v… (16k), cmv5hujud… (65k), cmv5humkt… (131k). At-depth numbers vary vs config A within ±5% with mixed signs — no direction, i.e. at-depth decode unchanged; the short-context code-v1 gain is the real move. Note the acceptance-rate convention unchanged (meanLen = outTokens/(outTokens−accepted)).

Verified-at-depth runs (canonical task at depth, idle window)

A follow-up measurement (measure-verified-at-depth.py, data/verified-at-depth.json) places the canonical reasoning-v1 prompt as the final turn of a 16k/65k/131k-token conversation (seeded filler + nonce ahead → cold prefill, evidence cached_tokens: 0), 512 output tokens, same protocol, run while the server had no competing traffic. All three payloads dry-ran verified: true, zero issues, and are now submitted and APPROVED (ids cmv3xv0hb… 16k, cmv3xv1b2… 65k, cmv3xv26p… 131k) — the first verified long-context runs on this model's page.

Correction to the step-down claim above: on the canonical task at depth, decode is 87–90.5 tok/s — essentially at the short-context baseline (90.0–95.7), a 0–9% decline, not 15–25%. The earlier filler-continuation sweep measured a different generation task (free continuation of synthetic records) with lower MTP acceptance (74.8–86.7% vs 88.1–92.6% here) and ran while the agent's own traffic shared the single server slot. Both factors inflated the apparent step-down. What survives from the cold sweep: the prefill plateau (~3,300 tok/s, TTFT ≈ 30 s per 100k tokens) — reproduced identically here — and the fact that decode at depth is stable, not cliff-shaped. The task-dependence of decode (via draft acceptance) is now the strongest signal across both datasets: structured, predictable output sustains acceptance; free continuation does not.

Quality evals

v0.1.43 full-coverage runs (submitted, server-verified coverage)

Chain: run-eval-full-chain.sh — --all-missing --submit, all 13 shards per dataset, reasoning off (--thinking-level off), temp 0, chat exact_match scoring on text suites, sandbox code execution on code suites. Paused by user after 2.2/5 datasets (PC needed); --all-missing makes re-running the same script a clean resume (covered shards are skipped by the duplicate guard).

Evidence note. The 15 speed/depth rows attached to this report are the full server-verified set for this quant (each shows verifiedRun: true on its run page; /api/speed-tests is authoritative). The eval shard rows are also on the server — 39 approved shards, visible on this model's Benchmarks tab and the per-dataset leaderboards — but the report API's evalRunIds field rejects them: the validator requires eval runs bound to a user account, and shards submitted with an API key carry no user binding (every id we submitted — shard-level and aggregate, POST and PATCH paths — returned "Eval runs must belong to you and target this model"). This affects all key-submitted evals on the site, which is why no published report links eval evidence. Disclosing rather than hiding it; the pooled numbers above are taken verbatim from the server's own aggregates.

Pooled scores read from the unauthenticated leaderboard API (GET /api/benchmarks/<slug>/leaderboard?hfId=…&quantization=Q4_K_XL&quantFormat=gguf) and cross-checked against the cumulative aggregate object in the final shard of each local run file — both sources agree exactly. Chat-scoring caveat from below applies to these runs as well.

Comparison vs the v0.1.42 samples: HellaSwag 92.66% pooled vs 94.0% on the earlier 200-question sample — within sampling noise (the earlier sample's Wilson CI is 89.8–96.5), so no regression is demonstrated, but neither is parity. GSM8K 98.10% pooled vs 95.05% on the earlier single shard — a genuine improvement beyond the earlier sample's CI.

v0.1.42 single-shard runs (superseded by the full-coverage runs above)

All evals ran against the same live server, sequential, concurrency 1, temperature 0.

Methodology caveat (important for comparability): Strata exposes only /v1/chat/completions — no /v1/completions, no echo-logprobs — so the canonical loglikelihood scoring is impossible on this engine (it errors on every question). All four scores above use chat-format exact_match/code-execution scoring. They are directly comparable to other chat-scored runs, and should not be blended blindly into loglikelihood-scored leaderboards. Code-eval scores are strict (HumanEval+/MBPP+ extended tests, sandboxed execution). Scores are single-shard samples (one shard draw each), not full-benchmark numbers.

Observations

• A 104 GB MoE at 95.7 tok/s on v0.1.42 — rising to 98.1 (reasoning) / 102.8 (code) tok/s on the v0.1.43 config B rows this report's title leads with — on 32 GB of VRAM + DDR5 + one NVMe pack is the headline: expert streaming with a profiled cache (--expert-profile, --pcie-frac 0.55) plus MTP (91% acceptance measured on the v0.1.42 suite) is what makes dual-16 GB viable for this model class.
• Power efficiency is notable: ≤380 W combined peak draw (259 W + 121 W, per-GPU maxima) measured at 95.7 tok/s on the v0.1.42 suite for a 180B MoE; per-GPU power was not separately logged for the config B rows.
• Quality holds at Q4_K_XL on these suites: GSM8K 98.10% pooled (13/13 shards) and HumanEval+ 88% on local probes, zero scoring errors; no quant-induced failure modes (malformed answers, truncation loops) observed in the retained per-question artifacts.
• Gotchas found for other Strata users: GGUF-only HF repos have no tokenizer files (use the base repo id for token-count tooling); lmx needs engineName: custom + provenance fields for Strata; verified-run MTP stats need the effective-proposals workaround described above; chat-only endpoints need --scoring exact_match for eval shards.

Reproduce

Canonical prompt files and the reasoning-off measurement script are stored alongside this report (canonical-reasoning-v1.txt, canonical-code-v1.txt, measure-reasoning-off.py). The v0.1.43 run files were built by build-v0143-canonical.py / build-verified-at-depth.py (from TEMPLATE-canonical-at-depth.json + data/*.json) and gated with lmx speed-test dry-run; the measurement chain wrapper is run-v0143-speed-chain.sh (re-verifies live engine args against the template before measuring).

The headline speed numbers were produced by the reasoning-off script, not by lmx speed-test run — lmx remote mode cannot inject reasoning_effort, and this server defaults to thinking-on. To reproduce:

python3 measure-reasoning-off.py reasoning-v1 canonical-reasoning-v1.txt
python3 measure-reasoning-off.py code-v1 canonical-code-v1.txt
# then write metrics + evidence into a run file via: lmx speed-test runs edit <run.json> --set-json '{...}'
# and validate with: lmx speed-test dry-run <run.json>

# quality evals (chat-scored; code evals need `lmx eval sandbox setup` first)
# NOTE: --model must be the catalog hfId + --quantization (the served alias errors on the eval path)
# NOTE: --questions N is silently capped at the drawn shard's size (GSM8K shard = 101 questions)
lmx eval shard hellaswag --base-url http://127.0.0.1:8080 --model unsloth/Qwen3.8-Flash-Next-GGUF --quantization Q4_K_XL \
  --questions 200 --scoring exact_match --dry-run --json > hellaswag.json
lmx eval shard gsm8k --base-url http://127.0.0.1:8080 --model unsloth/Qwen3.8-Flash-Next-GGUF --quantization Q4_K_XL \
  --questions 200 --scoring exact_match --dry-run --json > gsm8k.json
lmx eval shard humaneval-plus --base-url http://127.0.0.1:8080 --model unsloth/Qwen3.8-Flash-Next-GGUF --quantization Q4_K_XL \
  --questions 50 --dry-run --json > humaneval.json
lmx eval shard mbpp-plus --base-url http://127.0.0.1:8080 --model unsloth/Qwen3.8-Flash-Next-GGUF --quantization Q4_K_XL \
  --questions 50 --dry-run --json > mbpp.json

# Cold-context scaling sweep (this repo; per-level seeded filler + nonce defeats the prefix cache)
python3 measure-context-sweep.py   # baseline + 16k/32k/65k/98k/131k, 2 warmups + 3 timed, no submissions

# Superseded warm-prefix KV sweep (lmx kvcache pre-warms the prefix by design; token counting
# uses the base repo because the GGUF repo has no tokenizer files)
lmx kvcache run custom --mode remote --base-url http://127.0.0.1:8080 \
  --hf-id Qwen/Qwen3.8-Flash-Next --quantization Q4_K_XL \
  --levels 16384,32768,65536,98304,131072 --output-tokens 128

All measurements taken 2026-10-11 → 2026-10-12 on the author's machine. All 15 speed/depth rows (v0.1.42/v0.1.43 canonical prompts plus the config A/B verified-at-depth sweeps) were submitted after explicit authorization and are live, verified, and linked to this report; the 39 eval shards (13 per dataset × 3 datasets) are submitted and approved, with the link limitation covered by the evidence note above. This report was published 2026-10-13. Run files and eval payloads are local and validated.