The rig
| GPUs | 4x Intel Arc Pro B65, 32 GB each (128 GB total) |
| CPU / RAM | AMD EPYC 7343 (16 cores), 499 GB |
| OS | Ubuntu 26.04.1 LTS, xe driver |
| Engines | Upstream vLLM vllm/vllm-openai-xpu:v0.32.0; Intel llm-scaler 0.31.0-b1-experimental (llm-scaler 1.5 for one Muse upload); llama.cpp b11729 (SYCL, oneAPI 2026.1.1) |
| Bench | llm_decode_bench.py v0.8.0 through lmx v0.1.54 speed-test matrix run |
Every model below ran with tensor parallelism across 4 cards or on 1 card. All numbers come from the linked LocalMaxxing matrices and runs; nothing here was re-measured or edited for this report.
Method
- Grid: decode at contexts 0, 8K, 32K, 64K, 128K and 200K (204,800), by 1, 2, 4, 8 and 16 concurrent users; prefill-only at 8K, 32K and 128K. 30 s per cell, 256 output tokens, prefix caching on, up to 16 sequences.
- Settings: each arm reuses the config I already have listed on LocalMaxxing for that model, quant and card count, with speculative decoding off, at the model's own window. Cells beyond the served window are SKIPPED.
- Gate: the bench exited cleanly, the context-0 single-user cell is VALID and close to the model's earlier single-user run, no VALID cell collapses below 10% of context 0, and LOOPED/ERROR cells are a minority. Then LocalMaxxing's dry run.
- Stable fallback: each arm had a twin on the newest stable runtime (vLLM v0.31.0 or llm-scaler 1.5), run only when the current one failed or was held.
Results
Aggregate decode tok/s. "1u" = one user, "16u" = 16 users. "Deepest" is the largest context with a single-user cell. † = the bench flagged the cell capacity_limited.
| Model | Quant | Cards | Runtime | 1u ctx 0 | 1u deepest | Depth | 16u ctx 0 |
|---|---|---|---|---|---|---|---|
| Qwen3.6-27B | BF16 | 4 | vLLM 0.32.0 | 28.6 | 22.6 | 200K | 396.1 |
| Qwen3.6-27B | FP8 | 4 | llm-scaler | 48.1 | 32.1 | 200K | 626.7 |
| Qwen3.6-27B | INT4 | 4 | llm-scaler | 50.5 | 33.2 | 200K | 466.0 |
| Qwen3.6-27B | INT4 | 1 | llm-scaler | 20.3 | 14.8 | 128K | 162.2 |
| Qwen3.6-35B-A3B | BF16 | 4 | vLLM 0.32.0 | 88.7 | 54.2 | 200K | 855.6† |
| Qwen3.6-35B-A3B | FP8 | 4 | llm-scaler | 99.9 | 59.9 | 200K | 1,030.5 |
| Qwen3.6-35B-A3B | INT4 | 4 | llm-scaler | 98.6 | 58.8 | 200K | 833.4 |
| Qwen3.6-35B-A3B | INT4 | 1 | llm-scaler | 80.3 | 38.2 | 200K | 469.4 |
| Qwen3.8-27B | BF16 | 4 | vLLM 0.32.0 | 28.6 | 22.7 | 200K | 389.5 |
| Qwen3.8-27B | INT4 | 4 | llm-scaler | 50.4 | 33.8 | 200K | 458.1 |
| Qwen3.8-27B | INT4 | 1 | llm-scaler | 20.2 | 14.6 | 128K | 159.3 |
| Qwen3.8-27B-FP8 | FP8 | 4 | vLLM 0.32.0 | 30.5 | 23.2 | 200K | 404.2 |
| Qwen3.5-9B | BF16 | 4 | vLLM 0.32.0 | 82.8 | 54.7 | 200K | 1,132.8 |
| Qwen3.5-9B | FP8 | 4 | llm-scaler | 126.1 | 73.3 | 200K | 1,568.3 |
| Qwen3.5-9B | BF16 | 1 | vLLM 0.32.0 | 34.2 | 22.7 | 200K | 469.7 |
| Qwen3.5-122B-A10B-GPTQ-Int4 | GPTQ-4bit | 4 | llm-scaler | 52.7 | 35.7 | 200K | 539.6 |
| gpt-oss-20b | MXFP4 | 4 | vLLM 0.32.0 | 179.9 | 146.7 | 64K | 1,670.0 |
| gpt-oss-20b | MXFP4 | 1 | vLLM 0.32.0 | 97.3 | 72.1 | 64K | 760.9 |
| GLM-4.7-Flash | BF16 | 4 | vLLM 0.32.0 | 81.0 | 21.0 | 128K | 595.7 |
| Muse-Glimmer-30B | BF16 | 4 | vLLM 0.32.0 | 26.7 | 25.6 | 64K | 399.3 |
| Muse-Glimmer-30B | FP8 | 4 | llm-scaler | 36.3 | 34.4 | 64K | 476.4 |
| Muse-Glimmer-30B | FP8 | 4 | llm-scaler 1.5 | 48.8 | 45.1 | 64K | 343.7 |
| gemma-4-26B-A4B-it | BF16 | 4 | vLLM 0.32.0 | 78.9 | 25.9† | 64K | 674.3 |
| gemma-4-26B-A4B-it | FP8 | 4 | vLLM 0.32.0 | 85.8 | 26.6† | 64K | 856.0 |
| gemma-4-26B-A4B-it | FP8 | 1 | vLLM 0.32.0 | 59.5 | - | ctx 0 only | 425.3 |
| gemma-4-31B-it | BF16 | 4 | vLLM 0.32.0 | 22.5 | 12.3† | 32K | 284.5 |
| gemma-4-31B-it | FP8 | 4 | llm-scaler | 36.4 | 12.7† | 64K | 396.8 |
| gemma-4-31B-it | INT4 | 1 | llm-scaler | 13.7 | - | ctx 0 only | 207.3 |
| gemma-4-12B-it | BF16 | 4 | vLLM 0.32.0 | 50.7 | 16.8† | 64K | 648.0 |
| gemma-4-12B-it | BF16 | 1 | vLLM 0.32.0 | 21.8 | - | ctx 0 only | 262.4 |
"llm-scaler" without a version is 0.31.0-b1.
How much decode survives a long context
Four-card single-user decode as % of its own context-0 speed. "-" is beyond the served window.
| Model | Quant | ctx 0 tok/s | 8K | 32K | 64K | 128K | 200K |
|---|---|---|---|---|---|---|---|
| Muse-Glimmer-30B | BF16 | 26.7 | 100 | 98 | 96 | - | - |
| Muse-Glimmer-30B | FP8 (llm-scaler 1.5) | 48.8 | 98 | 95 | 92 | - | - |
| Qwen3.8-27B-FP8 | FP8 | 30.5 | 98 | 94 | 91 | 83 | 76 |
| Qwen3.6-27B / Qwen3.8-27B | BF16 | 28.6 | 98 | 95 | 91 | 84 | 79 |
| Qwen3.5-122B-A10B | GPTQ-4bit | 52.7 | 99 | 95 | 87 | 80 | 68 |
| Qwen3.8-27B | INT4 | 50.4 | 98 | 92 | 86 | 75 | 67 |
| Qwen3.5-9B | BF16 | 82.8 | 97 | 93 | 86 | 76 | 66 |
| Qwen3.6-35B-A3B | FP8 | 99.9 | 97 | 91 | 84 | 70 | 60 |
| Qwen3.5-9B | FP8 | 126.1 | 97 | 90 | 82 | 69 | 58 |
| gpt-oss-20b | MXFP4 | 179.9 | 97 | 91 | 82 | - | - |
| GLM-4.7-Flash | BF16 | 81.0 | 79 | 54 | 39 | 26 | - |
| gemma-4-31B-it | FP8 | 36.4 | 66 | 43† | 35† | - | - |
| gemma-4-26B-A4B-it | FP8 | 85.8 | 62 | 39 | 31† | - | - |
| gemma-4-12B-it | BF16 | 50.7 | 64 | 41† | 33† | - | - |
Qwen3.8-Flash-Next: prefill chunk size
Four cards, W4A16, 262K window, vLLM 0.30.0-based XPU build. --max-num-batched-tokens (the prefill chunk) was the bottleneck: the shipped default is 1,024.
| Matrix | Chunk | MTP | Prefill 8K | Prefill 32K | Prefill 128K (TTFT) | 1u ctx 0 | 1u 200K |
|---|---|---|---|---|---|---|---|
| 2048 | 2,048 | off | 3,856 | 3,628 | 2,586 (49.5 s) | 42.6 | 31.2 |
| 8192 | 8,192 | off | 5,306 | 4,950 | 3,603 (35.5 s) | 42.7 | 31.2 |
| 2048 MTP-3 | 2,048 | 3 | 3,961 | 3,488 | 2,454 (52.2 s) | 84.5 | 49.5 |
| 8192 MTP-3 | 8,192 | 3 | 5,192 | 4,842 | 3,420 (37.4 s) | 84.4 | 51.4 |
8,192 prefills 31-39% faster than 2,048 at 8K-128K, with the same decode. The cost is under mixed load: a stream that is already decoding pauses for up to 2-5 s while another user's long prompt prefills at 8,192, against at most 1.0-1.4 s at 1,024-2,048. A single user sees no pauses.
Findings
- Context costs Qwen, gpt-oss and Muse little. At 64K they keep 82-96% of their context-0 decode. Dense Qwen 27B BF16 still keeps 79% at 200K.
- gemma-4 and GLM-4.7-Flash fall off hard. At 64K they keep 31-39%. gemma-4 prefill is also slow on this stack: 26B BF16 does 2,219 / 679 / 341 tok/s at 8K / 32K / 64K (189 s TTFT at 64K), against 3,709 / 3,444 / 3,085 for Qwen3.6-27B BF16.
- gemma-4 at 128K+ stalls. At the full 262K window, the 26B decode cells at 128K and 200K returned no output on both vLLM 0.32.0 and 0.31.0 even though the 128K prefill completed, so every gemma-4 arm was run at a 131K window (31B BF16 at 64K, because it hung at 131K too).
- One B65 can't hold a useful gemma-4 KV cache. One card has room for 6-9K tokens of KV (12B BF16, 26B FP8, 31B INT4), so those three matrices are context-0 only.
- Best single user at 200K: Qwen3.5-9B FP8, 73.3 tok/s; then Qwen3.6-35B-A3B FP8 59.9 and INT4 58.8. Best 16-user throughput: gpt-oss-20b, 1,670 tok/s at context 0 and 1,400 at 32K.
- vLLM 0.32.0 vs the old nightly: 0.94-1.03x on every arm with a baseline, except GLM-4.7-Flash at 2.23x (81.0 against 36.4 tok/s). Its v0.31.0 twin ran at 12.0 and was not uploaded.
- Muse FP8 runtime trade-off: llm-scaler 1.5 wins single-user (48.8 against 36.3), 0.31.0-b1 wins at 16 users (476.4 against 343.7). Both are uploaded.
llama.cpp depth sweeps, 1 card
lmx kvcache run llama.cpp (llama-bench -p 512 -n 128 -d <depth>), b11729 SYCL. Decode tok/s, prefill in brackets.
| GGUF | 4K | 32K | 64K | 128K |
|---|---|---|---|---|
| unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL | 14.9 (615) | 10.8 (529) | 8.2 (458) | 5.6 (361) |
| Qwen/Qwen3.6-27B Q4_K_M | 20.4 (773) | 13.4 (642) | 9.7 (543) | 6.2 (413) |
| Qwen/Qwen3.6-35B-A3B UD-Q4_K_M | 69.9 (1,282) | 56.5 (1,090) | 46.4 (937) | 34.4 (733) |
At 128K the MoE keeps 49% of its 4K decode; the dense 27Bs keep 30-38%. Against vLLM INT4 on one card: Qwen3.6-27B is about even (20.4 vs 20.3), Qwen3.6-35B-A3B is behind (69.9 vs 80.3). Different benchmarks, so indicative only.
What this is not
- These are sustained-decode matrices, not LocalMaxxing Verified single-request runs. Speculative decoding was off, so single-user numbers are well below my spec-on uploads of the same configs (0.09x-0.63x).
- Speed only. No long-context answer-quality test.
- † cells carry the bench's own capacity flag and were accepted by LocalMaxxing as-is.