Qwen3.6-35B-A3B on dual RTX 3060 12GB (llama.cpp)
Reproducible local decode numbers for two NVIDIA GeForce RTX 3060 12GB cards (24GB combined, no NVLink). Model is Unsloth Qwen3.6-35B-A3B-UD-Q4_K_XL. Engine is llama.cpp b10682.
Repo: github.com/Captain-Tripps/qwen38-dual-rtx-3060
Setup
- GPUs: 2× RTX 3060 12GB, PCIe, no NVLink
- CPU: Xeon E5-2690 v4, 64GB RAM, Ubuntu 24.04
- Quant: UD-Q4_K_XL (MoE, ~3B active)
- Context: 90112 (88k), KV q4_0, MTP draft KV q4_0
- Speculation: native MTP only (
--spec-type draft-mtp --spec-draft-n-max 2) - Split:
--split-mode tensor --tensor-split 1,1 - Env:
GGML_CUDA_ALLREDUCE=internal
GGML_CUDA_ALLREDUCE=internal llama-server \
-m Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf \
-ngl 99 \
--split-mode tensor --tensor-split 1,1 \
-c 90112 -ctk q4_0 -ctv q4_0 \
--cache-type-k-draft q4_0 --cache-type-v-draft q4_0 \
-np 1 -fa on \
--spec-type draft-mtp --spec-draft-n-max 2
Results (29 Aug 2026)
Chat /v1/chat/completions, thinking off, 256 max tokens. Decode tok/s from llama-server timings.
| Config | Ctx | Code tok/s | Prose tok/s |
|---|---|---|---|
| b10413 layer 21,20 + MTP n=2 | 131k | 79.3 | 66.3 |
| b10682 tensor + MTP n=2 | 64k | 131.6 | 107.7 |
| b10682 tensor + MTP n=2, draft KV q4_0 (served) | 88k | 129.4 | 104.4 |
VRAM on the winner: 11887 MiB per card.
Context ceiling on 2×12GB
Tensor split + MTP at 131k OOMs in the draft KV alloc. Quantizing draft KV to q4_0 (it defaults to f16) is what unlocked 88k. 92k loads then dies on later requests. 96k + MTP does not fit. 96k without MTP does (~11255 MiB) but you give the decode win back.
Caveat for the leaderboard
These are served chat decode numbers with MTP on, not a no-spec llama-bench row. Short prompts, so prefill tok/s is not the story. Same dual-3060 box as the Qwen3.8-27B row (57 tok/s); this MoE is faster because ~3B experts are active.