CMP 170HX: Qwen3.8-27B AutoRound + DFlash2 at 131k

Setup

Single CMP 170HX 64GB (Ampere sm80, driver unlock to 64 GB HBM) on a Ryzen 5 5600G with 1×8 GB DDR4, Ubuntu 26.04.1. This is a daily chat box, not a short-context speed-run.

Weights: dbirks/Qwen3.8-27B-W4A16-AutoRound (~19 GB). Draft: local Qwen3.8-27B-DFlash2. Engine: vLLM 0.29 (CUDA 13.2).

vllm serve /home/dario/models/Qwen3.8-27B-W4A16-AutoRound \
  --served-model-name Qwen3.8-27B \
  --max-model-len 131072 --max-num-seqs 4 \
  --attention-backend FLASH_ATTN --kv-cache-dtype bfloat16 \
  --language-model-only --enable-prefix-caching \
  --reasoning-parser qwen3 --tool-call-parser qwen3_coder \
  --speculative-config '{"method":"dflash","model":"/home/dario/models/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'

What we optimized for

• Keep 131k context available (not 8k).
• 100+ tok/s on real decode beats a bigger empty window.
• Ampere: W4A16 + DFlash2. No NVFP4, no FP8-as-speed.

Results (batch=1, thinking off, stream decode, TTFT excluded)

Peak HBM 56.5 GB. KV pool at 131k was ~395k tokens.

What we are not claiming

• Club ~296 tok/s AutoRound rows used --max-model-len 8192. Different recipe.
• SGLang 1.3k tok/s / Ornith 3k tok/s are 8k-ctx or MoE-3B-active / 9B, not this 27B @ 131k.
• A repeated-prefix run hit ~345 tok/s here because DFlash cheated on copy-paste filler. Discarded.

MTP on this card was slower than DFlash2 (~102 vs ~118–133 tok/s short). 262k context fits in HBM but decode falls off a cliff (~17 tok/s at ~217k prefix), so production stay at 131k.