Speed test matrix

Qwen3.8-Flash-Next-W4A16-AutoRound on CMP 170HX 64GB ×2

vllm · 0.30.0 · W4A166 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 29, 2026, 07:26 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
GPU count: 2
VRAM: 64 GB (2 × 32 GB)
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.30.0
Quantization: W4A16
Backend: cuda
Repository: github.com/AlexBrandes/vllm
Build: ghcr.io/alexbrandes/cmp170hx-qwen38-flash-next@sha256:d6fc872d943b3e9e5cc9eb583b51d72af200e039fbbb7401700bb38d665f0957

Run

Served model: qwen3.8-flash-next-w4a16
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Published recipe run as-is: AlexBrandes' cmp170hx-qwen38-flash-next (github.com/AlexBrandes/cmp170hx-qwen38-flash-next @ c0db9613d1) on its pinned engine image. Thank you AlexBrandes for the recipe and engine fork, and Intel for the W4A16 AutoRound checkpoint (rev 4c67bf68). Only the port and local paths were set, and the recipe's optional host tune was skipped so the cards stay at our 180 W cap (the recipe sets 170 W). Spec decode: MTP, 5 tokens lookahead
# 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit each, PCIe Gen2 x4 links | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04
git clone https://github.com/AlexBrandes/cmp170hx-qwen38-flash-next && cd cmp170hx-qwen38-flash-next && git checkout c0db9613d1 && HOST_TUNE=0 ./start.sh
# start.sh launches: vllm serve Qwen3.8-Flash-Next-W4A16-AutoRound --pipeline-parallel-size 2 (VLLM_PP_LAYER_PARTITION=26,22) --max-model-len 262144 --block-size 1680 --mamba-cache-mode align --mamba-ssm-cache-dtype float32 --max-num-seqs 8 --max-num-batched-tokens 2048 --gpu-memory-utilization 0.95 --enable-prefix-caching -cc=<config/compilation.json> --speculative-config {"method":"mtp","num_speculative_tokens":5} (int4 draft head)

Prefill

prompt tok/s by context length

8K context5,4941,492 ms TTFT

16K context5,8502,765 ms TTFT

32K context5,9975,363 ms TTFT

64K context6,13510,452 ms TTFT

128K context6,13220,883 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	70.41,495 ms TTFT	110.6431 ms TTFT	170.4673 ms TTFT	298.8731 ms TTFT	305.3Requests queued783 ms TTFT	311.0Requests queued842 ms TTFT	313.0Requests queued959 ms TTFT	293.4Requests queued1,165 ms TTFT
16K	66.8736 ms TTFT	119.41,156 ms TTFT	153.41,632 ms TTFT	287.82,781 ms TTFT	286.2Requests queued2,784 ms TTFT	282.1Requests queued2,927 ms TTFT	Over capacity	Over capacity
32K	64.5726 ms TTFT	120.11,099 ms TTFT	168.11,663 ms TTFT	297.12,478 ms TTFT	296.4Requests queued2,627 ms TTFT	Over capacity	Over capacity	Over capacity
64K	70.1829 ms TTFT	121.11,358 ms TTFT	173.12,017 ms TTFT	290.42,999 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	72.11,340 ms TTFT	108.51,677 ms TTFT	165.42,450 ms TTFT	308.43,818 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
248K	82.61,997 ms TTFT	105.92,975 ms TTFT	176.34,370 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	5,494.0	—	1,492 ms	—	—	1	—	—
Prefill	16K	1	Valid	5,850.0	—	2,765 ms	—	—	1	—	—
Prefill	32K	1	Valid	5,997.0	—	5,363 ms	—	—	1	—	—
Prefill	64K	1	Valid	6,135.0	—	10,452 ms	—	—	1	—	—
Prefill	128K	1	Valid	6,132.0	—	20,883 ms	—	—	1	—	—
Sustained decode	0	1	Valid	70.4	70.9	1,495 ms	14.1 ms	1.0	0	2,110	30.0 s
Sustained decode	0	2	Valid	110.6	55.5	431 ms	18.0 ms	2.0	0	3,317	30.0 s
Sustained decode	0	4	Valid	170.4	40.4	673 ms	24.7 ms	4.0	0	5,107	30.0 s
Sustained decode	0	8	Valid	298.8	37.6	731 ms	26.6 ms	8.0	0	8,963	30.0 s
Sustained decode	0	16	Requests queued	305.3	38.5	783 ms	26.0 ms	8.0	0	9,145	30.0 s
Sustained decode	0	32	Requests queued	311.0	38.6	842 ms	25.9 ms	8.0	0	9,328	30.0 s
Sustained decode	0	64	Requests queued	313.0	38.0	959 ms	26.3 ms	8.0	0	9,379	30.0 s
Sustained decode	0	128	Requests queued	293.4	38.0	1,165 ms	26.3 ms	8.0	0	8,800	30.0 s
Sustained decode	16K	1	Valid	66.8	66.8	736 ms	15.0 ms	1.0	0	2,004	30.0 s
Sustained decode	16K	2	Valid	119.4	59.0	1,156 ms	17.0 ms	2.0	0	3,581	30.0 s
Sustained decode	16K	4	Valid	153.4	38.4	1,632 ms	26.1 ms	4.0	0	4,601	30.0 s
Sustained decode	16K	8	Valid	287.8	34.8	2,781 ms	28.7 ms	8.0	0	8,631	30.0 s
Sustained decode	16K	16	Requests queued	286.2	35.8	2,784 ms	28.0 ms	8.0	0	8,580	30.0 s
Sustained decode	16K	32	Requests queued	282.1	34.9	2,927 ms	28.7 ms	8.0	0	8,459	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	64.5	66.0	726 ms	15.2 ms	1.0	0	1,933	30.0 s
Sustained decode	32K	2	Valid	120.1	60.0	1,099 ms	16.7 ms	2.0	0	3,603	30.0 s
Sustained decode	32K	4	Valid	168.1	42.4	1,663 ms	23.6 ms	4.0	0	5,041	30.0 s
Sustained decode	32K	8	Valid	297.1	37.2	2,478 ms	26.9 ms	8.0	0	8,906	30.0 s
Sustained decode	32K	16	Requests queued	296.4	36.4	2,627 ms	27.5 ms	8.0	0	8,893	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	70.1	70.0	829 ms	14.3 ms	1.0	0	2,100	30.0 s
Sustained decode	64K	2	Valid	121.1	59.1	1,358 ms	16.9 ms	2.0	0	3,630	30.0 s
Sustained decode	64K	4	Valid	173.1	44.5	2,017 ms	22.5 ms	4.0	0	5,192	30.0 s
Sustained decode	64K	8	Valid	290.4	36.0	2,999 ms	27.8 ms	8.0	0	8,702	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	72.1	73.5	1,340 ms	13.6 ms	1.0	0	2,163	30.0 s
Sustained decode	128K	2	Valid	108.5	54.4	1,677 ms	18.4 ms	2.0	0	3,255	30.0 s
Sustained decode	128K	4	Valid	165.4	40.1	2,450 ms	25.0 ms	4.0	0	4,960	30.0 s
Sustained decode	128K	8	Valid	308.4	37.6	3,818 ms	26.6 ms	8.0	0	9,241	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	82.6	83.5	1,997 ms	12.0 ms	1.0	0	2,474	30.0 s
Sustained decode	248K	2	Valid	105.9	53.6	2,975 ms	18.7 ms	2.0	0	3,175	30.0 s
Sustained decode	248K	4	Valid	176.3	43.5	4,370 ms	23.0 ms	4.0	0	5,290	30.0 s
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.