Speed test matrix

Qwen3.8-Flash-Next-exl3 on CMP 170HX 64GB ×2

vllm · 0.32.0rc1.dev249+g18305948a · EXL3 5.05 bpw6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 29, 2026, 06:09 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
GPU count: 2
VRAM: 64 GB (2 × 32 GB)
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: EXL3 5.05 bpw
Backend: cuda
Repository: github.com/Morrowmake/vllm-cmp170hx
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: qwen3.8-flash-next
Revision: 5.05bpw_h6_ng6
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Published recipe run as-is: Morrowmake's qwen38-flash-next-cmp170hx-recipe (github.com/Morrowmake/qwen38-flash-next-cmp170hx-recipe @ 3506c23587) on the vllm-cmp170hx engine image. Thank you Morrowmake for the recipe and engine, and turboderp for the EXL3 weights. Only the port and local model path were set. The recipe was measured on x16 links; these cards run at PCIe Gen2 x4. Spec decode: MTP, 3 tokens lookahead. KV cache: bfloat16
# 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit each, PCIe Gen2 x4 links | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04
git clone https://github.com/Morrowmake/qwen38-flash-next-cmp170hx-recipe && cd qwen38-flash-next-cmp170hx-recipe && git checkout 3506c23587 && ./start.sh
# serve.sh launches: vllm serve Qwen3.8-Flash-Next-exl3 --tensor-parallel-size 2 --enable-expert-parallel --dtype bfloat16 --kv-cache-dtype bfloat16 --load-format exl3 --max-model-len 262144 --gpu-memory-utilization 0.95 --max-num-seqs 8 --max-num-batched-tokens 3460 --compilation-config {"cudagraph_mode":"PIECEWISE"} --speculative-config {"method":"mtp","num_speculative_tokens":3}

Prefill

prompt tok/s by context length

8K context1,3885,906 ms TTFT

16K context1,42911,315 ms TTFT

32K context1,44922,200 ms TTFT

64K context1,46243,871 ms TTFT

128K context1,45987,741 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	44.4277 ms TTFT	92.6325 ms TTFT	173.62,092 ms TTFT	278.0591 ms TTFT	271.4Requests queued646 ms TTFT	276.6Requests queued712 ms TTFT	267.1Requests queued809 ms TTFT	Over capacity
16K	41.2271 ms TTFT	90.4340 ms TTFT	176.2635 ms TTFT	271.9955 ms TTFT	282.6Requests queued1,055 ms TTFT	271.8Requests queued1,136 ms TTFT	Over capacity	Over capacity
32K	50.2393 ms TTFT	92.3466 ms TTFT	183.9663 ms TTFT	279.51,204 ms TTFT	287.6Requests queued1,262 ms TTFT	Over capacity	Over capacity	Over capacity
64K	45.1528 ms TTFT	96.6715 ms TTFT	181.81,112 ms TTFT	285.11,876 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	47.0915 ms TTFT	102.61,241 ms TTFT	191.81,661 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
248K	48.01,579 ms TTFT	97.42,263 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	1,388.0	—	5,906 ms	—	—	1	—	—
Prefill	16K	1	Valid	1,429.0	—	11,315 ms	—	—	1	—	—
Prefill	32K	1	Valid	1,449.0	—	22,200 ms	—	—	1	—	—
Prefill	64K	1	Valid	1,462.0	—	43,871 ms	—	—	1	—	—
Prefill	128K	1	Valid	1,459.0	—	87,741 ms	—	—	1	—	—
Sustained decode	0	1	Valid	44.4	44.8	277 ms	22.3 ms	1.0	0	1,331	30.0 s
Sustained decode	0	2	Valid	92.6	46.6	325 ms	21.6 ms	2.0	0	2,775	30.0 s
Sustained decode	0	4	Valid	173.6	40.5	2,092 ms	24.7 ms	4.0	0	5,209	30.0 s
Sustained decode	0	8	Valid	278.0	35.9	591 ms	27.9 ms	8.0	0	8,330	30.0 s
Sustained decode	0	16	Requests queued	271.4	34.7	646 ms	28.8 ms	8.0	0	8,127	29.9 s
Sustained decode	0	32	Requests queued	276.6	33.3	712 ms	30.1 ms	8.0	0	8,297	30.0 s
Sustained decode	0	64	Requests queued	267.1	34.5	809 ms	29.0 ms	8.0	0	8,009	30.0 s
Sustained decode	0	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	41.2	42.5	271 ms	23.5 ms	1.0	0	1,236	30.0 s
Sustained decode	16K	2	Valid	90.4	45.2	340 ms	22.1 ms	2.0	0	2,712	30.0 s
Sustained decode	16K	4	Valid	176.2	44.6	635 ms	22.4 ms	4.0	0	5,280	30.0 s
Sustained decode	16K	8	Valid	271.9	34.8	955 ms	28.7 ms	8.0	0	8,142	29.9 s
Sustained decode	16K	16	Requests queued	282.6	34.4	1,055 ms	29.1 ms	8.0	0	8,475	30.0 s
Sustained decode	16K	32	Requests queued	271.8	34.3	1,136 ms	29.1 ms	8.0	0	8,155	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	50.2	50.7	393 ms	19.7 ms	1.0	0	1,507	30.0 s
Sustained decode	32K	2	Valid	92.3	46.6	466 ms	21.5 ms	2.0	0	2,767	30.0 s
Sustained decode	32K	4	Valid	183.9	45.0	663 ms	22.2 ms	4.0	0	5,513	30.0 s
Sustained decode	32K	8	Valid	279.5	35.7	1,204 ms	28.0 ms	8.0	0	8,384	30.0 s
Sustained decode	32K	16	Requests queued	287.6	35.0	1,262 ms	28.6 ms	8.0	0	8,614	29.9 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	45.1	46.7	528 ms	21.4 ms	1.0	0	1,352	30.0 s
Sustained decode	64K	2	Valid	96.6	48.9	715 ms	20.5 ms	2.0	0	2,898	30.0 s
Sustained decode	64K	4	Valid	181.8	45.2	1,112 ms	22.1 ms	4.0	0	5,451	30.0 s
Sustained decode	64K	8	Valid	285.1	34.6	1,876 ms	28.9 ms	8.0	0	8,537	29.9 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	47.0	47.2	915 ms	21.2 ms	1.0	0	1,408	30.0 s
Sustained decode	128K	2	Valid	102.6	52.4	1,241 ms	19.1 ms	2.0	0	3,076	30.0 s
Sustained decode	128K	4	Valid	191.8	47.2	1,661 ms	21.2 ms	4.0	0	5,753	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	48.0	48.9	1,579 ms	20.4 ms	1.0	0	1,438	30.0 s
Sustained decode	248K	2	Valid	97.4	49.2	2,263 ms	20.3 ms	2.0	0	2,921	30.0 s
Sustained decode	248K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.