Speed test matrix

Qwen3.5-4B-AWQ-4bit on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 29, 2026, 03:33 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: AWQ 4-bit
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve cyankiwi/Qwen3.5-4B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}

Prefill

prompt tok/s by context length

8K context9,663843 ms TTFT

16K context9,0491,781 ms TTFT

32K context8,1733,926 ms TTFT

64K context6,8239,383 ms TTFT

128K context5,03525,400 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	229.9121 ms TTFT	441.5237 ms TTFT	824.9214 ms TTFT	1,217.3277 ms TTFT	1,769.6314 ms TTFT	2,269.4485 ms TTFT	2,975.3794 ms TTFT	3,451.6984 ms TTFT
16K	137.8225 ms TTFT	268.6337 ms TTFT	487.7567 ms TTFT	635.1771 ms TTFT	749.91,276 ms TTFT	868.72,211 ms TTFT	Over capacity	Over capacity
32K	101.2361 ms TTFT	188.8442 ms TTFT	349.5678 ms TTFT	412.51,057 ms TTFT	462.51,952 ms TTFT	530.23,548 ms TTFT	Over capacity	Over capacity
64K	62.3531 ms TTFT	123.0713 ms TTFT	214.91,008 ms TTFT	241.91,525 ms TTFT	265.72,625 ms TTFT	Over capacity	Over capacity	Over capacity
128K	32.9999 ms TTFT	68.41,163 ms TTFT	120.81,802 ms TTFT	137.12,842 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
248K	17.91,840 ms TTFT	38.92,320 ms TTFT	69.13,245 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	9,663.0	—	843 ms	—	—	1	—	—
Prefill	16K	1	Valid	9,049.0	—	1,781 ms	—	—	1	—	—
Prefill	32K	1	Valid	8,173.0	—	3,926 ms	—	—	1	—	—
Prefill	64K	1	Valid	6,823.0	—	9,383 ms	—	—	1	—	—
Prefill	128K	1	Valid	5,035.0	—	25,400 ms	—	—	1	—	—
Sustained decode	0	1	Valid	229.9	278.1	121 ms	3.7 ms	1.0	1	6,897	30.0 s
Sustained decode	0	2	Valid	441.5	232.8	237 ms	4.3 ms	2.0	1	13,246	30.0 s
Sustained decode	0	4	Valid	824.9	210.6	214 ms	4.8 ms	4.0	0	24,743	30.0 s
Sustained decode	0	8	Valid	1,217.3	159.1	277 ms	6.3 ms	8.0	0	36,515	30.0 s
Sustained decode	0	16	Valid	1,769.6	115.7	314 ms	8.6 ms	16.0	0	53,070	30.0 s
Sustained decode	0	32	Valid	2,269.4	73.7	485 ms	13.6 ms	32.0	0	68,054	30.0 s
Sustained decode	0	64	Valid	2,975.3	48.2	794 ms	20.7 ms	64.0	0	89,182	30.0 s
Sustained decode	0	128	Valid	3,451.6	28.1	984 ms	35.5 ms	128.0	0	103,647	30.0 s
Sustained decode	16K	1	Valid	137.8	139.8	225 ms	7.2 ms	1.0	0	4,134	30.0 s
Sustained decode	16K	2	Valid	268.6	136.8	337 ms	7.3 ms	2.0	0	8,054	30.0 s
Sustained decode	16K	4	Valid	487.7	123.9	567 ms	8.1 ms	4.0	0	14,626	30.0 s
Sustained decode	16K	8	Valid	635.1	80.6	771 ms	12.4 ms	8.0	0	19,038	30.0 s
Sustained decode	16K	16	Valid	749.9	47.1	1,276 ms	21.2 ms	16.0	0	22,485	30.0 s
Sustained decode	16K	32	Valid	868.7	27.1	2,211 ms	36.9 ms	32.0	0	26,020	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	101.2	101.8	361 ms	9.8 ms	1.0	0	3,035	30.0 s
Sustained decode	32K	2	Valid	188.8	95.1	442 ms	10.5 ms	2.0	0	5,664	30.0 s
Sustained decode	32K	4	Valid	349.5	88.0	678 ms	11.4 ms	4.0	0	10,484	30.0 s
Sustained decode	32K	8	Valid	412.5	51.6	1,057 ms	19.4 ms	8.0	0	12,361	30.0 s
Sustained decode	32K	16	Valid	462.5	29.1	1,952 ms	34.4 ms	16.0	0	13,857	30.0 s
Sustained decode	32K	32	Valid	530.2	16.4	3,548 ms	61.2 ms	32.0	0	15,876	29.9 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	62.3	63.2	531 ms	15.8 ms	1.0	0	1,868	30.0 s
Sustained decode	64K	2	Valid	123.0	62.1	713 ms	16.1 ms	2.0	0	3,688	30.0 s
Sustained decode	64K	4	Valid	214.9	54.5	1,008 ms	18.3 ms	4.0	0	6,443	30.0 s
Sustained decode	64K	8	Valid	241.9	30.0	1,525 ms	33.3 ms	8.0	0	7,244	29.9 s
Sustained decode	64K	16	Valid	265.7	16.9	2,625 ms	59.1 ms	16.0	0	7,939	29.9 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	32.9	34.1	999 ms	29.3 ms	1.0	0	985	30.0 s
Sustained decode	128K	2	Valid	68.4	35.2	1,163 ms	28.4 ms	2.0	0	2,050	30.0 s
Sustained decode	128K	4	Valid	120.8	31.7	1,802 ms	31.5 ms	4.0	0	3,617	29.9 s
Sustained decode	128K	8	Valid	137.1	17.9	2,842 ms	55.8 ms	8.0	0	4,108	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	17.9	18.9	1,840 ms	52.8 ms	1.0	0	536	29.9 s
Sustained decode	248K	2	Valid	38.9	20.1	2,320 ms	49.8 ms	2.0	0	1,166	30.0 s
Sustained decode	248K	4	Valid	69.1	18.0	3,245 ms	55.5 ms	4.0	0	2,073	30.0 s
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.