Speed test matrix

Qwen3.5-122B-A10B-GPTQ-Int4 on CMP 170HX 64GB ×2

vllm · 0.32.0rc1.dev249+g18305948a · GPTQ Int46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 29, 2026, 10:30 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
GPU count: 2
VRAM: 64 GB (2 × 32 GB)
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: GPTQ Int4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none. Layout: PP2 (stock vLLM flag)
# 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve Qwen/Qwen3.5-122B-A10B-GPTQ-Int4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Prefill

prompt tok/s by context length

8K context5,7831,408 ms TTFT

16K context5,8002,779 ms TTFT

32K context5,4955,838 ms TTFT

64K context4,72913,538 ms TTFT

128K context3,57435,786 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	65.0126 ms TTFT	121.7130 ms TTFT	205.4218 ms TTFT	334.6323 ms TTFT	491.8504 ms TTFT	752.0759 ms TTFT	1,117.51,009 ms TTFT	1,628.11,533 ms TTFT
16K	63.2597 ms TTFT	105.2860 ms TTFT	196.51,243 ms TTFT	333.12,042 ms TTFT	510.23,700 ms TTFT	701.96,964 ms TTFT	953.813,814 ms TTFT	Over capacity
32K	61.5530 ms TTFT	102.6696 ms TTFT	190.3979 ms TTFT	315.61,524 ms TTFT	471.82,664 ms TTFT	638.25,093 ms TTFT	Over capacity	Over capacity
64K	59.4990 ms TTFT	99.01,350 ms TTFT	179.51,894 ms TTFT	284.92,991 ms TTFT	412.05,189 ms TTFT	Over capacity	Over capacity	Over capacity
128K	55.6903 ms TTFT	93.01,047 ms TTFT	159.71,639 ms TTFT	241.72,749 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
248K	49.62,119 ms TTFT	82.92,566 ms TTFT	108.13,760 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	5,783.0	—	1,408 ms	—	—	1	—	—
Prefill	16K	1	Valid	5,800.0	—	2,779 ms	—	—	1	—	—
Prefill	32K	1	Valid	5,495.0	—	5,838 ms	—	—	1	—	—
Prefill	64K	1	Valid	4,729.0	—	13,538 ms	—	—	1	—	—
Prefill	128K	1	Valid	3,574.0	—	35,786 ms	—	—	1	—	—
Sustained decode	0	1	Valid	65.0	65.0	126 ms	15.4 ms	1.0	0	1,948	30.0 s
Sustained decode	0	2	Valid	121.7	61.0	130 ms	16.4 ms	2.0	0	3,652	30.0 s
Sustained decode	0	4	Valid	205.4	51.5	218 ms	19.4 ms	4.0	0	6,162	30.0 s
Sustained decode	0	8	Valid	334.6	42.4	323 ms	23.6 ms	8.0	0	10,038	30.0 s
Sustained decode	0	16	Valid	491.8	31.6	504 ms	31.6 ms	16.0	0	14,752	30.0 s
Sustained decode	0	32	Valid	752.0	24.4	759 ms	41.0 ms	32.0	0	22,560	30.0 s
Sustained decode	0	64	Valid	1,117.5	18.4	1,009 ms	54.4 ms	64.0	0	33,505	30.0 s
Sustained decode	0	128	Valid	1,628.1	13.1	1,533 ms	76.6 ms	128.0	0	48,839	30.0 s
Sustained decode	16K	1	Valid	63.2	63.2	597 ms	15.8 ms	1.0	0	1,894	30.0 s
Sustained decode	16K	2	Valid	105.2	52.3	860 ms	19.1 ms	2.0	0	3,155	30.0 s
Sustained decode	16K	4	Valid	196.5	48.3	1,243 ms	20.7 ms	4.0	0	5,894	30.0 s
Sustained decode	16K	8	Valid	333.1	40.2	2,042 ms	24.9 ms	8.0	0	9,992	30.0 s
Sustained decode	16K	16	Valid	510.2	29.6	3,700 ms	33.7 ms	16.0	0	15,304	30.0 s
Sustained decode	16K	32	Valid	701.9	18.8	6,964 ms	53.3 ms	32.0	0	21,056	30.0 s
Sustained decode	16K	64	Valid	953.8	11.0	13,814 ms	91.1 ms	64.0	0	28,608	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	61.5	61.6	530 ms	16.2 ms	1.0	0	1,844	30.0 s
Sustained decode	32K	2	Valid	102.6	51.2	696 ms	19.5 ms	2.0	0	3,078	30.0 s
Sustained decode	32K	4	Valid	190.3	47.2	979 ms	21.2 ms	4.0	0	5,708	30.0 s
Sustained decode	32K	8	Valid	315.6	38.6	1,524 ms	25.9 ms	8.0	0	9,468	30.0 s
Sustained decode	32K	16	Valid	471.8	28.1	2,664 ms	35.6 ms	16.0	0	14,152	30.0 s
Sustained decode	32K	32	Valid	638.2	17.8	5,093 ms	56.1 ms	32.0	0	19,136	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	59.4	59.5	990 ms	16.8 ms	1.0	0	1,780	30.0 s
Sustained decode	64K	2	Valid	99.0	49.2	1,350 ms	20.3 ms	2.0	0	2,971	30.0 s
Sustained decode	64K	4	Valid	179.5	44.0	1,894 ms	22.7 ms	4.0	0	5,384	30.0 s
Sustained decode	64K	8	Valid	284.9	33.9	2,991 ms	29.5 ms	8.0	0	8,544	30.0 s
Sustained decode	64K	16	Valid	412.0	23.1	5,189 ms	43.3 ms	16.0	0	12,360	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	55.6	55.8	903 ms	17.9 ms	1.0	0	1,668	30.0 s
Sustained decode	128K	2	Valid	93.0	46.6	1,047 ms	21.5 ms	2.0	0	2,789	30.0 s
Sustained decode	128K	4	Valid	159.7	39.9	1,639 ms	25.1 ms	4.0	0	4,791	30.0 s
Sustained decode	128K	8	Valid	241.7	29.9	2,749 ms	33.5 ms	8.0	0	7,252	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	49.6	49.9	2,119 ms	20.1 ms	1.0	0	1,487	30.0 s
Sustained decode	248K	2	Valid	82.9	41.4	2,566 ms	24.2 ms	2.0	0	2,488	30.0 s
Sustained decode	248K	4	Valid	108.1	26.6	3,760 ms	37.6 ms	4.0	0	3,244	30.0 s
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.