Speed test matrix

Qwen-AgentWorld-35B-A3B-FP8 on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 30, 2026, 11:03 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve lovedheart/Qwen-AgentWorld-35B-A3B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only

Prefill

prompt tok/s by context length

8K context7,7491,051 ms TTFT

16K context7,3722,186 ms TTFT

32K context7,0114,576 ms TTFT

64K context5,98010,706 ms TTFT

128K context4,43028,869 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	125.7137 ms TTFT	222.8235 ms TTFT	376.1231 ms TTFT	601.3347 ms TTFT	943.6400 ms TTFT	1,421.2607 ms TTFT	2,101.91,594 ms TTFT	2,849.51,141 ms TTFT
16K	120.8278 ms TTFT	207.3380 ms TTFT	335.5490 ms TTFT	504.4852 ms TTFT	728.51,530 ms TTFT	1,019.62,733 ms TTFT	Over capacity	Over capacity
32K	115.6334 ms TTFT	195.4399 ms TTFT	310.7642 ms TTFT	452.1978 ms TTFT	618.81,642 ms TTFT	Over capacity	Over capacity	Over capacity
64K	108.3569 ms TTFT	177.3707 ms TTFT	268.41,047 ms TTFT	367.91,568 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	96.3867 ms TTFT	147.81,121 ms TTFT	208.71,658 ms TTFT	269.72,620 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
248K	79.41,894 ms TTFT	112.52,371 ms TTFT	147.43,414 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	7,749.0	—	1,051 ms	—	—	1	—	—
Prefill	16K	1	Valid	7,372.0	—	2,186 ms	—	—	1	—	—
Prefill	32K	1	Valid	7,011.0	—	4,576 ms	—	—	1	—	—
Prefill	64K	1	Valid	5,980.0	—	10,706 ms	—	—	1	—	—
Prefill	128K	1	Valid	4,430.0	—	28,869 ms	—	—	1	—	—
Sustained decode	0	1	Valid	125.7	125.8	137 ms	7.9 ms	1.0	0	3,771	30.0 s
Sustained decode	0	2	Valid	222.8	111.6	235 ms	9.0 ms	2.0	0	6,683	30.0 s
Sustained decode	0	4	Valid	376.1	94.4	231 ms	10.6 ms	4.0	0	11,284	30.0 s
Sustained decode	0	8	Valid	601.3	76.1	347 ms	13.1 ms	8.0	0	18,040	30.0 s
Sustained decode	0	16	Valid	943.6	60.1	400 ms	16.6 ms	16.0	0	28,304	30.0 s
Sustained decode	0	32	Valid	1,421.2	45.7	607 ms	21.9 ms	32.0	0	42,624	30.0 s
Sustained decode	0	64	Valid	2,101.9	33.9	1,594 ms	29.5 ms	64.0	0	63,040	30.0 s
Sustained decode	0	128	Valid	2,849.5	22.7	1,141 ms	44.1 ms	128.0	0	85,504	30.0 s
Sustained decode	16K	1	Valid	120.8	120.9	278 ms	8.3 ms	1.0	0	3,622	30.0 s
Sustained decode	16K	2	Valid	207.3	103.8	380 ms	9.6 ms	2.0	0	6,220	30.0 s
Sustained decode	16K	4	Valid	335.5	83.9	490 ms	11.9 ms	4.0	0	10,064	30.0 s
Sustained decode	16K	8	Valid	504.4	62.8	852 ms	15.9 ms	8.0	0	15,128	30.0 s
Sustained decode	16K	16	Valid	728.5	45.1	1,530 ms	22.2 ms	16.0	0	21,856	30.0 s
Sustained decode	16K	32	Valid	1,019.6	31.0	2,733 ms	32.3 ms	32.0	0	30,592	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	115.6	115.8	334 ms	8.6 ms	1.0	0	3,467	30.0 s
Sustained decode	32K	2	Valid	195.4	97.8	399 ms	10.2 ms	2.0	0	5,860	30.0 s
Sustained decode	32K	4	Valid	310.7	77.7	642 ms	12.9 ms	4.0	0	9,320	30.0 s
Sustained decode	32K	8	Valid	452.1	56.3	978 ms	17.8 ms	8.0	0	13,560	30.0 s
Sustained decode	32K	16	Valid	618.8	38.1	1,642 ms	26.2 ms	16.0	0	18,560	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	108.3	108.7	569 ms	9.2 ms	1.0	0	3,249	30.0 s
Sustained decode	64K	2	Valid	177.3	88.8	707 ms	11.3 ms	2.0	0	5,318	30.0 s
Sustained decode	64K	4	Valid	268.4	66.8	1,047 ms	15.0 ms	4.0	0	8,051	30.0 s
Sustained decode	64K	8	Valid	367.9	45.4	1,568 ms	22.0 ms	8.0	0	11,032	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	96.3	96.8	867 ms	10.3 ms	1.0	0	2,890	30.0 s
Sustained decode	128K	2	Valid	147.8	74.2	1,121 ms	13.5 ms	2.0	0	4,432	30.0 s
Sustained decode	128K	4	Valid	208.7	52.3	1,658 ms	19.1 ms	4.0	0	6,260	30.0 s
Sustained decode	128K	8	Valid	269.7	33.6	2,620 ms	29.8 ms	8.0	0	8,088	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	79.4	80.0	1,894 ms	12.5 ms	1.0	0	2,382	30.0 s
Sustained decode	248K	2	Valid	112.5	56.4	2,371 ms	17.7 ms	2.0	0	3,374	30.0 s
Sustained decode	248K	4	Valid	147.4	36.5	3,414 ms	27.4 ms	4.0	0	4,420	30.0 s
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.