Speed test matrix

Qwen3.5-9B-FP8-dynamic on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · FP8 dynamic6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 30, 2026, 07:34 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: FP8 dynamic
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve RedHatAI/Qwen3.5-9B-FP8-dynamic --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}

Prefill

prompt tok/s by context length

8K context6,7001,216 ms TTFT

16K context6,5122,475 ms TTFT

32K context6,0735,283 ms TTFT

64K context5,21612,275 ms TTFT

128K context4,09331,245 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	129.0132 ms TTFT	248.9147 ms TTFT	478.1245 ms TTFT	781.8295 ms TTFT	1,328.5403 ms TTFT	1,892.8578 ms TTFT	2,420.1941 ms TTFT	2,915.21,364 ms TTFT
16K	99.5260 ms TTFT	195.6403 ms TTFT	361.4638 ms TTFT	499.1955 ms TTFT	672.91,589 ms TTFT	816.42,893 ms TTFT	Over capacity	Over capacity
32K	78.3415 ms TTFT	148.7568 ms TTFT	280.6844 ms TTFT	353.81,473 ms TTFT	441.72,352 ms TTFT	Over capacity	Over capacity	Over capacity
64K	52.9540 ms TTFT	101.6825 ms TTFT	191.61,156 ms TTFT	225.71,782 ms TTFT	260.73,100 ms TTFT	Over capacity	Over capacity	Over capacity
128K	33.81,012 ms TTFT	64.51,371 ms TTFT	117.41,937 ms TTFT	135.23,130 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
248K	18.81,911 ms TTFT	36.32,476 ms TTFT	65.03,414 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	6,700.0	—	1,216 ms	—	—	1	—	—
Prefill	16K	1	Valid	6,512.0	—	2,475 ms	—	—	1	—	—
Prefill	32K	1	Valid	6,073.0	—	5,283 ms	—	—	1	—	—
Prefill	64K	1	Valid	5,216.0	—	12,275 ms	—	—	1	—	—
Prefill	128K	1	Valid	4,093.0	—	31,245 ms	—	—	1	—	—
Sustained decode	0	1	Valid	129.0	132.2	132 ms	7.6 ms	1.0	0	3,869	30.0 s
Sustained decode	0	2	Valid	248.9	127.7	147 ms	7.8 ms	2.0	0	7,465	30.0 s
Sustained decode	0	4	Valid	478.1	121.6	245 ms	8.2 ms	4.0	0	14,337	30.0 s
Sustained decode	0	8	Valid	781.8	99.9	295 ms	10.0 ms	8.0	0	23,450	30.0 s
Sustained decode	0	16	Valid	1,328.5	85.6	403 ms	11.7 ms	16.0	0	39,840	30.0 s
Sustained decode	0	32	Valid	1,892.8	60.7	578 ms	16.5 ms	32.0	0	56,773	30.0 s
Sustained decode	0	64	Valid	2,420.1	38.8	941 ms	25.8 ms	64.0	0	72,536	30.0 s
Sustained decode	0	128	Valid	2,915.2	23.5	1,364 ms	42.5 ms	128.0	0	87,502	30.0 s
Sustained decode	16K	1	Valid	99.5	100.8	260 ms	9.9 ms	1.0	0	2,985	30.0 s
Sustained decode	16K	2	Valid	195.6	98.6	403 ms	10.1 ms	2.0	0	5,869	30.0 s
Sustained decode	16K	4	Valid	361.4	91.5	638 ms	10.9 ms	4.0	0	10,834	30.0 s
Sustained decode	16K	8	Valid	499.1	62.9	955 ms	15.9 ms	8.0	0	14,974	30.0 s
Sustained decode	16K	16	Valid	672.9	42.2	1,589 ms	23.7 ms	16.0	0	20,176	30.0 s
Sustained decode	16K	32	Valid	816.4	25.2	2,893 ms	39.7 ms	32.0	0	24,437	29.9 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	78.3	79.0	415 ms	12.7 ms	1.0	0	2,348	30.0 s
Sustained decode	32K	2	Valid	148.7	75.4	568 ms	13.3 ms	2.0	0	4,457	30.0 s
Sustained decode	32K	4	Valid	280.6	71.1	844 ms	14.1 ms	4.0	0	8,418	30.0 s
Sustained decode	32K	8	Valid	353.8	44.6	1,473 ms	22.4 ms	8.0	0	10,614	30.0 s
Sustained decode	32K	16	Valid	441.7	27.6	2,352 ms	36.3 ms	16.0	0	13,252	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	52.9	53.5	540 ms	18.7 ms	1.0	0	1,586	30.0 s
Sustained decode	64K	2	Valid	101.6	52.2	825 ms	19.2 ms	2.0	0	3,047	30.0 s
Sustained decode	64K	4	Valid	191.6	48.5	1,156 ms	20.6 ms	4.0	0	5,744	30.0 s
Sustained decode	64K	8	Valid	225.7	28.6	1,782 ms	35.0 ms	8.0	0	6,760	29.9 s
Sustained decode	64K	16	Valid	260.7	16.3	3,100 ms	61.2 ms	16.0	0	7,790	29.9 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	33.8	34.7	1,012 ms	28.9 ms	1.0	0	1,013	30.0 s
Sustained decode	128K	2	Valid	64.5	33.0	1,371 ms	30.3 ms	2.0	0	1,935	30.0 s
Sustained decode	128K	4	Valid	117.4	29.9	1,937 ms	33.4 ms	4.0	0	3,517	30.0 s
Sustained decode	128K	8	Valid	135.2	17.4	3,130 ms	57.5 ms	8.0	0	4,047	29.9 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	18.8	19.5	1,911 ms	51.2 ms	1.0	0	562	30.0 s
Sustained decode	248K	2	Valid	36.3	18.7	2,476 ms	53.5 ms	2.0	0	1,087	29.9 s
Sustained decode	248K	4	Valid	65.0	16.2	3,414 ms	61.7 ms	4.0	0	1,949	30.0 s
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.