Speed test matrix

granite-4.2-8b-fp8 on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · FP85 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 30, 2026, 08:57 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve ibm-granite/granite-4.2-8b-fp8 --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code

Prefill

prompt tok/s by context length

8K context6,0241,361 ms TTFT

16K context5,2003,117 ms TTFT

32K context4,1787,722 ms TTFT

64K context2,96521,708 ms TTFT

120K context1,96561,354 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	106.664 ms TTFT	201.157 ms TTFT	378.465 ms TTFT	681.693 ms TTFT	1,167.5144 ms TTFT	1,636.8218 ms TTFT	Over capacity	Over capacity
16K	85.8168 ms TTFT	143.7236 ms TTFT	225.7273 ms TTFT	316.9410 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
32K	71.6238 ms TTFT	110.2332 ms TTFT	154.5539 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
64K	53.9459 ms TTFT	73.2631 ms TTFT	93.6930 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
120K	37.5841 ms TTFT	45.31,025 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	6,024.0	—	1,361 ms	—	—	1	—	—
Prefill	16K	1	Valid	5,200.0	—	3,117 ms	—	—	1	—	—
Prefill	32K	1	Valid	4,178.0	—	7,722 ms	—	—	1	—	—
Prefill	64K	1	Valid	2,965.0	—	21,708 ms	—	—	1	—	—
Prefill	120K	1	Valid	1,965.0	—	61,354 ms	—	—	1	—	—
Sustained decode	0	1	Valid	106.6	107.3	64 ms	9.3 ms	1.0	0	3,196	30.0 s
Sustained decode	0	2	Valid	201.1	101.6	57 ms	9.8 ms	2.0	0	6,032	30.0 s
Sustained decode	0	4	Valid	378.4	96.3	65 ms	10.4 ms	4.0	0	11,352	30.0 s
Sustained decode	0	8	Valid	681.6	87.8	93 ms	11.4 ms	8.0	0	20,449	30.0 s
Sustained decode	0	16	Valid	1,167.5	76.7	144 ms	13.0 ms	16.0	0	35,021	30.0 s
Sustained decode	0	32	Valid	1,636.8	53.7	218 ms	18.6 ms	32.0	0	49,120	30.0 s
Sustained decode	0	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	0	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	85.8	86.1	168 ms	11.6 ms	1.0	0	2,574	30.0 s
Sustained decode	16K	2	Valid	143.7	72.2	236 ms	13.9 ms	2.0	0	4,312	30.0 s
Sustained decode	16K	4	Valid	225.7	56.7	273 ms	17.6 ms	4.0	0	6,768	30.0 s
Sustained decode	16K	8	Valid	316.9	39.7	410 ms	25.2 ms	8.0	0	9,504	30.0 s
Sustained decode	16K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	71.6	71.8	238 ms	13.9 ms	1.0	0	2,146	30.0 s
Sustained decode	32K	2	Valid	110.2	55.3	332 ms	18.1 ms	2.0	0	3,306	30.0 s
Sustained decode	32K	4	Valid	154.5	38.7	539 ms	25.8 ms	4.0	0	4,632	30.0 s
Sustained decode	32K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	53.9	54.1	459 ms	18.5 ms	1.0	0	1,617	30.0 s
Sustained decode	64K	2	Valid	73.2	36.7	631 ms	27.2 ms	2.0	0	2,196	30.0 s
Sustained decode	64K	4	Valid	93.6	23.4	930 ms	42.6 ms	4.0	0	2,804	30.0 s
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	1	Valid	37.5	37.7	841 ms	26.5 ms	1.0	0	1,125	30.0 s
Sustained decode	120K	2	Valid	45.3	22.8	1,025 ms	43.9 ms	2.0	0	1,358	30.0 s
Sustained decode	120K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.