Speed test matrix

Sage-A1.5-FP8 on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 29, 2026, 22:15 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve InternScience/Sage-A1.5-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only

Prefill

prompt tok/s by context length

8K context7,5131,084 ms TTFT

16K context6,5372,466 ms TTFT

32K context6,8244,702 ms TTFT

64K context6,04610,589 ms TTFT

128K context4,42428,907 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	140.7135 ms TTFT	244.9166 ms TTFT	408.5231 ms TTFT	649.2349 ms TTFT	1,014.6430 ms TTFT	1,528.2684 ms TTFT	2,251.9834 ms TTFT	3,054.11,178 ms TTFT
16K	134.6289 ms TTFT	228.7346 ms TTFT	367.9542 ms TTFT	554.5838 ms TTFT	813.61,424 ms TTFT	1,115.62,910 ms TTFT	Over capacity	Over capacity
32K	128.2357 ms TTFT	214.5475 ms TTFT	336.9674 ms TTFT	489.3958 ms TTFT	677.81,506 ms TTFT	Over capacity	Over capacity	Over capacity
64K	119.8621 ms TTFT	192.1671 ms TTFT	286.11,141 ms TTFT	390.41,704 ms TTFT	512.82,607 ms TTFT	Over capacity	Over capacity	Over capacity
128K	105.1899 ms TTFT	157.21,107 ms TTFT	218.91,699 ms TTFT	280.52,826 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
248K	85.22,054 ms TTFT	117.12,541 ms TTFT	152.43,670 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	7,513.0	—	1,084 ms	—	—	1	—	—
Prefill	16K	1	Valid	6,537.0	—	2,466 ms	—	—	1	—	—
Prefill	32K	1	Valid	6,824.0	—	4,702 ms	—	—	1	—	—
Prefill	64K	1	Valid	6,046.0	—	10,589 ms	—	—	1	—	—
Prefill	128K	1	Valid	4,424.0	—	28,907 ms	—	—	1	—	—
Sustained decode	0	1	Valid	140.7	140.9	135 ms	7.1 ms	1.0	0	4,220	30.0 s
Sustained decode	0	2	Valid	244.9	122.8	166 ms	8.1 ms	2.0	0	7,345	30.0 s
Sustained decode	0	4	Valid	408.5	102.7	231 ms	9.7 ms	4.0	0	12,256	30.0 s
Sustained decode	0	8	Valid	649.2	82.2	349 ms	12.2 ms	8.0	0	19,472	30.0 s
Sustained decode	0	16	Valid	1,014.6	64.7	430 ms	15.5 ms	16.0	0	30,448	30.0 s
Sustained decode	0	32	Valid	1,528.2	48.7	684 ms	20.5 ms	32.0	0	45,856	30.0 s
Sustained decode	0	64	Valid	2,251.9	36.0	834 ms	27.8 ms	64.0	0	67,584	30.0 s
Sustained decode	0	128	Valid	3,054.1	24.2	1,178 ms	41.3 ms	128.0	0	91,648	30.0 s
Sustained decode	16K	1	Valid	134.6	134.8	289 ms	7.4 ms	1.0	0	4,039	30.0 s
Sustained decode	16K	2	Valid	228.7	114.4	346 ms	8.7 ms	2.0	0	6,860	30.0 s
Sustained decode	16K	4	Valid	367.9	91.9	542 ms	10.9 ms	4.0	0	11,035	30.0 s
Sustained decode	16K	8	Valid	554.5	69.0	838 ms	14.5 ms	8.0	0	16,632	30.0 s
Sustained decode	16K	16	Valid	813.6	50.1	1,424 ms	19.9 ms	16.0	0	24,432	30.0 s
Sustained decode	16K	32	Valid	1,115.6	33.4	2,910 ms	29.9 ms	32.0	0	33,472	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	128.2	128.6	357 ms	7.8 ms	1.0	0	3,846	30.0 s
Sustained decode	32K	2	Valid	214.5	107.5	475 ms	9.3 ms	2.0	0	6,434	30.0 s
Sustained decode	32K	4	Valid	336.9	84.2	674 ms	11.9 ms	4.0	0	10,104	30.0 s
Sustained decode	32K	8	Valid	489.3	60.8	958 ms	16.4 ms	8.0	0	14,696	30.0 s
Sustained decode	32K	16	Valid	677.8	41.7	1,506 ms	24.0 ms	16.0	0	20,336	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	119.8	120.2	621 ms	8.3 ms	1.0	0	3,593	30.0 s
Sustained decode	64K	2	Valid	192.1	96.1	671 ms	10.4 ms	2.0	0	5,763	30.0 s
Sustained decode	64K	4	Valid	286.1	71.3	1,141 ms	14.0 ms	4.0	0	8,584	30.0 s
Sustained decode	64K	8	Valid	390.4	48.0	1,704 ms	20.8 ms	8.0	0	11,712	30.0 s
Sustained decode	64K	16	Valid	512.8	30.7	2,607 ms	32.6 ms	16.0	0	15,392	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	105.1	105.6	899 ms	9.5 ms	1.0	0	3,154	30.0 s
Sustained decode	128K	2	Valid	157.2	79.0	1,107 ms	12.7 ms	2.0	0	4,718	30.0 s
Sustained decode	128K	4	Valid	218.9	54.9	1,699 ms	18.2 ms	4.0	0	6,564	30.0 s
Sustained decode	128K	8	Valid	280.5	35.0	2,826 ms	28.6 ms	8.0	0	8,416	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	85.2	85.9	2,054 ms	11.6 ms	1.0	0	2,557	30.0 s
Sustained decode	248K	2	Valid	117.1	58.8	2,541 ms	17.0 ms	2.0	0	3,512	30.0 s
Sustained decode	248K	4	Valid	152.4	37.8	3,670 ms	26.5 ms	4.0	0	4,568	30.0 s
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.