Speed test matrix

Ornith-1.5-9B-NVFP4 on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · NVFP46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 29, 2026, 02:39 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: NVFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve ornith-ai/Ornith-1.5-9B-NVFP4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only

Prefill

prompt tok/s by context length

8K context5,8261,399 ms TTFT

16K context5,4392,964 ms TTFT

32K context5,1026,289 ms TTFT

64K context4,43714,430 ms TTFT

128K context3,46736,888 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	143.0114 ms TTFT	275.4204 ms TTFT	527.5209 ms TTFT	973.0258 ms TTFT	1,597.8391 ms TTFT	2,433.0593 ms TTFT	3,119.0984 ms TTFT	3,578.11,491 ms TTFT
16K	136.6218 ms TTFT	257.3253 ms TTFT	467.6423 ms TTFT	779.7619 ms TTFT	1,140.81,008 ms TTFT	1,510.41,935 ms TTFT	1,736.83,304 ms TTFT	1,916.97,360 ms TTFT
32K	131.1340 ms TTFT	240.2410 ms TTFT	418.2567 ms TTFT	648.6818 ms TTFT	883.11,387 ms TTFT	1,083.82,494 ms TTFT	1,196.75,025 ms TTFT	Over capacity
64K	121.8646 ms TTFT	213.5749 ms TTFT	346.51,033 ms TTFT	483.31,581 ms TTFT	604.02,528 ms TTFT	689.84,866 ms TTFT	Over capacity	Over capacity
128K	106.6866 ms TTFT	174.71,121 ms TTFT	257.71,597 ms TTFT	319.72,562 ms TTFT	369.44,199 ms TTFT	Over capacity	Over capacity	Over capacity
248K	86.61,964 ms TTFT	129.72,452 ms TTFT	173.63,400 ms TTFT	195.65,611 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	5,826.0	—	1,399 ms	—	—	1	—	—
Prefill	16K	1	Valid	5,439.0	—	2,964 ms	—	—	1	—	—
Prefill	32K	1	Valid	5,102.0	—	6,289 ms	—	—	1	—	—
Prefill	64K	1	Valid	4,437.0	—	14,430 ms	—	—	1	—	—
Prefill	128K	1	Valid	3,467.0	—	36,888 ms	—	—	1	—	—
Sustained decode	0	1	Valid	143.0	143.2	114 ms	7.0 ms	1.0	0	4,288	30.0 s
Sustained decode	0	2	Valid	275.4	138.1	204 ms	7.2 ms	2.0	0	8,261	30.0 s
Sustained decode	0	4	Valid	527.5	132.5	209 ms	7.5 ms	4.0	0	15,824	30.0 s
Sustained decode	0	8	Valid	973.0	122.6	258 ms	8.2 ms	8.0	0	29,216	30.0 s
Sustained decode	0	16	Valid	1,597.8	101.0	391 ms	9.9 ms	16.0	0	47,936	30.0 s
Sustained decode	0	32	Valid	2,433.0	76.8	593 ms	13.0 ms	32.0	0	73,000	30.0 s
Sustained decode	0	64	Valid	3,119.0	49.0	984 ms	20.4 ms	64.0	0	93,576	30.0 s
Sustained decode	0	128	Valid	3,578.1	27.6	1,491 ms	36.2 ms	128.0	0	107,392	30.0 s
Sustained decode	16K	1	Valid	136.6	136.9	218 ms	7.3 ms	1.0	0	4,098	30.0 s
Sustained decode	16K	2	Valid	257.3	128.8	253 ms	7.8 ms	2.0	0	7,718	30.0 s
Sustained decode	16K	4	Valid	467.6	116.9	423 ms	8.6 ms	4.0	0	14,029	30.0 s
Sustained decode	16K	8	Valid	779.7	97.3	619 ms	10.3 ms	8.0	0	23,392	30.0 s
Sustained decode	16K	16	Valid	1,140.8	70.8	1,008 ms	14.1 ms	16.0	0	34,225	30.0 s
Sustained decode	16K	32	Valid	1,510.4	46.3	1,935 ms	21.6 ms	32.0	0	45,312	30.0 s
Sustained decode	16K	64	Valid	1,736.8	26.2	3,304 ms	38.1 ms	64.0	0	52,096	30.0 s
Sustained decode	16K	128	Valid	1,916.9	14.5	7,360 ms	69.1 ms	128.0	0	57,472	30.0 s
Sustained decode	32K	1	Valid	131.1	131.5	340 ms	7.6 ms	1.0	0	3,933	30.0 s
Sustained decode	32K	2	Valid	240.2	120.2	410 ms	8.3 ms	2.0	0	7,205	30.0 s
Sustained decode	32K	4	Valid	418.2	104.5	567 ms	9.6 ms	4.0	0	12,548	30.0 s
Sustained decode	32K	8	Valid	648.6	80.5	818 ms	12.4 ms	8.0	0	19,456	30.0 s
Sustained decode	32K	16	Valid	883.1	54.3	1,387 ms	18.4 ms	16.0	0	26,496	30.0 s
Sustained decode	32K	32	Valid	1,083.8	32.7	2,494 ms	30.6 ms	32.0	0	32,514	30.0 s
Sustained decode	32K	64	Valid	1,196.7	17.6	5,025 ms	56.9 ms	64.0	0	35,906	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	121.8	122.3	646 ms	8.2 ms	1.0	0	3,653	30.0 s
Sustained decode	64K	2	Valid	213.5	106.9	749 ms	9.4 ms	2.0	0	6,406	30.0 s
Sustained decode	64K	4	Valid	346.5	86.1	1,033 ms	11.6 ms	4.0	0	10,396	30.0 s
Sustained decode	64K	8	Valid	483.3	59.3	1,581 ms	16.9 ms	8.0	0	14,496	30.0 s
Sustained decode	64K	16	Valid	604.0	36.2	2,528 ms	27.6 ms	16.0	0	18,112	30.0 s
Sustained decode	64K	32	Valid	689.8	19.9	4,866 ms	50.4 ms	32.0	0	20,704	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	106.6	107.1	866 ms	9.3 ms	1.0	0	3,199	30.0 s
Sustained decode	128K	2	Valid	174.7	87.7	1,121 ms	11.4 ms	2.0	0	5,240	30.0 s
Sustained decode	128K	4	Valid	257.7	64.6	1,597 ms	15.5 ms	4.0	0	7,728	30.0 s
Sustained decode	128K	8	Valid	319.7	40.2	2,562 ms	24.9 ms	8.0	0	9,585	30.0 s
Sustained decode	128K	16	Valid	369.4	23.1	4,199 ms	43.2 ms	16.0	0	11,072	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	86.6	87.2	1,964 ms	11.5 ms	1.0	0	2,597	30.0 s
Sustained decode	248K	2	Valid	129.7	65.0	2,452 ms	15.4 ms	2.0	0	3,890	30.0 s
Sustained decode	248K	4	Valid	173.6	43.2	3,400 ms	23.2 ms	4.0	0	5,204	30.0 s
Sustained decode	248K	8	Valid	195.6	24.3	5,611 ms	41.2 ms	8.0	0	5,864	30.0 s
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.