Speed test matrix

Muse-Glimmer-30B-NVFP4-W4A4 on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · NVFP4 W4A45 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K

by @Sandbergllm-inference-bench v0.8.0Measured Nov 1, 2026, 14:01 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: NVFP4 W4A4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve Inferact/Muse-Glimmer-30B-NVFP4-W4A4 --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code

Prefill

prompt tok/s by context length

8K context1,9544,179 ms TTFT

16K context1,8918,524 ms TTFT

32K context1,82617,545 ms TTFT

64K context1,72736,999 ms TTFT

120K context1,57376,036 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	45.489 ms TTFT	89.095 ms TTFT	171.8124 ms TTFT	315.1158 ms TTFT	516.8263 ms TTFT	804.4410 ms TTFT	1,099.8543 ms TTFT	1,320.9820 ms TTFT
16K	42.9153 ms TTFT	83.5245 ms TTFT	155.1312 ms TTFT	262.1443 ms TTFT	385.9803 ms TTFT	510.81,176 ms TTFT	589.62,187 ms TTFT	Over capacity
32K	42.5236 ms TTFT	82.5352 ms TTFT	151.6531 ms TTFT	252.9855 ms TTFT	366.31,390 ms TTFT	478.02,405 ms TTFT	Over capacity	Over capacity
64K	41.8456 ms TTFT	80.3584 ms TTFT	145.0894 ms TTFT	235.11,357 ms TTFT	331.92,229 ms TTFT	Over capacity	Over capacity	Over capacity
120K	40.6857 ms TTFT	76.61,027 ms TTFT	134.01,499 ms TTFT	208.62,290 ms TTFT	284.44,244 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	1,954.0	—	4,179 ms	—	—	1	—	—
Prefill	16K	1	Valid	1,891.0	—	8,524 ms	—	—	1	—	—
Prefill	32K	1	Valid	1,826.0	—	17,545 ms	—	—	1	—	—
Prefill	64K	1	Valid	1,727.0	—	36,999 ms	—	—	1	—	—
Prefill	120K	1	Valid	1,573.0	—	76,036 ms	—	—	1	—	—
Sustained decode	0	1	Valid	45.4	45.7	89 ms	21.9 ms	1.0	0	1,362	30.0 s
Sustained decode	0	2	Valid	89.0	44.8	95 ms	22.3 ms	2.0	0	2,670	30.0 s
Sustained decode	0	4	Valid	171.8	43.4	124 ms	23.0 ms	4.0	0	5,152	30.0 s
Sustained decode	0	8	Valid	315.1	40.1	158 ms	24.9 ms	8.0	0	9,448	30.0 s
Sustained decode	0	16	Valid	516.8	33.1	263 ms	30.2 ms	16.0	0	15,504	30.0 s
Sustained decode	0	32	Valid	804.4	25.9	410 ms	38.6 ms	32.0	0	24,128	30.0 s
Sustained decode	0	64	Valid	1,099.8	17.6	543 ms	56.7 ms	64.0	0	32,960	30.0 s
Sustained decode	0	128	Valid	1,320.9	10.4	820 ms	96.1 ms	128.0	0	39,552	29.9 s
Sustained decode	16K	1	Valid	42.9	70.4	153 ms	14.2 ms	1.0	0	1,287	30.0 s
Sustained decode	16K	2	Valid	83.5	47.9	245 ms	21.2 ms	2.0	0	2,504	30.0 s
Sustained decode	16K	4	Valid	155.1	57.4	312 ms	17.5 ms	4.0	0	4,652	30.0 s
Sustained decode	16K	8	Valid	262.1	32.8	443 ms	30.5 ms	8.0	0	7,856	30.0 s
Sustained decode	16K	16	Valid	385.9	24.1	803 ms	41.5 ms	16.0	0	11,568	30.0 s
Sustained decode	16K	32	Valid	510.8	15.9	1,176 ms	62.9 ms	32.0	0	15,328	30.0 s
Sustained decode	16K	64	Valid	589.6	9.1	2,187 ms	109.8 ms	64.0	0	17,664	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	42.5	42.6	236 ms	23.5 ms	1.0	0	1,274	30.0 s
Sustained decode	32K	2	Valid	82.5	48.9	352 ms	21.0 ms	2.0	0	2,474	30.0 s
Sustained decode	32K	4	Valid	151.6	44.5	531 ms	23.0 ms	4.0	0	4,548	30.0 s
Sustained decode	32K	8	Valid	252.9	35.0	855 ms	28.9 ms	8.0	0	7,584	30.0 s
Sustained decode	32K	16	Valid	366.3	22.8	1,390 ms	43.9 ms	16.0	0	10,976	30.0 s
Sustained decode	32K	32	Valid	478.0	14.7	2,405 ms	68.2 ms	32.0	0	14,336	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	41.8	74.5	456 ms	13.4 ms	1.0	0	1,253	30.0 s
Sustained decode	64K	2	Valid	80.3	51.0	584 ms	20.5 ms	2.0	0	2,408	30.0 s
Sustained decode	64K	4	Valid	145.0	50.1	894 ms	20.0 ms	4.0	0	4,348	30.0 s
Sustained decode	64K	8	Valid	235.1	34.8	1,357 ms	28.7 ms	8.0	0	7,048	30.0 s
Sustained decode	64K	16	Valid	331.9	20.5	2,229 ms	48.8 ms	16.0	0	9,959	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	1	Valid	40.6	71.1	857 ms	14.1 ms	1.0	0	1,217	30.0 s
Sustained decode	120K	2	Valid	76.6	81.8	1,027 ms	12.3 ms	2.0	0	2,298	30.0 s
Sustained decode	120K	4	Valid	134.0	51.8	1,499 ms	19.3 ms	4.0	0	4,020	30.0 s
Sustained decode	120K	8	Valid	208.6	30.0	2,290 ms	33.3 ms	8.0	0	6,256	30.0 s
Sustained decode	120K	16	Valid	284.4	17.4	4,244 ms	57.3 ms	16.0	0	8,528	30.0 s
Sustained decode	120K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.