Speed test matrix

Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · NVFP46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Nov 1, 2026, 15:25 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: NVFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: 56c8f43e1ef9423de0edbadd95a413ea80d62374
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Multimodal-NVFP4-MTP --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}

Prefill

prompt tok/s by context length

8K context1,9104,264 ms TTFT

16K context1,8058,927 ms TTFT

32K context1,69518,933 ms TTFT

64K context1,50042,677 ms TTFT

128K context1,218105,007 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	71.6192 ms TTFT	137.0399 ms TTFT	240.5412 ms TTFT	414.7611 ms TTFT	669.2961 ms TTFT	807.41,555 ms TTFT	Over capacity	Over capacity
16K	52.3676 ms TTFT	102.51,139 ms TTFT	148.22,144 ms TTFT	231.73,599 ms TTFT	282.35,830 ms TTFT	Over capacity	Over capacity	Over capacity
32K	44.0815 ms TTFT	80.81,291 ms TTFT	109.22,420 ms TTFT	156.94,028 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
64K	30.21,144 ms TTFT	56.01,667 ms TTFT	68.52,960 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
128K	19.12,387 ms TTFT	37.13,875 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
248K	10.7Requests queued3,635 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	1,910.0	—	4,264 ms	—	—	1	—	—
Prefill	16K	1	Valid	1,805.0	—	8,927 ms	—	—	1	—	—
Prefill	32K	1	Valid	1,695.0	—	18,933 ms	—	—	1	—	—
Prefill	64K	1	Valid	1,500.0	—	42,677 ms	—	—	1	—	—
Prefill	128K	1	Valid	1,218.0	—	105,007 ms	—	—	1	—	—
Sustained decode	0	1	Valid	71.6	72.0	192 ms	13.9 ms	1.0	0	2,146	30.0 s
Sustained decode	0	2	Valid	137.0	69.4	399 ms	14.4 ms	2.0	0	4,108	30.0 s
Sustained decode	0	4	Valid	240.5	61.4	412 ms	16.3 ms	4.0	0	7,209	30.0 s
Sustained decode	0	8	Valid	414.7	53.2	611 ms	18.8 ms	8.0	0	12,425	30.0 s
Sustained decode	0	16	Valid	669.2	43.1	961 ms	23.2 ms	16.0	0	20,047	30.0 s
Sustained decode	0	32	Valid	807.4	25.9	1,555 ms	38.5 ms	32.0	0	24,196	30.0 s
Sustained decode	0	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	0	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	52.3	52.9	676 ms	18.9 ms	1.0	0	1,566	30.0 s
Sustained decode	16K	2	Valid	102.5	51.7	1,139 ms	19.4 ms	2.0	0	3,073	30.0 s
Sustained decode	16K	4	Valid	148.2	37.0	2,144 ms	27.0 ms	4.0	0	4,440	30.0 s
Sustained decode	16K	8	Valid	231.7	28.9	3,599 ms	34.6 ms	8.0	0	6,945	30.0 s
Sustained decode	16K	16	Valid	282.3	16.5	5,830 ms	60.5 ms	16.0	0	8,449	29.9 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	44.0	44.7	815 ms	22.4 ms	1.0	0	1,319	30.0 s
Sustained decode	32K	2	Valid	80.8	41.0	1,291 ms	24.4 ms	2.0	0	2,421	30.0 s
Sustained decode	32K	4	Valid	109.2	28.0	2,420 ms	35.8 ms	4.0	0	3,270	29.9 s
Sustained decode	32K	8	Valid	156.9	19.5	4,028 ms	51.2 ms	8.0	0	4,697	29.9 s
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	30.2	30.7	1,144 ms	32.6 ms	1.0	0	905	30.0 s
Sustained decode	64K	2	Valid	56.0	28.7	1,667 ms	34.9 ms	2.0	0	1,678	30.0 s
Sustained decode	64K	4	Valid	68.5	17.4	2,960 ms	57.5 ms	4.0	0	2,048	29.9 s
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	19.1	19.5	2,387 ms	51.3 ms	1.0	0	573	30.0 s
Sustained decode	128K	2	Valid	37.1	18.8	3,875 ms	53.3 ms	2.0	0	1,113	30.0 s
Sustained decode	128K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Requests queued	10.7	11.9	3,635 ms	84.2 ms	0.9	0	319	29.9 s
Sustained decode	248K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.