Speed test matrix

Qwen3.6-35B-A3B-NVFP4 on CMP 170HX 64GB

vllm · 0.32.0rc1.dev249+g18305948a · NVFP46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 29, 2026, 02:28 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
VRAM: 64 GB
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: NVFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only

Prefill

prompt tok/s by context length

8K context6,7851,200 ms TTFT

16K context7,0432,288 ms TTFT

32K context6,6804,803 ms TTFT

64K context5,79811,042 ms TTFT

128K context4,36929,270 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	155.9141 ms TTFT	275.4199 ms TTFT	480.4248 ms TTFT	775.2345 ms TTFT	1,210.2410 ms TTFT	1,751.2650 ms TTFT	2,430.8913 ms TTFT	3,006.51,115 ms TTFT
16K	149.5265 ms TTFT	256.4410 ms TTFT	435.0556 ms TTFT	660.5909 ms TTFT	955.21,526 ms TTFT	1,252.62,788 ms TTFT	1,550.85,276 ms TTFT	Over capacity
32K	141.4332 ms TTFT	241.9504 ms TTFT	389.6669 ms TTFT	570.51,113 ms TTFT	770.31,751 ms TTFT	964.53,062 ms TTFT	Over capacity	Over capacity
64K	132.1541 ms TTFT	215.3782 ms TTFT	324.51,143 ms TTFT	448.81,484 ms TTFT	563.72,315 ms TTFT	Over capacity	Over capacity	Over capacity
128K	116.0910 ms TTFT	173.81,135 ms TTFT	243.31,599 ms TTFT	308.52,612 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
248K	92.71,882 ms TTFT	127.32,384 ms TTFT	164.33,378 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	6,785.0	—	1,200 ms	—	—	1	—	—
Prefill	16K	1	Valid	7,043.0	—	2,288 ms	—	—	1	—	—
Prefill	32K	1	Valid	6,680.0	—	4,803 ms	—	—	1	—	—
Prefill	64K	1	Valid	5,798.0	—	11,042 ms	—	—	1	—	—
Prefill	128K	1	Valid	4,369.0	—	29,270 ms	—	—	1	—	—
Sustained decode	0	1	Valid	155.9	156.2	141 ms	6.4 ms	1.0	0	4,677	30.0 s
Sustained decode	0	2	Valid	275.4	138.1	199 ms	7.2 ms	2.0	0	8,262	30.0 s
Sustained decode	0	4	Valid	480.4	120.8	248 ms	8.3 ms	4.0	0	14,408	30.0 s
Sustained decode	0	8	Valid	775.2	97.8	345 ms	10.2 ms	8.0	0	23,256	30.0 s
Sustained decode	0	16	Valid	1,210.2	76.7	410 ms	13.0 ms	16.0	0	36,304	30.0 s
Sustained decode	0	32	Valid	1,751.2	55.8	650 ms	17.9 ms	32.0	0	52,544	30.0 s
Sustained decode	0	64	Valid	2,430.8	38.7	913 ms	25.8 ms	64.0	0	72,960	30.0 s
Sustained decode	0	128	Valid	3,006.5	23.7	1,115 ms	42.2 ms	128.0	0	90,240	30.0 s
Sustained decode	16K	1	Valid	149.5	149.7	265 ms	6.7 ms	1.0	0	4,486	30.0 s
Sustained decode	16K	2	Valid	256.4	128.1	410 ms	7.8 ms	2.0	0	7,690	30.0 s
Sustained decode	16K	4	Valid	435.0	108.5	556 ms	9.2 ms	4.0	0	13,048	30.0 s
Sustained decode	16K	8	Valid	660.5	82.0	909 ms	12.2 ms	8.0	0	19,816	30.0 s
Sustained decode	16K	16	Valid	955.2	58.6	1,526 ms	17.1 ms	16.0	0	28,656	30.0 s
Sustained decode	16K	32	Valid	1,252.6	37.7	2,788 ms	26.5 ms	32.0	0	37,568	30.0 s
Sustained decode	16K	64	Valid	1,550.8	22.2	5,276 ms	44.9 ms	64.0	0	46,528	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	141.4	141.6	332 ms	7.1 ms	1.0	0	4,242	30.0 s
Sustained decode	32K	2	Valid	241.9	121.1	504 ms	8.3 ms	2.0	0	7,257	30.0 s
Sustained decode	32K	4	Valid	389.6	97.2	669 ms	10.3 ms	4.0	0	11,688	30.0 s
Sustained decode	32K	8	Valid	570.5	70.5	1,113 ms	14.2 ms	8.0	0	17,112	30.0 s
Sustained decode	32K	16	Valid	770.3	47.1	1,751 ms	21.2 ms	16.0	0	23,104	30.0 s
Sustained decode	32K	32	Valid	964.5	28.5	3,062 ms	35.1 ms	32.0	0	28,928	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	132.1	132.5	541 ms	7.5 ms	1.0	0	3,963	30.0 s
Sustained decode	64K	2	Valid	215.3	107.8	782 ms	9.3 ms	2.0	0	6,458	30.0 s
Sustained decode	64K	4	Valid	324.5	80.7	1,143 ms	12.4 ms	4.0	0	9,732	30.0 s
Sustained decode	64K	8	Valid	448.8	55.1	1,484 ms	18.2 ms	8.0	0	13,464	30.0 s
Sustained decode	64K	16	Valid	563.7	33.6	2,315 ms	29.8 ms	16.0	0	16,912	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	116.0	116.5	910 ms	8.6 ms	1.0	0	3,480	30.0 s
Sustained decode	128K	2	Valid	173.8	87.2	1,135 ms	11.5 ms	2.0	0	5,214	30.0 s
Sustained decode	128K	4	Valid	243.3	60.9	1,599 ms	16.4 ms	4.0	0	7,300	30.0 s
Sustained decode	128K	8	Valid	308.5	38.3	2,612 ms	26.1 ms	8.0	0	9,248	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	92.7	93.5	1,882 ms	10.7 ms	1.0	0	2,780	30.0 s
Sustained decode	248K	2	Valid	127.3	63.8	2,384 ms	15.7 ms	2.0	0	3,818	30.0 s
Sustained decode	248K	4	Valid	164.3	40.6	3,378 ms	24.6 ms	4.0	0	4,928	30.0 s
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.