Speed test matrix

Apex-35B-V1.0 on CMP 170HX 64GB ×2

vllm · 0.32.0rc1.dev249+g18305948a · BF166 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Nov 4, 2026, 02:40 UTC

Hardware

Type: Discrete GPU
GPU: CMP 170HX
GPU count: 2
VRAM: 64 GB (2 × 32 GB)
CPU: 2x Intel Xeon E5-2630 v3
RAM: 64 GB
OS: Ubuntu 26.04.1

Engine

Engine: vllm
Version: 0.32.0rc1.dev249+g18305948a
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79

Run

Served model: bench
Revision: 5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none. Layout: PP2 (stock vLLM flag)
# 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve Apexintelligence-AI/Apex-35B-V1.0 --revision 5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Prefill

prompt tok/s by context length

8K context11.7K699 ms TTFT

16K context11.2K1,434 ms TTFT

32K context11K2,913 ms TTFT

64K context9,9936,406 ms TTFT

128K context7,67716,658 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	126.4134 ms TTFT	215.3159 ms TTFT	354.4173 ms TTFT	493.1303 ms TTFT	774.0366 ms TTFT	1,133.2467 ms TTFT	1,930.6565 ms TTFT	2,901.5850 ms TTFT
16K	121.7292 ms TTFT	208.0347 ms TTFT	335.1472 ms TTFT	499.8728 ms TTFT	674.11,235 ms TTFT	1,083.52,202 ms TTFT	1,528.14,226 ms TTFT	Over capacity
32K	116.7389 ms TTFT	199.9493 ms TTFT	321.2640 ms TTFT	Looped	622.91,450 ms TTFT	956.52,619 ms TTFT	Over capacity	Over capacity
64K	110.3644 ms TTFT	189.3714 ms TTFT	295.5984 ms TTFT	420.21,507 ms TTFT	538.22,514 ms TTFT	785.44,295 ms TTFT	Over capacity	Over capacity
128K	99.7969 ms TTFT	170.91,184 ms TTFT	Looped	345.82,565 ms TTFT	332.74,530 ms TTFT	Over capacity	Over capacity	Over capacity
248K	84.91,909 ms TTFT	144.92,485 ms TTFT	155.43,523 ms TTFT	187.25,379 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	11.7K	—	699 ms	—	—	1	—	—
Prefill	16K	1	Valid	11.2K	—	1,434 ms	—	—	1	—	—
Prefill	32K	1	Valid	11K	—	2,913 ms	—	—	1	—	—
Prefill	64K	1	Valid	9,993.0	—	6,406 ms	—	—	1	—	—
Prefill	128K	1	Valid	7,677.0	—	16,658 ms	—	—	1	—	—
Sustained decode	0	1	Valid	126.4	126.5	134 ms	7.9 ms	1.0	0	3,791	30.0 s
Sustained decode	0	2	Valid	215.3	107.5	159 ms	9.3 ms	2.0	0	6,457	30.0 s
Sustained decode	0	4	Valid	354.4	88.7	173 ms	11.3 ms	4.0	0	10,631	30.0 s
Sustained decode	0	8	Valid	493.1	61.8	303 ms	16.2 ms	8.0	0	14,792	30.0 s
Sustained decode	0	16	Valid	774.0	49.5	366 ms	20.2 ms	16.0	0	23,216	30.0 s
Sustained decode	0	32	Valid	1,133.2	36.3	467 ms	27.5 ms	32.0	0	34,016	30.0 s
Sustained decode	0	64	Valid	1,930.6	30.9	565 ms	32.3 ms	64.0	0	57,920	30.0 s
Sustained decode	0	128	Valid	2,901.5	23.4	850 ms	42.8 ms	128.0	0	87,040	30.0 s
Sustained decode	16K	1	Valid	121.7	121.9	292 ms	8.2 ms	1.0	0	3,651	30.0 s
Sustained decode	16K	2	Valid	208.0	104.0	347 ms	9.6 ms	2.0	0	6,241	30.0 s
Sustained decode	16K	4	Valid	335.1	83.5	472 ms	12.0 ms	4.0	0	10,052	30.0 s
Sustained decode	16K	8	Valid	499.8	62.0	728 ms	16.1 ms	8.0	0	14,992	30.0 s
Sustained decode	16K	16	Valid	674.1	41.4	1,235 ms	24.2 ms	16.0	0	20,224	30.0 s
Sustained decode	16K	32	Valid	1,083.5	32.8	2,202 ms	30.5 ms	32.0	0	32,512	30.0 s
Sustained decode	16K	64	Valid	1,528.1	22.5	4,226 ms	44.5 ms	64.0	0	45,856	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	116.7	116.9	389 ms	8.6 ms	1.0	0	3,500	30.0 s
Sustained decode	32K	2	Valid	199.9	100.0	493 ms	10.0 ms	2.0	0	5,996	30.0 s
Sustained decode	32K	4	Valid	321.2	80.0	640 ms	12.5 ms	4.0	0	9,636	30.0 s
Sustained decode	32K	8	Looped	—	—	—	—	—	0	0	—
Sustained decode	32K	16	Valid	622.9	38.2	1,450 ms	26.2 ms	16.0	0	18,680	30.0 s
Sustained decode	32K	32	Valid	956.5	28.8	2,619 ms	34.7 ms	32.0	0	28,688	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	110.3	110.7	644 ms	9.0 ms	1.0	0	3,308	30.0 s
Sustained decode	64K	2	Valid	189.3	94.7	714 ms	10.6 ms	2.0	0	5,679	30.0 s
Sustained decode	64K	4	Valid	295.5	73.4	984 ms	13.6 ms	4.0	0	8,866	30.0 s
Sustained decode	64K	8	Valid	420.2	51.5	1,507 ms	19.4 ms	8.0	0	12,604	30.0 s
Sustained decode	64K	16	Valid	538.2	32.6	2,514 ms	30.7 ms	16.0	0	16,147	30.0 s
Sustained decode	64K	32	Valid	785.4	22.7	4,295 ms	44.0 ms	32.0	0	23,552	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	99.7	100.1	969 ms	10.0 ms	1.0	0	2,991	30.0 s
Sustained decode	128K	2	Valid	170.9	85.7	1,184 ms	11.7 ms	2.0	0	5,127	30.0 s
Sustained decode	128K	4	Looped	—	—	—	—	—	0	0	—
Sustained decode	128K	8	Valid	345.8	43.0	2,565 ms	23.2 ms	8.0	0	10,372	30.0 s
Sustained decode	128K	16	Valid	332.7	20.7	4,530 ms	48.2 ms	16.0	0	9,979	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	84.9	85.5	1,909 ms	11.7 ms	1.0	0	2,546	30.0 s
Sustained decode	248K	2	Valid	144.9	72.6	2,485 ms	13.8 ms	2.0	0	4,347	30.0 s
Sustained decode	248K	4	Valid	155.4	38.8	3,523 ms	25.8 ms	4.0	0	4,660	30.0 s
Sustained decode	248K	8	Valid	187.2	23.2	5,379 ms	43.2 ms	8.0	0	5,616	30.0 s
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.