Speed test matrix

Trinity-Mini-W4A16 on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 2, 2026, 19:32 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: trinity
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-arcee-vllm/bin/vllm" serve arcee-ai/Trinity-Mini-W4A16 --served-model-name trinity --tensor-parallel-size 1 --reasoning-parser deepseek_r1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16

Prefill

prompt tok/s by context length

8K context35.4K231 ms TTFT

16K context33.6K484 ms TTFT

32K context33.7K960 ms TTFT

64K context31K2,080 ms TTFT

123K context25.4K4,877 ms TTFT

128K context24.9K5,186 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	238.853 ms TTFT	447.076 ms TTFT	805.389 ms TTFT	1,379.4122 ms TTFT	2,238.7140 ms TTFT	3,496.9321 ms TTFT	5,100.1430 ms TTFT	7,297.7905 ms TTFT
16K	243.9141 ms TTFT	441.492 ms TTFT	794.8148 ms TTFT	1,328.7151 ms TTFT	2,135.2167 ms TTFT	3,300.7508 ms TTFT	4,674.0795 ms TTFT	6,397.71,526 ms TTFT
32K	232.8123 ms TTFT	431.4152 ms TTFT	768.9180 ms TTFT	1,234.4203 ms TTFT	1,960.9198 ms TTFT	2,897.7807 ms TTFT	3,980.01,595 ms TTFT	5,087.62,824 ms TTFT
64K	232.7196 ms TTFT	412.2244 ms TTFT	727.9269 ms TTFT	1,140.6260 ms TTFT	1,730.61,005 ms TTFT	2,397.01,753 ms TTFT	3,050.43,526 ms TTFT	Over capacity
123K	218.8374 ms TTFT	379.1388 ms TTFT	652.2387 ms TTFT	960.8555 ms TTFT	1,404.72,084 ms TTFT	1,795.23,297 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	35.4K	—	231 ms	—	—	1	—	—
Prefill	16K	1	Valid	33.6K	—	484 ms	—	—	1	—	—
Prefill	32K	1	Valid	33.7K	—	960 ms	—	—	1	—	—
Prefill	64K	1	Valid	31K	—	2,080 ms	—	—	1	—	—
Prefill	123K	1	Valid	25.4K	—	4,877 ms	—	—	1	—	—
Prefill	128K	1	Valid	24.9K	—	5,186 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	238.8	239.3	53 ms	4.2 ms	1.0	1	7,167	30.0 s
Sustained decode	8K	2	Valid	447.0	224.3	76 ms	4.5 ms	2.0	2	13,409	30.0 s
Sustained decode	8K	4	Valid	805.3	202.5	89 ms	4.9 ms	4.0	4	24,157	30.0 s
Sustained decode	8K	8	Valid	1,379.4	173.8	122 ms	5.8 ms	7.8	8	41,381	30.0 s
Sustained decode	8K	16	Valid	2,238.7	142.9	140 ms	7.0 ms	16.0	16	67,163	30.0 s
Sustained decode	8K	32	Valid	3,496.9	110.3	321 ms	9.1 ms	32.0	0	104,896	30.0 s
Sustained decode	8K	64	Valid	5,100.1	80.4	430 ms	12.4 ms	64.0	0	153,024	30.0 s
Sustained decode	8K	128	Valid	7,297.7	57.8	905 ms	17.3 ms	128.0	0	218,881	30.0 s
Sustained decode	16K	1	Valid	243.9	245.7	141 ms	4.1 ms	1.0	1	7,317	30.0 s
Sustained decode	16K	2	Valid	441.4	221.9	92 ms	4.5 ms	2.0	2	13,242	30.0 s
Sustained decode	16K	4	Valid	794.8	200.0	148 ms	5.0 ms	3.9	4	23,845	30.0 s
Sustained decode	16K	8	Valid	1,328.7	166.8	151 ms	6.0 ms	8.0	8	39,859	30.0 s
Sustained decode	16K	16	Valid	2,135.2	135.9	167 ms	7.4 ms	16.0	16	64,059	30.0 s
Sustained decode	16K	32	Valid	3,300.7	103.9	508 ms	9.6 ms	32.0	0	99,008	30.0 s
Sustained decode	16K	64	Valid	4,674.0	73.4	795 ms	13.6 ms	64.0	0	140,220	30.0 s
Sustained decode	16K	128	Valid	6,397.7	50.5	1,526 ms	19.8 ms	128.0	0	191,866	30.0 s
Sustained decode	32K	1	Valid	232.8	234.0	123 ms	4.3 ms	1.0	1	6,985	30.0 s
Sustained decode	32K	2	Valid	431.4	217.2	152 ms	4.6 ms	2.0	2	12,941	30.0 s
Sustained decode	32K	4	Valid	768.9	193.9	180 ms	5.2 ms	4.0	4	23,066	30.0 s
Sustained decode	32K	8	Valid	1,234.4	156.1	203 ms	6.4 ms	8.0	8	37,030	30.0 s
Sustained decode	32K	16	Valid	1,960.9	123.6	198 ms	8.1 ms	16.0	16	58,830	30.0 s
Sustained decode	32K	32	Valid	2,897.7	91.0	807 ms	11.0 ms	32.0	0	86,912	30.0 s
Sustained decode	32K	64	Valid	3,980.0	62.5	1,595 ms	16.0 ms	64.0	0	119,424	30.0 s
Sustained decode	32K	128	Valid	5,087.6	40.1	2,824 ms	25.0 ms	128.0	0	152,576	30.0 s
Sustained decode	64K	1	Valid	232.7	234.8	196 ms	4.3 ms	1.0	1	6,982	30.0 s
Sustained decode	64K	2	Valid	412.2	208.1	244 ms	4.8 ms	2.0	2	12,373	30.0 s
Sustained decode	64K	4	Valid	727.9	183.7	269 ms	5.4 ms	3.9	4	21,837	30.0 s
Sustained decode	64K	8	Valid	1,140.6	143.7	260 ms	7.0 ms	7.9	8	34,214	30.0 s
Sustained decode	64K	16	Valid	1,730.6	108.2	1,005 ms	9.2 ms	16.0	0	51,920	30.0 s
Sustained decode	64K	32	Valid	2,397.0	75.0	1,753 ms	13.3 ms	32.0	0	71,904	30.0 s
Sustained decode	64K	64	Valid	3,050.4	47.8	3,526 ms	20.9 ms	64.0	0	91,520	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	218.8	222.2	374 ms	4.5 ms	1.0	1	6,564	30.0 s
Sustained decode	123K	2	Valid	379.1	192.6	388 ms	5.2 ms	1.9	2	11,372	30.0 s
Sustained decode	123K	4	Valid	652.2	165.5	387 ms	6.0 ms	4.0	4	19,565	30.0 s
Sustained decode	123K	8	Valid	960.8	121.7	555 ms	8.2 ms	7.9	8	28,802	30.0 s
Sustained decode	123K	16	Valid	1,404.7	87.4	2,084 ms	11.4 ms	16.0	0	42,161	30.0 s
Sustained decode	123K	32	Valid	1,795.2	56.0	3,297 ms	17.9 ms	32.0	0	53,856	30.0 s
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.