Speed test matrix

Trinity-Mini on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 2, 2026, 19:59 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: trinity
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-arcee-vllm/bin/vllm" serve arcee-ai/Trinity-Mini --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --reasoning-parser deepseek_r1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16

Prefill

prompt tok/s by context length

8K context41.4K198 ms TTFT

16K context39.9K407 ms TTFT

32K context40.5K799 ms TTFT

64K context34.5K1,875 ms TTFT

123K context28.3K4,381 ms TTFT

128K context28K4,604 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	223.056 ms TTFT	386.066 ms TTFT	599.177 ms TTFT	1,020.492 ms TTFT	1,549.9218 ms TTFT	2,511.9337 ms TTFT	3,947.9525 ms TTFT	5,784.5917 ms TTFT
16K	221.589 ms TTFT	381.5111 ms TTFT	599.9167 ms TTFT	1,019.3176 ms TTFT	1,526.5305 ms TTFT	2,375.4542 ms TTFT	3,689.2885 ms TTFT	5,205.21,460 ms TTFT
32K	218.0129 ms TTFT	373.1153 ms TTFT	584.7152 ms TTFT	955.6268 ms TTFT	1,435.5588 ms TTFT	2,205.5782 ms TTFT	3,236.11,677 ms TTFT	Over capacity
64K	211.3187 ms TTFT	357.8243 ms TTFT	556.4250 ms TTFT	907.5608 ms TTFT	1,307.71,034 ms TTFT	1,904.22,018 ms TTFT	2,600.33,524 ms TTFT	Over capacity
123K	200.4336 ms TTFT	331.5429 ms TTFT	510.2413 ms TTFT	805.21,063 ms TTFT	1,110.61,937 ms TTFT	1,498.93,375 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	41.4K	—	198 ms	—	—	1	—	—
Prefill	16K	1	Valid	39.9K	—	407 ms	—	—	1	—	—
Prefill	32K	1	Valid	40.5K	—	799 ms	—	—	1	—	—
Prefill	64K	1	Valid	34.5K	—	1,875 ms	—	—	1	—	—
Prefill	123K	1	Valid	28.3K	—	4,381 ms	—	—	1	—	—
Prefill	128K	1	Valid	28K	—	4,604 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	223.0	223.6	56 ms	4.5 ms	1.0	1	6,691	30.0 s
Sustained decode	8K	2	Valid	386.0	193.7	66 ms	5.2 ms	2.0	2	11,588	30.0 s
Sustained decode	8K	4	Valid	599.1	151.3	77 ms	6.6 ms	4.0	4	17,973	30.0 s
Sustained decode	8K	8	Valid	1,020.4	129.9	92 ms	7.7 ms	7.9	8	30,608	30.0 s
Sustained decode	8K	16	Valid	1,549.9	97.5	218 ms	10.3 ms	16.0	0	46,496	30.0 s
Sustained decode	8K	32	Valid	2,511.9	79.1	337 ms	12.6 ms	32.0	0	75,360	30.0 s
Sustained decode	8K	64	Valid	3,947.9	62.3	525 ms	16.0 ms	64.0	0	118,400	30.0 s
Sustained decode	8K	128	Valid	5,784.5	45.8	917 ms	21.8 ms	128.0	0	173,568	30.0 s
Sustained decode	16K	1	Valid	221.5	222.5	89 ms	4.5 ms	1.0	1	6,645	30.0 s
Sustained decode	16K	2	Valid	381.5	191.8	111 ms	5.2 ms	2.0	2	11,444	30.0 s
Sustained decode	16K	4	Valid	599.9	150.8	167 ms	6.6 ms	4.0	4	17,998	30.0 s
Sustained decode	16K	8	Valid	1,019.3	131.5	176 ms	7.6 ms	8.0	8	30,574	30.0 s
Sustained decode	16K	16	Valid	1,526.5	96.2	305 ms	10.4 ms	16.0	0	45,792	30.0 s
Sustained decode	16K	32	Valid	2,375.4	75.2	542 ms	13.3 ms	32.0	0	71,264	30.0 s
Sustained decode	16K	64	Valid	3,689.2	58.1	885 ms	17.2 ms	64.0	0	110,656	30.0 s
Sustained decode	16K	128	Valid	5,205.2	41.1	1,460 ms	24.3 ms	128.0	0	156,162	30.0 s
Sustained decode	32K	1	Valid	218.0	219.2	129 ms	4.6 ms	1.0	1	6,538	30.0 s
Sustained decode	32K	2	Valid	373.1	187.7	153 ms	5.3 ms	2.0	2	11,192	30.0 s
Sustained decode	32K	4	Valid	584.7	147.2	152 ms	6.8 ms	4.0	4	17,541	30.0 s
Sustained decode	32K	8	Valid	955.6	120.2	268 ms	8.3 ms	8.0	8	28,682	30.0 s
Sustained decode	32K	16	Valid	1,435.5	90.0	588 ms	11.1 ms	16.0	0	43,056	30.0 s
Sustained decode	32K	32	Valid	2,205.5	69.2	782 ms	14.5 ms	32.0	0	66,144	30.0 s
Sustained decode	32K	64	Valid	3,236.1	50.9	1,677 ms	19.7 ms	64.0	0	97,088	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	211.3	213.1	187 ms	4.7 ms	1.0	1	6,341	30.0 s
Sustained decode	64K	2	Valid	357.8	180.8	243 ms	5.5 ms	1.9	2	10,732	30.0 s
Sustained decode	64K	4	Valid	556.4	140.4	250 ms	7.1 ms	4.0	4	16,693	30.0 s
Sustained decode	64K	8	Valid	907.5	113.4	608 ms	8.8 ms	8.0	0	27,224	30.0 s
Sustained decode	64K	16	Valid	1,307.7	81.7	1,034 ms	12.2 ms	16.0	0	39,232	30.0 s
Sustained decode	64K	32	Valid	1,904.2	59.7	2,018 ms	16.8 ms	32.0	0	57,120	30.0 s
Sustained decode	64K	64	Valid	2,600.3	40.8	3,524 ms	24.5 ms	64.0	0	78,017	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	200.4	203.4	336 ms	4.9 ms	1.0	1	6,011	30.0 s
Sustained decode	123K	2	Valid	331.5	168.7	429 ms	5.9 ms	2.0	2	9,945	30.0 s
Sustained decode	123K	4	Valid	510.2	129.3	413 ms	7.7 ms	4.0	4	15,304	30.0 s
Sustained decode	123K	8	Valid	805.2	100.5	1,063 ms	10.0 ms	8.0	0	24,152	30.0 s
Sustained decode	123K	16	Valid	1,110.6	69.2	1,937 ms	14.5 ms	16.0	0	33,328	30.0 s
Sustained decode	123K	32	Valid	1,498.9	46.8	3,375 ms	21.4 ms	32.0	0	44,960	30.0 s
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.