Speed test matrix

Trinity-Nano-Preview on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 2, 2026, 22:40 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: trinity
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-arcee-vllm/bin/vllm" serve arcee-ai/Trinity-Nano-Preview --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.884631 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context73.6K111 ms TTFT

16K context73.2K222 ms TTFT

32K context77.8K416 ms TTFT

64K context75.1K860 ms TTFT

123K context63.7K1,946 ms TTFT

128K context63K2,049 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	198.157 ms TTFT	383.767 ms TTFT	688.9129 ms TTFT	887.3184 ms TTFT	2,265.8182 ms TTFT	3,870.0186 ms TTFT	5,442.6575 ms TTFT	6,792.3966 ms TTFT
16K	196.383 ms TTFT	370.6106 ms TTFT	697.0124 ms TTFT	823.6224 ms TTFT	2,315.8165 ms TTFT	3,403.9464 ms TTFT	4,463.8768 ms TTFT	5,721.31,525 ms TTFT
32K	207.2126 ms TTFT	363.4172 ms TTFT	623.7205 ms TTFT	773.4308 ms TTFT	2,190.3269 ms TTFT	2,835.0884 ms TTFT	3,241.91,497 ms TTFT	4,225.02,952 ms TTFT
64K	208.6206 ms TTFT	345.9219 ms TTFT	609.1232 ms TTFT	698.4600 ms TTFT	1,642.5800 ms TTFT	1,952.41,878 ms TTFT	2,178.03,507 ms TTFT	Over capacity
123K	177.8352 ms TTFT	331.1410 ms TTFT	598.0421 ms TTFT	646.81,121 ms TTFT	1,103.21,972 ms TTFT	1,420.63,405 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	73.6K	—	111 ms	—	—	1	—	—
Prefill	16K	1	Valid	73.2K	—	222 ms	—	—	1	—	—
Prefill	32K	1	Valid	77.8K	—	416 ms	—	—	1	—	—
Prefill	64K	1	Valid	75.1K	—	860 ms	—	—	1	—	—
Prefill	123K	1	Valid	63.7K	—	1,946 ms	—	—	1	—	—
Prefill	128K	1	Valid	63K	—	2,049 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	198.1	199.4	57 ms	5.0 ms	1.0	1	5,943	30.0 s
Sustained decode	8K	2	Valid	383.7	197.6	67 ms	5.1 ms	2.0	2	11,515	30.0 s
Sustained decode	8K	4	Valid	688.9	179.4	129 ms	5.6 ms	4.0	4	19,643	28.5 s
Sustained decode	8K	8	Valid	887.3	114.2	184 ms	8.8 ms	8.0	0	26,619	30.0 s
Sustained decode	8K	16	Valid	2,265.8	145.1	182 ms	6.9 ms	16.0	16	67,973	30.0 s
Sustained decode	8K	32	Valid	3,870.0	121.0	186 ms	8.3 ms	31.9	32	116,095	30.0 s
Sustained decode	8K	64	Valid	5,442.6	92.4	575 ms	10.8 ms	64.0	0	163,293	30.0 s
Sustained decode	8K	128	Valid	6,792.3	68.3	966 ms	14.6 ms	128.0	0	203,747	30.0 s
Sustained decode	16K	1	Valid	196.3	205.2	83 ms	4.9 ms	1.0	1	5,891	30.0 s
Sustained decode	16K	2	Valid	370.6	193.1	106 ms	5.2 ms	2.0	2	10,506	28.3 s
Sustained decode	16K	4	Valid	697.0	175.7	124 ms	5.7 ms	4.0	4	20,909	30.0 s
Sustained decode	16K	8	Valid	823.6	111.3	224 ms	9.0 ms	8.0	0	24,709	30.0 s
Sustained decode	16K	16	Valid	2,315.8	140.3	165 ms	7.1 ms	15.8	16	69,475	30.0 s
Sustained decode	16K	32	Valid	3,403.9	114.6	464 ms	8.7 ms	32.0	0	102,119	30.0 s
Sustained decode	16K	64	Valid	4,463.8	85.0	768 ms	11.8 ms	64.0	0	133,916	30.0 s
Sustained decode	16K	128	Valid	5,721.3	60.0	1,525 ms	16.7 ms	128.0	0	171,559	30.0 s
Sustained decode	32K	1	Valid	207.2	199.4	126 ms	5.0 ms	1.0	1	3,821	18.4 s
Sustained decode	32K	2	Valid	363.4	189.1	172 ms	5.3 ms	1.9	2	10,646	29.3 s
Sustained decode	32K	4	Valid	623.7	170.4	205 ms	5.9 ms	4.0	4	17,362	27.8 s
Sustained decode	32K	8	Valid	773.4	109.6	308 ms	9.1 ms	8.0	0	23,201	30.0 s
Sustained decode	32K	16	Valid	2,190.3	131.9	269 ms	7.6 ms	15.8	16	65,711	30.0 s
Sustained decode	32K	32	Valid	2,835.0	101.9	884 ms	9.8 ms	32.0	0	85,052	30.0 s
Sustained decode	32K	64	Valid	3,241.9	72.7	1,497 ms	13.7 ms	64.0	0	97,224	30.0 s
Sustained decode	32K	128	Valid	4,225.0	48.0	2,952 ms	20.9 ms	128.0	0	126,730	30.0 s
Sustained decode	64K	1	Valid	208.6	2409.9	206 ms	2.7 ms	1.0	1	4,717	22.6 s
Sustained decode	64K	2	Valid	345.9	181.5	219 ms	5.5 ms	2.0	2	10,374	30.0 s
Sustained decode	64K	4	Valid	609.1	162.8	232 ms	6.1 ms	4.0	4	18,270	30.0 s
Sustained decode	64K	8	Valid	698.4	102.4	600 ms	9.8 ms	8.0	0	20,965	30.0 s
Sustained decode	64K	16	Valid	1,642.5	118.5	800 ms	8.4 ms	16.0	6	49,267	30.0 s
Sustained decode	64K	32	Valid	1,952.4	84.5	1,878 ms	11.8 ms	32.0	0	58,557	30.0 s
Sustained decode	64K	64	Valid	2,178.0	55.6	3,507 ms	18.0 ms	64.0	0	65,317	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	177.8	181.1	352 ms	5.5 ms	1.0	1	4,632	26.0 s
Sustained decode	123K	2	Valid	331.1	173.1	410 ms	5.8 ms	2.0	2	9,455	28.6 s
Sustained decode	123K	4	Valid	598.0	150.8	421 ms	6.6 ms	3.9	4	17,941	30.0 s
Sustained decode	123K	8	Valid	646.8	91.0	1,121 ms	11.0 ms	8.0	0	19,415	30.0 s
Sustained decode	123K	16	Valid	1,103.2	94.2	1,972 ms	10.6 ms	16.0	0	33,092	30.0 s
Sustained decode	123K	32	Valid	1,420.6	62.2	3,405 ms	16.1 ms	32.0	0	42,614	30.0 s
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.