Speed test matrix

Trinity-Nano-Preview-FP8-Block on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 2, 2026, 21:04 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: trinity
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_DEEP_GEMM=1 "$HOME/env_h200-arcee-vllm/bin/vllm" serve arcee-ai/Trinity-Nano-Preview-FP8-Block --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.884645 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context62.7K131 ms TTFT

16K context67.3K241 ms TTFT

32K context76.8K421 ms TTFT

64K context75.3K858 ms TTFT

123K context63.7K1,947 ms TTFT

128K context17.3K7,455 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	175.061 ms TTFT	313.989 ms TTFT	608.182 ms TTFT	762.2166 ms TTFT	1,708.5262 ms TTFT	2,776.6318 ms TTFT	4,467.4551 ms TTFT	6,528.7931 ms TTFT
16K	173.4100 ms TTFT	304.3121 ms TTFT	594.6144 ms TTFT	670.5239 ms TTFT	1,336.3367 ms TTFT	2,571.8483 ms TTFT	4,065.7822 ms TTFT	5,335.91,488 ms TTFT
32K	181.6131 ms TTFT	311.0142 ms TTFT	560.1168 ms TTFT	596.5323 ms TTFT	1,499.3498 ms TTFT	2,242.1862 ms TTFT	3,203.81,507 ms TTFT	3,819.52,762 ms TTFT
64K	165.4195 ms TTFT	304.8242 ms TTFT	549.6230 ms TTFT	481.6614 ms TTFT	973.41,069 ms TTFT	1,709.01,806 ms TTFT	1,978.63,247 ms TTFT	Over capacity
123K	157.8344 ms TTFT	305.7401 ms TTFT	530.8399 ms TTFT	459.31,062 ms TTFT	781.81,877 ms TTFT	983.23,248 ms TTFT	1,522.26,274 ms TTFT	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	62.7K	—	131 ms	—	—	1	—	—
Prefill	16K	1	Valid	67.3K	—	241 ms	—	—	1	—	—
Prefill	32K	1	Valid	76.8K	—	421 ms	—	—	1	—	—
Prefill	64K	1	Valid	75.3K	—	858 ms	—	—	1	—	—
Prefill	123K	1	Valid	63.7K	—	1,947 ms	—	—	1	—	—
Prefill	128K	1	Valid	17.3K	—	7,455 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	175.0	199.8	61 ms	5.1 ms	1.0	1	4,340	24.8 s
Sustained decode	8K	2	Valid	313.9	166.4	89 ms	6.0 ms	1.9	2	9,416	30.0 s
Sustained decode	8K	4	Valid	608.1	153.4	82 ms	6.5 ms	4.0	4	18,239	30.0 s
Sustained decode	8K	8	Valid	762.2	98.8	166 ms	10.1 ms	8.0	0	22,864	30.0 s
Sustained decode	8K	16	Valid	1,708.5	113.8	262 ms	8.8 ms	16.0	0	51,256	30.0 s
Sustained decode	8K	32	Valid	2,776.6	100.2	318 ms	10.0 ms	32.0	0	83,281	30.0 s
Sustained decode	8K	64	Valid	4,467.4	82.6	551 ms	12.1 ms	64.0	0	134,021	30.0 s
Sustained decode	8K	128	Valid	6,528.7	62.4	931 ms	16.0 ms	128.0	0	195,893	30.0 s
Sustained decode	16K	1	Valid	173.4	263.9	100 ms	4.3 ms	1.0	1	5,202	30.0 s
Sustained decode	16K	2	Valid	304.3	164.4	121 ms	6.1 ms	2.0	2	8,650	28.4 s
Sustained decode	16K	4	Valid	594.6	150.9	144 ms	6.6 ms	4.0	4	17,838	30.0 s
Sustained decode	16K	8	Valid	670.5	98.0	239 ms	10.2 ms	8.0	0	20,111	30.0 s
Sustained decode	16K	16	Valid	1,336.3	111.3	367 ms	9.0 ms	16.0	0	40,087	30.0 s
Sustained decode	16K	32	Valid	2,571.8	95.7	483 ms	10.5 ms	32.0	0	77,157	30.0 s
Sustained decode	16K	64	Valid	4,065.7	75.9	822 ms	13.2 ms	64.0	0	121,940	30.0 s
Sustained decode	16K	128	Valid	5,335.9	55.6	1,488 ms	18.0 ms	128.0	0	160,035	30.0 s
Sustained decode	32K	1	Valid	181.6	175.1	131 ms	5.7 ms	1.0	1	4,699	25.9 s
Sustained decode	32K	2	Valid	311.0	224.7	142 ms	4.8 ms	2.0	2	8,994	28.9 s
Sustained decode	32K	4	Valid	560.1	147.8	168 ms	6.8 ms	3.9	4	15,635	27.9 s
Sustained decode	32K	8	Valid	596.5	95.1	323 ms	10.5 ms	8.0	0	16,898	28.3 s
Sustained decode	32K	16	Valid	1,499.3	104.2	498 ms	9.6 ms	16.0	0	44,980	30.0 s
Sustained decode	32K	32	Valid	2,242.1	87.9	862 ms	11.4 ms	32.0	0	67,260	30.0 s
Sustained decode	32K	64	Valid	3,203.8	65.9	1,507 ms	15.2 ms	64.0	0	96,088	30.0 s
Sustained decode	32K	128	Valid	3,819.5	45.3	2,762 ms	22.1 ms	128.0	0	114,548	30.0 s
Sustained decode	64K	1	Valid	165.4	432.9	195 ms	3.6 ms	1.0	1	3,755	22.7 s
Sustained decode	64K	2	Valid	304.8	183.2	242 ms	5.5 ms	2.0	2	8,477	27.8 s
Sustained decode	64K	4	Valid	549.6	145.0	230 ms	6.9 ms	4.0	4	16,484	30.0 s
Sustained decode	64K	8	Valid	481.6	90.3	614 ms	11.1 ms	8.0	0	14,447	30.0 s
Sustained decode	64K	16	Valid	973.4	94.8	1,069 ms	10.6 ms	16.0	0	29,194	30.0 s
Sustained decode	64K	32	Valid	1,709.0	73.5	1,806 ms	13.6 ms	32.0	0	51,261	30.0 s
Sustained decode	64K	64	Valid	1,978.6	51.5	3,247 ms	19.4 ms	64.0	0	59,337	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	157.8	163.6	344 ms	6.1 ms	1.0	1	4,406	27.9 s
Sustained decode	123K	2	Valid	305.7	150.0	401 ms	6.7 ms	1.9	2	9,170	30.0 s
Sustained decode	123K	4	Valid	530.8	132.3	399 ms	7.6 ms	3.9	4	15,924	30.0 s
Sustained decode	123K	8	Valid	459.3	82.7	1,062 ms	12.1 ms	8.0	0	13,778	30.0 s
Sustained decode	123K	16	Valid	781.8	78.7	1,877 ms	12.7 ms	16.0	0	23,449	30.0 s
Sustained decode	123K	32	Valid	983.2	56.8	3,248 ms	17.6 ms	32.0	0	29,488	30.0 s
Sustained decode	123K	64	Valid	1,522.2	36.6	6,274 ms	27.3 ms	64.0	0	45,635	30.0 s
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.