Speed test matrix

Trinity-Mini-FP8-Block on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 2, 2026, 18:29 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: trinity
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_DEEP_GEMM=1 "$HOME/env_h200-arcee-vllm/bin/vllm" serve arcee-ai/Trinity-Mini-FP8-Block --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --reasoning-parser deepseek_r1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context46.4K177 ms TTFT

16K context46.9K347 ms TTFT

32K context48.7K664 ms TTFT

64K context43.1K1,498 ms TTFT

123K context32.8K3,784 ms TTFT

128K context31.9K4,039 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	218.861 ms TTFT	372.663 ms TTFT	595.185 ms TTFT	912.2173 ms TTFT	1,318.4246 ms TTFT	2,284.8348 ms TTFT	3,840.4621 ms TTFT	6,041.8902 ms TTFT
16K	217.289 ms TTFT	369.3119 ms TTFT	587.3162 ms TTFT	895.7203 ms TTFT	1,276.9299 ms TTFT	2,191.4487 ms TTFT	3,593.9829 ms TTFT	5,423.51,467 ms TTFT
32K	213.1127 ms TTFT	361.7125 ms TTFT	573.9195 ms TTFT	873.0301 ms TTFT	1,225.1473 ms TTFT	2,031.3848 ms TTFT	3,167.91,400 ms TTFT	4,488.42,833 ms TTFT
64K	207.2188 ms TTFT	346.8245 ms TTFT	546.0248 ms TTFT	815.8607 ms TTFT	1,124.61,096 ms TTFT	1,763.21,862 ms TTFT	2,559.23,729 ms TTFT	Over capacity
123K	196.3372 ms TTFT	322.9404 ms TTFT	500.7449 ms TTFT	728.71,069 ms TTFT	971.51,988 ms TTFT	1,412.73,374 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	46.4K	—	177 ms	—	—	1	—	—
Prefill	16K	1	Valid	46.9K	—	347 ms	—	—	1	—	—
Prefill	32K	1	Valid	48.7K	—	664 ms	—	—	1	—	—
Prefill	64K	1	Valid	43.1K	—	1,498 ms	—	—	1	—	—
Prefill	123K	1	Valid	32.8K	—	3,784 ms	—	—	1	—	—
Prefill	128K	1	Valid	31.9K	—	4,039 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	218.8	219.5	61 ms	4.6 ms	1.0	1	6,564	30.0 s
Sustained decode	8K	2	Valid	372.6	186.9	63 ms	5.3 ms	2.0	2	11,178	30.0 s
Sustained decode	8K	4	Valid	595.1	149.3	85 ms	6.7 ms	4.0	4	17,852	30.0 s
Sustained decode	8K	8	Valid	912.2	114.4	173 ms	8.7 ms	8.0	0	27,360	30.0 s
Sustained decode	8K	16	Valid	1,318.4	82.7	246 ms	12.1 ms	16.0	0	39,552	30.0 s
Sustained decode	8K	32	Valid	2,284.8	71.9	348 ms	13.9 ms	32.0	0	68,545	30.0 s
Sustained decode	8K	64	Valid	3,840.4	60.4	621 ms	16.6 ms	64.0	0	115,200	30.0 s
Sustained decode	8K	128	Valid	6,041.8	47.6	902 ms	21.0 ms	128.0	0	181,248	30.0 s
Sustained decode	16K	1	Valid	217.2	218.1	89 ms	4.6 ms	1.0	1	6,514	30.0 s
Sustained decode	16K	2	Valid	369.3	185.7	119 ms	5.4 ms	2.0	2	11,080	30.0 s
Sustained decode	16K	4	Valid	587.3	147.8	162 ms	6.8 ms	3.9	4	17,616	30.0 s
Sustained decode	16K	8	Valid	895.7	112.2	203 ms	8.9 ms	8.0	0	26,872	30.0 s
Sustained decode	16K	16	Valid	1,276.9	80.2	299 ms	12.5 ms	16.0	0	38,304	30.0 s
Sustained decode	16K	32	Valid	2,191.4	68.8	487 ms	14.5 ms	32.0	0	65,729	30.0 s
Sustained decode	16K	64	Valid	3,593.9	56.4	829 ms	17.7 ms	64.0	0	107,840	30.0 s
Sustained decode	16K	128	Valid	5,423.5	42.7	1,467 ms	23.4 ms	128.0	0	162,688	30.0 s
Sustained decode	32K	1	Valid	213.1	214.5	127 ms	4.7 ms	1.0	1	6,396	30.0 s
Sustained decode	32K	2	Valid	361.7	181.9	125 ms	5.5 ms	2.0	2	10,851	30.0 s
Sustained decode	32K	4	Valid	573.9	144.8	195 ms	6.9 ms	4.0	4	17,215	30.0 s
Sustained decode	32K	8	Valid	873.0	109.3	301 ms	9.1 ms	8.0	0	26,192	30.0 s
Sustained decode	32K	16	Valid	1,225.1	76.8	473 ms	13.0 ms	16.0	0	36,752	30.0 s
Sustained decode	32K	32	Valid	2,031.3	63.7	848 ms	15.7 ms	32.0	0	60,928	30.0 s
Sustained decode	32K	64	Valid	3,167.9	49.8	1,400 ms	20.1 ms	64.0	0	95,040	30.0 s
Sustained decode	32K	128	Valid	4,488.4	35.3	2,833 ms	28.3 ms	128.0	0	134,656	30.0 s
Sustained decode	64K	1	Valid	207.2	209.1	188 ms	4.8 ms	1.0	1	6,217	30.0 s
Sustained decode	64K	2	Valid	346.8	175.3	245 ms	5.7 ms	2.0	2	10,404	30.0 s
Sustained decode	64K	4	Valid	546.0	137.7	248 ms	7.3 ms	4.0	4	16,376	30.0 s
Sustained decode	64K	8	Valid	815.8	102.0	607 ms	9.8 ms	8.0	0	24,472	30.0 s
Sustained decode	64K	16	Valid	1,124.6	70.4	1,096 ms	14.2 ms	16.0	0	33,728	30.0 s
Sustained decode	64K	32	Valid	1,763.2	55.2	1,862 ms	18.1 ms	32.0	0	52,896	30.0 s
Sustained decode	64K	64	Valid	2,559.2	40.1	3,729 ms	24.9 ms	64.0	0	76,736	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	196.3	199.3	372 ms	5.0 ms	1.0	1	5,889	30.0 s
Sustained decode	123K	2	Valid	322.9	164.2	404 ms	6.1 ms	2.0	2	9,692	30.0 s
Sustained decode	123K	4	Valid	500.7	127.0	449 ms	7.9 ms	3.9	4	15,019	30.0 s
Sustained decode	123K	8	Valid	728.7	91.0	1,069 ms	11.0 ms	8.0	0	21,856	30.0 s
Sustained decode	123K	16	Valid	971.5	60.6	1,988 ms	16.5 ms	16.0	0	29,136	30.0 s
Sustained decode	123K	32	Valid	1,412.7	44.1	3,374 ms	22.7 ms	32.0	0	42,368	30.0 s
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.