Speed test matrix

Trinity-Nano-Preview-NVFP4 on H200 NVL 141GB

vllm · 0.18.0 · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 2, 2026, 21:32 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.18.0
Quantization: NVFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: trinity
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-arcee-nvfp4-vllm/bin/vllm" serve arcee-ai/Trinity-Nano-Preview-NVFP4 --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.871965 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16

Prefill

prompt tok/s by context length

8K context52.7K155 ms TTFT

16K context69.1K235 ms TTFT

32K context67K483 ms TTFT

64K context63.5K1,017 ms TTFT

123K context55.4K2,239 ms TTFT

128K context54.4K2,373 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	178.145 ms TTFT	334.283 ms TTFT	645.4135 ms TTFT	878.2208 ms TTFT	2,308.6362 ms TTFT	3,845.3624 ms TTFT	5,729.4719 ms TTFT	8,393.91,501 ms TTFT
16K	176.174 ms TTFT	328.4113 ms TTFT	631.2205 ms TTFT	867.2335 ms TTFT	2,200.9583 ms TTFT	3,656.0958 ms TTFT	5,208.71,122 ms TTFT	7,383.82,079 ms TTFT
32K	172.6109 ms TTFT	321.4202 ms TTFT	612.6340 ms TTFT	841.5604 ms TTFT	2,029.21,038 ms TTFT	3,245.22,050 ms TTFT	4,497.92,250 ms TTFT	5,944.34,188 ms TTFT
64K	167.4203 ms TTFT	310.8381 ms TTFT	575.5673 ms TTFT	795.5826 ms TTFT	1,841.31,257 ms TTFT	2,719.12,317 ms TTFT	3,445.14,288 ms TTFT	Over capacity
123K	159.5358 ms TTFT	289.8789 ms TTFT	527.01,082 ms TTFT	721.82,523 ms TTFT	1,505.34,922 ms TTFT	2,023.09,534 ms TTFT	2,430.58,960 ms TTFT	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	52.7K	—	155 ms	—	—	1	—	—
Prefill	16K	1	Valid	69.1K	—	235 ms	—	—	1	—	—
Prefill	32K	1	Valid	67K	—	483 ms	—	—	1	—	—
Prefill	64K	1	Valid	63.5K	—	1,017 ms	—	—	1	—	—
Prefill	123K	1	Valid	55.4K	—	2,239 ms	—	—	1	—	—
Prefill	128K	1	Valid	54.4K	—	2,373 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	178.1	261.0	45 ms	3.8 ms	1.0	1	5,344	30.0 s
Sustained decode	8K	2	Valid	334.2	208.8	83 ms	5.0 ms	2.0	2	10,028	30.0 s
Sustained decode	8K	4	Valid	645.4	183.8	135 ms	5.4 ms	4.0	4	19,366	30.0 s
Sustained decode	8K	8	Valid	878.2	110.1	208 ms	9.1 ms	8.0	0	26,352	30.0 s
Sustained decode	8K	16	Valid	2,308.6	149.2	362 ms	6.7 ms	16.0	16	69,260	30.0 s
Sustained decode	8K	32	Valid	3,845.3	122.1	624 ms	8.2 ms	31.9	32	115,334	30.0 s
Sustained decode	8K	64	Valid	5,729.4	90.2	719 ms	11.1 ms	64.0	0	171,904	30.0 s
Sustained decode	8K	128	Valid	8,393.9	96.3	1,501 ms	10.4 ms	128.0	0	251,776	30.0 s
Sustained decode	16K	1	Valid	176.1	196.5	74 ms	5.1 ms	1.0	1	5,282	30.0 s
Sustained decode	16K	2	Valid	328.4	201.9	113 ms	5.0 ms	2.0	2	9,853	30.0 s
Sustained decode	16K	4	Valid	631.2	165.1	205 ms	6.1 ms	4.0	4	18,933	30.0 s
Sustained decode	16K	8	Valid	867.2	108.8	335 ms	9.2 ms	8.0	0	26,012	30.0 s
Sustained decode	16K	16	Valid	2,200.9	144.9	583 ms	6.9 ms	15.5	16	66,032	30.0 s
Sustained decode	16K	32	Valid	3,656.0	114.9	958 ms	8.7 ms	32.0	0	109,664	30.0 s
Sustained decode	16K	64	Valid	5,208.7	86.5	1,122 ms	11.6 ms	64.0	0	156,224	30.0 s
Sustained decode	16K	128	Valid	7,383.8	80.3	2,079 ms	12.5 ms	128.0	0	221,568	30.0 s
Sustained decode	32K	1	Valid	172.6	317.4	109 ms	3.4 ms	1.0	1	5,177	30.0 s
Sustained decode	32K	2	Valid	321.4	207.3	202 ms	4.9 ms	2.0	2	9,640	30.0 s
Sustained decode	32K	4	Valid	612.6	177.4	340 ms	5.6 ms	3.9	4	18,376	30.0 s
Sustained decode	32K	8	Valid	841.5	112.0	604 ms	9.0 ms	8.0	0	25,248	30.0 s
Sustained decode	32K	16	Valid	2,029.2	130.7	1,038 ms	7.6 ms	15.7	16	60,870	30.0 s
Sustained decode	32K	32	Valid	3,245.2	107.7	2,050 ms	9.3 ms	32.0	0	97,344	30.0 s
Sustained decode	32K	64	Valid	4,497.9	90.8	2,250 ms	11.0 ms	64.0	0	134,912	30.0 s
Sustained decode	32K	128	Valid	5,944.3	60.8	4,188 ms	16.5 ms	128.0	0	178,304	30.0 s
Sustained decode	64K	1	Valid	167.4	188.2	203 ms	5.4 ms	1.0	1	5,022	30.0 s
Sustained decode	64K	2	Valid	310.8	163.0	381 ms	6.1 ms	2.0	2	9,325	30.0 s
Sustained decode	64K	4	Valid	575.5	157.9	673 ms	6.3 ms	3.9	4	17,272	30.0 s
Sustained decode	64K	8	Valid	795.5	115.1	826 ms	8.7 ms	8.0	0	23,864	30.0 s
Sustained decode	64K	16	Valid	1,841.3	118.4	1,257 ms	8.5 ms	16.0	0	55,232	30.0 s
Sustained decode	64K	32	Valid	2,719.1	98.9	2,317 ms	10.1 ms	32.0	0	81,568	30.0 s
Sustained decode	64K	64	Valid	3,445.1	73.2	4,288 ms	13.7 ms	64.0	0	103,360	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	159.5	162.0	358 ms	6.2 ms	1.0	1	4,784	30.0 s
Sustained decode	123K	2	Valid	289.8	227.4	789 ms	5.0 ms	2.0	2	8,692	30.0 s
Sustained decode	123K	4	Valid	527.0	164.8	1,082 ms	6.1 ms	3.9	4	15,808	30.0 s
Sustained decode	123K	8	Valid	721.8	114.3	2,523 ms	8.9 ms	8.0	0	21,665	30.0 s
Sustained decode	123K	16	Valid	1,505.3	102.3	4,922 ms	9.8 ms	16.0	0	45,152	30.0 s
Sustained decode	123K	32	Valid	2,023.0	89.5	9,534 ms	11.2 ms	32.0	0	60,672	30.0 s
Sustained decode	123K	64	Valid	2,430.5	63.4	8,960 ms	15.8 ms	64.0	0	72,896	30.0 s
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.