Speed test matrix

Trinity-Nano-Preview-W4A16 on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 2, 2026, 22:09 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: trinity
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-arcee-vllm/bin/vllm" serve arcee-ai/Trinity-Nano-Preview-W4A16 --served-model-name trinity --trust-remote-code --tensor-parallel-size 1 --max-model-len 131072 --gpu-memory-utilization 0.884365 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16

Prefill

prompt tok/s by context length

8K context62.4K131 ms TTFT

16K context61.7K263 ms TTFT

32K context67.4K480 ms TTFT

64K context66.2K976 ms TTFT

123K context57.3K2,163 ms TTFT

128K context56.3K2,292 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	180.960 ms TTFT	351.393 ms TTFT	673.481 ms TTFT	915.5187 ms TTFT	2,429.0153 ms TTFT	4,124.3197 ms TTFT	6,190.2590 ms TTFT	8,973.3845 ms TTFT
16K	179.291 ms TTFT	346.380 ms TTFT	664.9127 ms TTFT	902.2277 ms TTFT	2,293.2187 ms TTFT	3,854.1267 ms TTFT	5,655.9789 ms TTFT	7,909.61,528 ms TTFT
32K	175.3128 ms TTFT	337.5115 ms TTFT	643.2199 ms TTFT	857.9282 ms TTFT	2,138.6334 ms TTFT	3,458.5854 ms TTFT	4,796.51,512 ms TTFT	6,263.32,867 ms TTFT
64K	170.2190 ms TTFT	327.0214 ms TTFT	611.8289 ms TTFT	823.9594 ms TTFT	1,884.2887 ms TTFT	2,809.61,777 ms TTFT	3,690.73,336 ms TTFT	Over capacity
123K	162.1362 ms TTFT	308.3412 ms TTFT	559.4414 ms TTFT	746.51,110 ms TTFT	1,529.91,829 ms TTFT	2,094.33,450 ms TTFT	2,519.86,301 ms TTFT	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	62.4K	—	131 ms	—	—	1	—	—
Prefill	16K	1	Valid	61.7K	—	263 ms	—	—	1	—	—
Prefill	32K	1	Valid	67.4K	—	480 ms	—	—	1	—	—
Prefill	64K	1	Valid	66.2K	—	976 ms	—	—	1	—	—
Prefill	123K	1	Valid	57.3K	—	2,163 ms	—	—	1	—	—
Prefill	128K	1	Valid	56.3K	—	2,292 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	180.9	186.1	60 ms	5.4 ms	1.0	1	5,426	30.0 s
Sustained decode	8K	2	Valid	351.3	176.6	93 ms	5.7 ms	2.0	2	10,544	30.0 s
Sustained decode	8K	4	Valid	673.4	169.5	81 ms	5.9 ms	4.0	4	20,203	30.0 s
Sustained decode	8K	8	Valid	915.5	124.2	187 ms	8.1 ms	8.0	0	27,472	30.0 s
Sustained decode	8K	16	Valid	2,429.0	162.7	153 ms	6.1 ms	16.0	16	72,862	30.0 s
Sustained decode	8K	32	Valid	4,124.3	130.5	197 ms	7.7 ms	31.6	32	123,732	30.0 s
Sustained decode	8K	64	Valid	6,190.2	97.4	590 ms	10.3 ms	64.0	0	185,728	30.0 s
Sustained decode	8K	128	Valid	8,973.3	70.7	845 ms	14.1 ms	128.0	0	269,184	30.0 s
Sustained decode	16K	1	Valid	179.2	233.6	91 ms	4.3 ms	1.0	1	5,377	30.0 s
Sustained decode	16K	2	Valid	346.3	218.7	80 ms	4.8 ms	2.0	2	10,387	30.0 s
Sustained decode	16K	4	Valid	664.9	167.5	127 ms	6.0 ms	4.0	4	19,947	30.0 s
Sustained decode	16K	8	Valid	902.2	113.1	277 ms	8.8 ms	8.0	0	27,064	30.0 s
Sustained decode	16K	16	Valid	2,293.2	144.6	187 ms	6.9 ms	16.0	16	68,796	30.0 s
Sustained decode	16K	32	Valid	3,854.1	121.3	267 ms	8.2 ms	31.8	32	115,583	30.0 s
Sustained decode	16K	64	Valid	5,655.9	88.9	789 ms	11.2 ms	64.0	0	169,664	30.0 s
Sustained decode	16K	128	Valid	7,909.6	63.1	1,528 ms	15.8 ms	128.0	0	237,312	30.0 s
Sustained decode	32K	1	Valid	175.3	178.4	128 ms	5.6 ms	1.0	1	5,261	30.0 s
Sustained decode	32K	2	Valid	337.5	225.1	115 ms	4.7 ms	2.0	2	10,124	30.0 s
Sustained decode	32K	4	Valid	643.2	208.6	199 ms	5.0 ms	4.0	4	19,294	30.0 s
Sustained decode	32K	8	Valid	857.9	107.5	282 ms	9.3 ms	8.0	0	25,737	30.0 s
Sustained decode	32K	16	Valid	2,138.6	135.1	334 ms	7.4 ms	15.7	16	64,147	30.0 s
Sustained decode	32K	32	Valid	3,458.5	108.0	854 ms	9.3 ms	32.0	0	103,744	30.0 s
Sustained decode	32K	64	Valid	4,796.5	75.4	1,512 ms	13.3 ms	64.0	0	143,872	30.0 s
Sustained decode	32K	128	Valid	6,263.3	69.5	2,867 ms	14.4 ms	128.0	0	187,904	30.0 s
Sustained decode	64K	1	Valid	170.2	289.1	190 ms	4.1 ms	1.0	1	5,105	30.0 s
Sustained decode	64K	2	Valid	327.0	187.1	214 ms	5.4 ms	2.0	2	9,813	30.0 s
Sustained decode	64K	4	Valid	611.8	154.6	289 ms	6.5 ms	3.9	4	18,351	30.0 s
Sustained decode	64K	8	Valid	823.9	103.2	594 ms	9.7 ms	8.0	0	24,712	30.0 s
Sustained decode	64K	16	Valid	1,884.2	118.0	887 ms	8.5 ms	16.0	4	56,512	30.0 s
Sustained decode	64K	32	Valid	2,809.6	87.9	1,777 ms	11.4 ms	32.0	0	84,289	30.0 s
Sustained decode	64K	64	Valid	3,690.7	73.2	3,336 ms	13.7 ms	64.0	0	110,720	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	162.1	253.3	362 ms	4.2 ms	1.0	1	4,864	30.0 s
Sustained decode	123K	2	Valid	308.3	231.5	412 ms	4.4 ms	2.0	2	9,247	30.0 s
Sustained decode	123K	4	Valid	559.4	156.2	414 ms	6.4 ms	3.9	4	16,782	30.0 s
Sustained decode	123K	8	Valid	746.5	97.9	1,110 ms	10.2 ms	8.0	0	22,400	30.0 s
Sustained decode	123K	16	Valid	1,529.9	102.7	1,829 ms	9.8 ms	16.0	0	45,888	30.0 s
Sustained decode	123K	32	Valid	2,094.3	74.7	3,450 ms	13.4 ms	32.0	0	62,816	30.0 s
Sustained decode	123K	64	Valid	2,519.8	70.2	6,301 ms	14.3 ms	64.0	0	75,584	30.0 s
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.