Speed test matrix

gpt-oss-safeguard-120b on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 5, 2026, 07:50 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: MXFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: gpt-oss
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-gpt-oss-vllm/bin/vllm" serve openai/gpt-oss-safeguard-120b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.877719 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context29.7K276 ms TTFT

16K context29.7K545 ms TTFT

32K context26.3K1,221 ms TTFT

64K context21.1K3,029 ms TTFT

123K context15.3K8,054 ms TTFT

128K context14.1K9,088 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	193.952 ms TTFT	343.459 ms TTFT	530.969 ms TTFT	824.7166 ms TTFT	1,207.8252 ms TTFT	1,821.0377 ms TTFT	2,761.7743 ms TTFT	3,931.61,237 ms TTFT
16K	190.971 ms TTFT	332.374 ms TTFT	501.583 ms TTFT	764.6101 ms TTFT	1,134.8122 ms TTFT	1,652.5619 ms TTFT	2,385.5994 ms TTFT	Over capacity
32K	185.3100 ms TTFT	317.3111 ms TTFT	480.9193 ms TTFT	690.1120 ms TTFT	996.1152 ms TTFT	1,407.3189 ms TTFT	Over capacity	Over capacity
64K	174.7148 ms TTFT	287.2160 ms TTFT	417.9165 ms TTFT	584.2170 ms TTFT	792.0205 ms TTFT	Over capacity	Over capacity	Over capacity
123K	158.7212 ms TTFT	251.6233 ms TTFT	341.1232 ms TTFT	455.4260 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	29.7K	—	276 ms	—	—	1	—	—
Prefill	16K	1	Valid	29.7K	—	545 ms	—	—	1	—	—
Prefill	32K	1	Valid	26.3K	—	1,221 ms	—	—	1	—	—
Prefill	64K	1	Valid	21.1K	—	3,029 ms	—	—	1	—	—
Prefill	123K	1	Valid	15.3K	—	8,054 ms	—	—	1	—	—
Prefill	128K	1	Valid	14.1K	—	9,088 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	193.9	196.5	52 ms	5.1 ms	1.0	4	5,818	30.0 s
Sustained decode	8K	2	Valid	343.4	173.7	59 ms	5.8 ms	2.0	5	10,300	30.0 s
Sustained decode	8K	4	Valid	530.9	134.7	69 ms	7.4 ms	4.0	7	15,927	30.0 s
Sustained decode	8K	8	Valid	824.7	104.8	166 ms	9.5 ms	8.0	4	24,736	30.0 s
Sustained decode	8K	16	Valid	1,207.8	77.7	252 ms	12.9 ms	16.0	14	36,223	30.0 s
Sustained decode	8K	32	Valid	1,821.0	59.0	377 ms	16.9 ms	32.0	23	54,637	30.0 s
Sustained decode	8K	64	Valid	2,761.7	44.6	743 ms	22.4 ms	63.9	38	82,872	30.0 s
Sustained decode	8K	128	Valid	3,931.6	31.8	1,237 ms	31.5 ms	127.8	71	117,903	30.0 s
Sustained decode	16K	1	Valid	190.9	194.6	71 ms	5.1 ms	1.0	5	5,728	30.0 s
Sustained decode	16K	2	Valid	332.3	167.7	74 ms	6.0 ms	2.0	7	9,968	30.0 s
Sustained decode	16K	4	Valid	501.5	127.2	83 ms	7.9 ms	4.0	9	15,043	30.0 s
Sustained decode	16K	8	Valid	764.6	97.2	101 ms	10.3 ms	8.0	12	22,936	30.0 s
Sustained decode	16K	16	Valid	1,134.8	71.4	122 ms	14.0 ms	15.9	24	34,046	30.0 s
Sustained decode	16K	32	Valid	1,652.5	53.4	619 ms	18.7 ms	31.8	29	49,551	30.0 s
Sustained decode	16K	64	Valid	2,385.5	38.6	994 ms	25.9 ms	63.9	52	71,534	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	185.3	186.2	100 ms	5.4 ms	1.0	1	5,560	30.0 s
Sustained decode	32K	2	Valid	317.3	160.9	111 ms	6.2 ms	2.0	4	9,519	30.0 s
Sustained decode	32K	4	Valid	480.9	124.5	193 ms	8.0 ms	4.0	4	14,424	30.0 s
Sustained decode	32K	8	Valid	690.1	88.0	120 ms	11.4 ms	8.0	19	20,698	30.0 s
Sustained decode	32K	16	Valid	996.1	62.8	152 ms	15.9 ms	16.0	23	29,883	30.0 s
Sustained decode	32K	32	Valid	1,407.3	44.2	189 ms	22.6 ms	31.8	43	42,216	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	174.7	178.1	148 ms	5.6 ms	1.0	3	5,242	30.0 s
Sustained decode	64K	2	Valid	287.2	148.3	160 ms	6.7 ms	2.0	9	8,617	30.0 s
Sustained decode	64K	4	Valid	417.9	108.0	165 ms	9.3 ms	4.0	17	12,536	30.0 s
Sustained decode	64K	8	Valid	584.2	74.6	170 ms	13.4 ms	8.0	20	17,528	30.0 s
Sustained decode	64K	16	Valid	792.0	50.5	205 ms	19.8 ms	15.9	19	23,756	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	158.7	164.8	212 ms	6.1 ms	1.0	4	4,758	30.0 s
Sustained decode	123K	2	Valid	251.6	129.2	233 ms	7.7 ms	2.0	6	7,545	30.0 s
Sustained decode	123K	4	Valid	341.1	89.1	232 ms	11.2 ms	4.0	20	10,235	30.0 s
Sustained decode	123K	8	Valid	455.4	58.4	260 ms	17.1 ms	8.0	17	13,661	30.0 s
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.