Speed test matrix

gpt-oss-safeguard-20b on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 5, 2026, 08:16 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: MXFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: gpt-oss
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-gpt-oss-vllm/bin/vllm" serve openai/gpt-oss-safeguard-20b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.877674 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context43K190 ms TTFT

16K context45.2K358 ms TTFT

32K context40.7K789 ms TTFT

64K context29.2K2,195 ms TTFT

123K context23.4K5,253 ms TTFT

128K context22K5,813 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	280.849 ms TTFT	495.644 ms TTFT	853.598 ms TTFT	1,371.472 ms TTFT	2,306.5140 ms TTFT	3,711.5394 ms TTFT	5,577.4679 ms TTFT	7,201.81,407 ms TTFT
16K	275.671 ms TTFT	486.070 ms TTFT	814.670 ms TTFT	1,281.690 ms TTFT	2,085.1124 ms TTFT	3,192.5609 ms TTFT	4,536.61,018 ms TTFT	5,503.31,731 ms TTFT
32K	267.988 ms TTFT	467.3105 ms TTFT	761.6106 ms TTFT	1,164.4179 ms TTFT	1,794.2572 ms TTFT	2,543.51,111 ms TTFT	3,304.91,744 ms TTFT	Over capacity
64K	253.8139 ms TTFT	424.0148 ms TTFT	665.2152 ms TTFT	954.6177 ms TTFT	1,372.3970 ms TTFT	1,776.11,989 ms TTFT	2,082.83,437 ms TTFT	Over capacity
123K	230.2209 ms TTFT	368.5231 ms TTFT	539.7226 ms TTFT	741.1246 ms TTFT	945.9237 ms TTFT	1,125.0277 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	43K	—	190 ms	—	—	1	—	—
Prefill	16K	1	Valid	45.2K	—	358 ms	—	—	1	—	—
Prefill	32K	1	Valid	40.7K	—	789 ms	—	—	1	—	—
Prefill	64K	1	Valid	29.2K	—	2,195 ms	—	—	1	—	—
Prefill	123K	1	Valid	23.4K	—	5,253 ms	—	—	1	—	—
Prefill	128K	1	Valid	22K	—	5,813 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	280.8	281.7	49 ms	3.5 ms	1.0	2	8,425	30.0 s
Sustained decode	8K	2	Valid	495.6	249.7	44 ms	4.0 ms	2.0	7	14,868	30.0 s
Sustained decode	8K	4	Valid	853.5	215.6	98 ms	4.6 ms	4.0	5	25,601	30.0 s
Sustained decode	8K	8	Valid	1,371.4	171.0	72 ms	5.8 ms	8.0	14	41,141	30.0 s
Sustained decode	8K	16	Valid	2,306.5	144.1	140 ms	6.9 ms	16.0	31	69,199	30.0 s
Sustained decode	8K	32	Valid	3,711.5	118.0	394 ms	8.5 ms	32.0	30	111,324	30.0 s
Sustained decode	8K	64	Valid	5,577.4	88.5	679 ms	11.3 ms	63.9	54	167,290	30.0 s
Sustained decode	8K	128	Valid	7,201.8	57.3	1,407 ms	17.4 ms	127.9	73	216,048	30.0 s
Sustained decode	16K	1	Valid	275.6	278.7	71 ms	3.6 ms	1.0	3	8,266	30.0 s
Sustained decode	16K	2	Valid	486.0	246.3	70 ms	4.1 ms	2.0	6	14,581	30.0 s
Sustained decode	16K	4	Valid	814.6	206.3	70 ms	4.8 ms	4.0	11	24,439	30.0 s
Sustained decode	16K	8	Valid	1,281.6	163.8	90 ms	6.1 ms	7.9	18	38,448	30.0 s
Sustained decode	16K	16	Valid	2,085.1	132.4	124 ms	7.6 ms	16.0	24	62,558	30.0 s
Sustained decode	16K	32	Valid	3,192.5	101.5	609 ms	9.9 ms	32.0	17	95,782	30.0 s
Sustained decode	16K	64	Valid	4,536.6	72.2	1,018 ms	13.9 ms	63.9	46	136,077	30.0 s
Sustained decode	16K	128	Valid	5,503.3	44.1	1,731 ms	22.7 ms	127.9	59	165,074	30.0 s
Sustained decode	32K	1	Valid	267.9	271.5	88 ms	3.7 ms	1.0	3	8,038	30.0 s
Sustained decode	32K	2	Valid	467.3	234.6	105 ms	4.3 ms	2.0	3	14,018	30.0 s
Sustained decode	32K	4	Valid	761.6	193.3	106 ms	5.2 ms	4.0	9	22,847	30.0 s
Sustained decode	32K	8	Valid	1,164.4	147.0	179 ms	6.8 ms	8.0	12	34,928	30.0 s
Sustained decode	32K	16	Valid	1,794.2	113.6	572 ms	8.8 ms	16.0	7	53,822	30.0 s
Sustained decode	32K	32	Valid	2,543.5	81.0	1,111 ms	12.4 ms	31.9	20	76,303	30.0 s
Sustained decode	32K	64	Valid	3,304.9	52.8	1,744 ms	18.9 ms	63.9	28	99,098	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	253.8	255.4	139 ms	3.9 ms	1.0	2	7,614	30.0 s
Sustained decode	64K	2	Valid	424.0	220.0	148 ms	4.5 ms	2.0	9	12,719	30.0 s
Sustained decode	64K	4	Valid	665.2	169.4	152 ms	5.9 ms	4.0	13	19,947	30.0 s
Sustained decode	64K	8	Valid	954.6	121.1	177 ms	8.3 ms	8.0	14	28,633	30.0 s
Sustained decode	64K	16	Valid	1,372.3	86.9	970 ms	11.5 ms	16.0	11	41,161	30.0 s
Sustained decode	64K	32	Valid	1,776.1	56.9	1,989 ms	17.6 ms	32.0	12	53,289	30.0 s
Sustained decode	64K	64	Valid	2,082.8	34.0	3,437 ms	29.4 ms	63.9	22	62,462	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	230.2	242.6	209 ms	4.1 ms	1.0	5	6,906	30.0 s
Sustained decode	123K	2	Valid	368.5	192.6	231 ms	5.2 ms	1.9	7	11,056	30.0 s
Sustained decode	123K	4	Valid	539.7	140.7	226 ms	7.1 ms	3.9	12	16,192	30.0 s
Sustained decode	123K	8	Valid	741.1	94.5	246 ms	10.6 ms	8.0	9	22,228	30.0 s
Sustained decode	123K	16	Valid	945.9	60.7	237 ms	16.5 ms	15.8	23	28,376	30.0 s
Sustained decode	123K	32	Valid	1,125.0	36.4	277 ms	27.5 ms	31.7	54	33,741	30.0 s
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.