Speed test matrix

gpt-oss-20b on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 5, 2026, 07:27 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: MXFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: gpt-oss
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-gpt-oss-vllm/bin/vllm" serve openai/gpt-oss-20b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.877674 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context19.2K426 ms TTFT

16K context27.8K583 ms TTFT

32K context31.7K1,013 ms TTFT

64K context28.6K2,238 ms TTFT

123K context23.3K5,269 ms TTFT

128K context21.9K5,845 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	280.544 ms TTFT	492.545 ms TTFT	810.849 ms TTFT	1,320.158 ms TTFT	2,167.766 ms TTFT	3,697.967 ms TTFT	5,434.592 ms TTFT	6,952.6140 ms TTFT
16K	272.465 ms TTFT	484.569 ms TTFT	795.569 ms TTFT	1,247.575 ms TTFT	2,052.879 ms TTFT	3,103.588 ms TTFT	4,429.6120 ms TTFT	5,075.2191 ms TTFT
32K	264.290 ms TTFT	463.8101 ms TTFT	740.098 ms TTFT	1,137.396 ms TTFT	1,742.3105 ms TTFT	2,531.7125 ms TTFT	3,279.4166 ms TTFT	Over capacity
64K	254.2123 ms TTFT	423.5144 ms TTFT	648.5145 ms TTFT	954.3149 ms TTFT	1,331.0154 ms TTFT	1,745.3177 ms TTFT	2,097.6252 ms TTFT	Over capacity
123K	231.6207 ms TTFT	362.2223 ms TTFT	511.9229 ms TTFT	723.6233 ms TTFT	855.4245 ms TTFT	1,102.2296 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	19.2K	—	426 ms	—	—	1	—	—
Prefill	16K	1	Valid	27.8K	—	583 ms	—	—	1	—	—
Prefill	32K	1	Valid	31.7K	—	1,013 ms	—	—	1	—	—
Prefill	64K	1	Valid	28.6K	—	2,238 ms	—	—	1	—	—
Prefill	123K	1	Valid	23.3K	—	5,269 ms	—	—	1	—	—
Prefill	128K	1	Valid	21.9K	—	5,845 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	280.5	283.2	44 ms	3.5 ms	1.0	5	8,414	30.0 s
Sustained decode	8K	2	Valid	492.5	251.5	45 ms	4.0 ms	2.0	19	14,775	30.0 s
Sustained decode	8K	4	Valid	810.8	208.3	49 ms	4.8 ms	3.9	23	24,325	30.0 s
Sustained decode	8K	8	Valid	1,320.1	172.0	58 ms	5.8 ms	8.0	70	39,597	30.0 s
Sustained decode	8K	16	Valid	2,167.7	141.1	66 ms	7.1 ms	16.0	86	65,023	30.0 s
Sustained decode	8K	32	Valid	3,697.9	120.4	67 ms	8.3 ms	31.8	138	110,929	30.0 s
Sustained decode	8K	64	Valid	5,434.5	88.1	92 ms	11.4 ms	63.9	276	163,013	30.0 s
Sustained decode	8K	128	Valid	6,952.6	55.5	140 ms	18.0 ms	127.6	490	208,714	30.0 s
Sustained decode	16K	1	Valid	272.4	283.3	65 ms	3.5 ms	1.0	13	8,171	30.0 s
Sustained decode	16K	2	Valid	484.5	249.9	69 ms	4.0 ms	2.0	11	14,535	30.0 s
Sustained decode	16K	4	Valid	795.5	204.6	69 ms	4.9 ms	4.0	24	23,863	30.0 s
Sustained decode	16K	8	Valid	1,247.5	163.1	75 ms	6.1 ms	8.0	45	37,418	30.0 s
Sustained decode	16K	16	Valid	2,052.8	133.1	79 ms	7.5 ms	15.9	72	61,585	30.0 s
Sustained decode	16K	32	Valid	3,103.5	101.5	88 ms	9.9 ms	31.9	159	93,104	30.0 s
Sustained decode	16K	64	Valid	4,429.6	70.6	120 ms	14.2 ms	63.9	175	132,874	30.0 s
Sustained decode	16K	128	Valid	5,075.2	41.0	191 ms	24.4 ms	127.4	315	152,291	30.0 s
Sustained decode	32K	1	Valid	264.2	274.6	90 ms	3.6 ms	1.0	12	7,926	30.0 s
Sustained decode	32K	2	Valid	463.8	240.0	101 ms	4.2 ms	2.0	13	13,913	30.0 s
Sustained decode	32K	4	Valid	740.0	191.3	98 ms	5.2 ms	3.9	19	22,199	30.0 s
Sustained decode	32K	8	Valid	1,137.3	146.1	96 ms	6.8 ms	8.0	25	34,113	30.0 s
Sustained decode	32K	16	Valid	1,742.3	112.4	105 ms	8.9 ms	16.0	40	52,261	30.0 s
Sustained decode	32K	32	Valid	2,531.7	80.8	125 ms	12.4 ms	31.9	66	75,936	30.0 s
Sustained decode	32K	64	Valid	3,279.4	52.4	166 ms	19.1 ms	63.9	128	98,350	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	254.2	266.4	123 ms	3.8 ms	1.0	4	7,591	29.9 s
Sustained decode	64K	2	Valid	423.5	220.4	144 ms	4.5 ms	2.0	9	12,706	30.0 s
Sustained decode	64K	4	Valid	648.5	169.1	145 ms	5.9 ms	3.9	19	19,456	30.0 s
Sustained decode	64K	8	Valid	954.3	122.8	149 ms	8.1 ms	8.0	28	28,626	30.0 s
Sustained decode	64K	16	Valid	1,331.0	86.5	154 ms	11.6 ms	15.8	51	39,932	30.0 s
Sustained decode	64K	32	Valid	1,745.3	56.1	177 ms	17.8 ms	31.9	84	52,357	30.0 s
Sustained decode	64K	64	Valid	2,097.6	33.7	252 ms	29.7 ms	63.5	136	62,911	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	231.6	243.5	207 ms	4.1 ms	1.0	3	6,947	30.0 s
Sustained decode	123K	2	Valid	362.2	194.6	223 ms	5.1 ms	2.0	11	10,866	30.0 s
Sustained decode	123K	4	Valid	511.9	138.9	229 ms	7.2 ms	3.9	38	15,358	30.0 s
Sustained decode	123K	8	Valid	723.6	91.6	233 ms	10.9 ms	8.0	33	21,683	30.0 s
Sustained decode	123K	16	Valid	855.4	56.7	245 ms	17.6 ms	15.8	90	25,674	30.0 s
Sustained decode	123K	32	Valid	1,102.2	35.6	296 ms	28.1 ms	31.8	111	33,047	30.0 s
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.