Speed test matrix

gpt-oss-120b on H200 NVL 141GB

vllm · 0.26.1rc1.dev914+gc3dabcddc · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 8, 2026, 06:02 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev914+gc3dabcddc
Quantization: MXFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: c3dabcddc328

Run

Served model: gpt-oss
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_V2_MODEL_RUNNER=1 "$HOME/env_h200-z-lab-vllm/bin/vllm" serve openai/gpt-oss-120b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.818164 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/gpt-oss-120b-DFlash","num_speculative_tokens":9}'

Prefill

prompt tok/s by context length

8K context9,330878 ms TTFT

16K context14.6K1,106 ms TTFT

32K context19K1,694 ms TTFT

64K context18.7K3,419 ms TTFT

123K context15.7K7,841 ms TTFT

128K context12.8K9,966 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	215.7108 ms TTFT	367.285 ms TTFT	568.6132 ms TTFT	827.8161 ms TTFT	1,243.8187 ms TTFT	1,704.5257 ms TTFT	2,383.3370 ms TTFT	Over capacity
16K	196.1125 ms TTFT	310.097 ms TTFT	454.9149 ms TTFT	691.9169 ms TTFT	1,022.2217 ms TTFT	1,356.0312 ms TTFT	Over capacity	Over capacity
32K	160.3151 ms TTFT	254.7Requests queued156 ms TTFT	385.8174 ms TTFT	532.5209 ms TTFT	738.4253 ms TTFT	Over capacity	Over capacity	Over capacity
64K	112.3196 ms TTFT	179.1201 ms TTFT	260.4189 ms TTFT	345.6236 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
123K	88.4255 ms TTFT	128.3277 ms TTFT	171.9280 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	9,330.0	—	878 ms	—	—	1	—	—
Prefill	16K	1	Valid	14.6K	—	1,106 ms	—	—	1	—	—
Prefill	32K	1	Valid	19K	—	1,694 ms	—	—	1	—	—
Prefill	64K	1	Valid	18.7K	—	3,419 ms	—	—	1	—	—
Prefill	123K	1	Valid	15.7K	—	7,841 ms	—	—	1	—	—
Prefill	128K	1	Valid	12.8K	—	9,966 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	215.7	249.4	108 ms	4.0 ms	1.0	11	6,469	30.0 s
Sustained decode	8K	2	Valid	367.2	208.5	85 ms	4.8 ms	1.9	38	11,014	30.0 s
Sustained decode	8K	4	Valid	568.6	158.2	132 ms	6.3 ms	3.9	75	17,055	30.0 s
Sustained decode	8K	8	Valid	827.8	116.6	161 ms	8.6 ms	7.9	112	24,835	30.0 s
Sustained decode	8K	16	Valid	1,243.8	85.8	187 ms	11.7 ms	15.8	118	37,289	30.0 s
Sustained decode	8K	32	Valid	1,704.5	59.4	257 ms	16.8 ms	31.8	402	51,089	30.0 s
Sustained decode	8K	64	Valid	2,383.3	40.8	370 ms	24.5 ms	63.1	266	71,512	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	196.1	218.8	125 ms	4.6 ms	1.0	11	5,881	30.0 s
Sustained decode	16K	2	Valid	310.0	172.7	97 ms	5.8 ms	1.9	24	9,299	30.0 s
Sustained decode	16K	4	Valid	454.9	126.5	149 ms	7.9 ms	3.9	53	13,641	30.0 s
Sustained decode	16K	8	Valid	691.9	96.9	169 ms	10.3 ms	7.9	88	20,755	30.0 s
Sustained decode	16K	16	Valid	1,022.2	70.0	217 ms	14.3 ms	15.7	100	30,670	30.0 s
Sustained decode	16K	32	Valid	1,356.0	45.8	312 ms	21.9 ms	31.6	129	40,617	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	160.3	183.6	151 ms	5.4 ms	0.9	21	4,810	30.0 s
Sustained decode	32K	2	Requests queued	254.7	138.5	156 ms	7.2 ms	1.9	29	7,640	30.0 s
Sustained decode	32K	4	Valid	385.8	103.5	174 ms	9.7 ms	3.8	42	11,573	30.0 s
Sustained decode	32K	8	Valid	532.5	73.8	209 ms	13.6 ms	7.9	49	15,973	30.0 s
Sustained decode	32K	16	Valid	738.4	50.7	253 ms	19.7 ms	15.8	85	22,131	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	112.3	122.5	196 ms	8.2 ms	0.9	12	3,368	30.0 s
Sustained decode	64K	2	Valid	179.1	95.4	201 ms	10.5 ms	2.0	15	5,373	30.0 s
Sustained decode	64K	4	Valid	260.4	68.5	189 ms	14.6 ms	4.0	28	7,808	30.0 s
Sustained decode	64K	8	Valid	345.6	45.4	236 ms	22.0 ms	7.8	40	10,364	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	88.4	93.3	255 ms	10.7 ms	1.0	6	2,650	30.0 s
Sustained decode	123K	2	Valid	128.3	68.3	277 ms	14.6 ms	2.0	13	3,849	30.0 s
Sustained decode	123K	4	Valid	171.9	44.5	280 ms	22.5 ms	3.9	14	5,156	30.0 s
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.