Speed test matrix

gpt-oss-20b on RTX PRO 6000 Blackwell Server Edition 96GB

vllm · 0.30.1rc1.dev452+g73a583112 · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Oct 11, 2026, 16:56 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev452+g73a583112
Quantization: MXFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 73a5831127a9

Run

Served model: gpt-oss
Revision: main
Benchmark tool: llm-inference-bench v0.7.6
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-z-lab-vllm-73a5831127a9/bin/vllm" serve openai/gpt-oss-20b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --speculative-config '{"method":"dflash","model":"z-lab/gpt-oss-20b-DFlash","num_speculative_tokens":7}'

Prefill

prompt tok/s by context length

8K context36.4K225 ms TTFT

16K context34.8K465 ms TTFT

32K context30.5K1,055 ms TTFT

64K context23.6K2,711 ms TTFT

123K context16.7K7,377 ms TTFT

128K context16.3K7,835 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	177.733 ms TTFT	336.546 ms TTFT	631.658 ms TTFT	1,013.699 ms TTFT	1,575.5208 ms TTFT	1,711.8338 ms TTFT	1,982.5708 ms TTFT	Over capacity
16K	166.760 ms TTFT	291.981 ms TTFT	393.5109 ms TTFT	671.4173 ms TTFT	799.5326 ms TTFT	1,133.2625 ms TTFT	Over capacity	Over capacity
32K	112.974 ms TTFT	164.9103 ms TTFT	284.5160 ms TTFT	456.4285 ms TTFT	523.4585 ms TTFT	Over capacity	Over capacity	Over capacity
64K	69.7109 ms TTFT	128.6147 ms TTFT	164.5284 ms TTFT	204.5533 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
123K	50.2172 ms TTFT	86.4290 ms TTFT	99.9456 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	36.4K	—	225 ms	—	—	1	—	—
Prefill	16K	1	Valid	34.8K	—	465 ms	—	—	1	—	—
Prefill	32K	1	Valid	30.5K	—	1,055 ms	—	—	1	—	—
Prefill	64K	1	Valid	23.6K	—	2,711 ms	—	—	1	—	—
Prefill	123K	1	Valid	16.7K	—	7,377 ms	—	—	1	—	—
Prefill	128K	1	Valid	16.3K	—	7,835 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	177.7	196.7	33 ms	5.1 ms	1.0	1	3,923	22.1 s
Sustained decode	8K	2	Valid	336.5	172.4	46 ms	5.8 ms	2.0	2	9,608	28.6 s
Sustained decode	8K	4	Valid	631.6	154.0	58 ms	6.5 ms	4.0	4	18,287	29.0 s
Sustained decode	8K	8	Valid	1,013.6	135.6	99 ms	7.4 ms	8.0	7	30,393	30.0 s
Sustained decode	8K	16	Valid	1,575.5	98.3	208 ms	10.2 ms	16.0	1	47,266	30.0 s
Sustained decode	8K	32	Valid	1,711.8	72.3	338 ms	13.8 ms	32.0	0	51,323	30.0 s
Sustained decode	8K	64	Valid	1,982.5	43.4	708 ms	23.1 ms	64.0	0	59,455	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	166.7	156.2	60 ms	6.4 ms	1.0	1	5,002	30.0 s
Sustained decode	16K	2	Valid	291.9	146.7	81 ms	6.8 ms	2.0	2	8,754	30.0 s
Sustained decode	16K	4	Valid	393.5	132.5	109 ms	7.6 ms	4.0	2	11,803	30.0 s
Sustained decode	16K	8	Valid	671.4	94.4	173 ms	10.6 ms	8.0	0	20,137	30.0 s
Sustained decode	16K	16	Valid	799.5	67.2	326 ms	14.9 ms	16.0	0	23,991	30.0 s
Sustained decode	16K	32	Valid	1,133.2	45.6	625 ms	22.0 ms	32.0	0	33,960	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	112.9	113.0	74 ms	8.8 ms	1.0	0	2,521	22.3 s
Sustained decode	32K	2	Valid	164.9	98.5	103 ms	10.2 ms	2.0	0	4,946	30.0 s
Sustained decode	32K	4	Valid	284.5	73.5	160 ms	13.6 ms	4.0	0	8,534	30.0 s
Sustained decode	32K	8	Valid	456.4	58.9	285 ms	17.0 ms	8.0	0	13,689	30.0 s
Sustained decode	32K	16	Valid	523.4	38.1	585 ms	26.3 ms	16.0	0	15,697	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	69.7	70.1	109 ms	14.3 ms	1.0	0	2,090	30.0 s
Sustained decode	64K	2	Valid	128.6	64.8	147 ms	15.4 ms	2.0	0	3,856	30.0 s
Sustained decode	64K	4	Valid	164.5	45.9	284 ms	21.8 ms	4.0	0	4,936	30.0 s
Sustained decode	64K	8	Valid	204.5	32.8	533 ms	30.5 ms	8.0	0	6,130	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	50.2	49.5	172 ms	20.2 ms	1.0	0	1,506	30.0 s
Sustained decode	123K	2	Valid	86.4	43.3	290 ms	23.1 ms	2.0	0	2,591	30.0 s
Sustained decode	123K	4	Valid	99.9	29.1	456 ms	34.3 ms	4.0	0	2,996	30.0 s
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.