Speed test matrix

gpt-oss-20b on RTX PRO 6000 Blackwell Server Edition 96GB

vllm · 0.30.1rc1.dev452+g73a583112 · MXFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Oct 10, 2026, 01:56 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev452+g73a583112
Quantization: MXFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 73a5831127a9

Run

Served model: gpt-oss
Revision: main
Benchmark tool: llm-inference-bench v0.7.6
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-openai-vllm-73a5831127a9/bin/vllm" serve openai/gpt-oss-20b --served-model-name gpt-oss --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser openai --max-model-len 131072 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER --moe-backend marlin

Prefill

prompt tok/s by context length

8K context36.3K225 ms TTFT

16K context34.4K470 ms TTFT

32K context30K1,072 ms TTFT

64K context23.5K2,728 ms TTFT

123K context16.7K7,376 ms TTFT

128K context16.3K7,852 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	248.033 ms TTFT	377.542 ms TTFT	642.561 ms TTFT	1,062.594 ms TTFT	1,565.1186 ms TTFT	2,654.4362 ms TTFT	4,496.6720 ms TTFT	6,781.71,053 ms TTFT
16K	239.839 ms TTFT	371.256 ms TTFT	607.380 ms TTFT	979.1167 ms TTFT	1,540.5321 ms TTFT	2,194.8606 ms TTFT	4,002.71,220 ms TTFT	Over capacity
32K	225.049 ms TTFT	352.487 ms TTFT	574.4141 ms TTFT	919.9280 ms TTFT	1,252.0625 ms TTFT	2,127.41,165 ms TTFT	2,994.51,826 ms TTFT	Over capacity
64K	200.987 ms TTFT	344.2159 ms TTFT	557.4255 ms TTFT	698.6532 ms TTFT	1,114.31,077 ms TTFT	1,324.32,267 ms TTFT	Over capacity	Over capacity
123K	178.0146 ms TTFT	268.7195 ms TTFT	494.4510 ms TTFT	665.5799 ms TTFT	780.91,890 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	36.3K	—	225 ms	—	—	1	—	—
Prefill	16K	1	Valid	34.4K	—	470 ms	—	—	1	—	—
Prefill	32K	1	Valid	30K	—	1,072 ms	—	—	1	—	—
Prefill	64K	1	Valid	23.5K	—	2,728 ms	—	—	1	—	—
Prefill	123K	1	Valid	16.7K	—	7,376 ms	—	—	1	—	—
Prefill	128K	1	Valid	16.3K	—	7,852 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	248.0	248.8	33 ms	4.0 ms	1.0	2	7,441	30.0 s
Sustained decode	8K	2	Valid	377.5	194.5	42 ms	5.1 ms	2.0	2	11,275	29.9 s
Sustained decode	8K	4	Valid	642.5	161.2	61 ms	6.2 ms	4.0	4	19,271	30.0 s
Sustained decode	8K	8	Valid	1,062.5	141.6	94 ms	7.1 ms	8.0	8	31,867	30.0 s
Sustained decode	8K	16	Valid	1,565.1	106.1	186 ms	9.4 ms	16.0	0	46,944	30.0 s
Sustained decode	8K	32	Valid	2,654.4	89.0	362 ms	11.2 ms	32.0	0	79,633	30.0 s
Sustained decode	8K	64	Valid	4,496.6	77.3	720 ms	12.9 ms	64.0	0	134,899	30.0 s
Sustained decode	8K	128	Valid	6,781.7	61.8	1,053 ms	16.2 ms	128.0	0	203,453	30.0 s
Sustained decode	16K	1	Valid	239.8	240.1	39 ms	4.2 ms	1.0	1	7,184	30.0 s
Sustained decode	16K	2	Valid	371.2	190.1	56 ms	5.3 ms	2.0	2	11,136	30.0 s
Sustained decode	16K	4	Valid	607.3	158.7	80 ms	6.3 ms	4.0	4	18,218	30.0 s
Sustained decode	16K	8	Valid	979.1	148.5	167 ms	6.8 ms	7.7	8	29,370	30.0 s
Sustained decode	16K	16	Valid	1,540.5	98.5	321 ms	10.2 ms	16.0	0	46,210	30.0 s
Sustained decode	16K	32	Valid	2,194.8	81.8	606 ms	12.2 ms	32.0	0	65,845	30.0 s
Sustained decode	16K	64	Valid	4,002.7	70.1	1,220 ms	14.3 ms	64.0	0	120,083	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	225.0	225.6	49 ms	4.4 ms	1.0	1	6,691	29.7 s
Sustained decode	32K	2	Valid	352.4	177.6	87 ms	5.6 ms	2.0	2	10,571	30.0 s
Sustained decode	32K	4	Valid	574.4	154.2	141 ms	6.5 ms	4.0	4	17,232	30.0 s
Sustained decode	32K	8	Valid	919.9	113.5	280 ms	8.8 ms	8.0	0	27,590	30.0 s
Sustained decode	32K	16	Valid	1,252.0	89.4	625 ms	11.2 ms	16.0	0	37,554	30.0 s
Sustained decode	32K	32	Valid	2,127.4	70.4	1,165 ms	14.2 ms	32.0	0	63,797	30.0 s
Sustained decode	32K	64	Valid	2,994.5	57.5	1,826 ms	17.4 ms	64.0	0	89,825	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	200.9	201.8	87 ms	5.0 ms	1.0	1	5,878	29.3 s
Sustained decode	64K	2	Valid	344.2	163.0	159 ms	6.1 ms	2.0	2	10,326	30.0 s
Sustained decode	64K	4	Valid	557.4	139.5	255 ms	7.2 ms	4.0	4	16,720	30.0 s
Sustained decode	64K	8	Valid	698.6	100.9	532 ms	9.9 ms	8.0	0	20,956	30.0 s
Sustained decode	64K	16	Valid	1,114.3	76.2	1,077 ms	13.1 ms	16.0	0	33,430	30.0 s
Sustained decode	64K	32	Valid	1,324.3	54.1	2,267 ms	18.5 ms	32.0	0	39,729	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	178.0	169.9	146 ms	5.9 ms	1.0	1	4,037	22.7 s
Sustained decode	123K	2	Valid	268.7	141.7	195 ms	7.1 ms	2.0	2	8,061	30.0 s
Sustained decode	123K	4	Valid	494.4	118.3	510 ms	8.5 ms	4.0	0	14,828	30.0 s
Sustained decode	123K	8	Valid	665.5	84.1	799 ms	11.9 ms	8.0	0	19,963	30.0 s
Sustained decode	123K	16	Valid	780.9	58.8	1,890 ms	17.0 ms	16.0	0	23,422	30.0 s
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.