Speed test matrix

Ministral-3-14B-Instruct-2512 on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 4, 2026, 08:23 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: mistral
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Ministral-3-14B-Instruct-2512 --served-model-name mistral --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.872342 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer_mode mistral --config_format mistral --load_format mistral

Prefill

prompt tok/s by context length

8K context18K455 ms TTFT

16K context20.2K807 ms TTFT

32K context18.4K1,763 ms TTFT

64K context14.2K4,562 ms TTFT

128K context9,23613,979 ms TTFT

251K context5,39246,928 ms TTFT

256K context5,34148,318 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	168.7193 ms TTFT	326.2308 ms TTFT	606.0390 ms TTFT	1,038.6639 ms TTFT	1,126.91,712 ms TTFT	2,079.14,446 ms TTFT	Over capacity	Over capacity
16K	159.2210 ms TTFT	301.9315 ms TTFT	538.1438 ms TTFT	674.01,199 ms TTFT	1,174.11,953 ms TTFT	1,527.54,006 ms TTFT	Over capacity	Over capacity
32K	143.9234 ms TTFT	261.0350 ms TTFT	330.2578 ms TTFT	643.41,241 ms TTFT	855.62,269 ms TTFT	Over capacity	Over capacity	Over capacity
64K	122.2232 ms TTFT	150.1359 ms TTFT	317.1615 ms TTFT	427.41,318 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	92.8336 ms TTFT	151.4473 ms TTFT	207.3779 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	63.4446 ms TTFT	96.7627 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	18K	—	455 ms	—	—	1	—	—
Prefill	16K	1	Valid	20.2K	—	807 ms	—	—	1	—	—
Prefill	32K	1	Valid	18.4K	—	1,763 ms	—	—	1	—	—
Prefill	64K	1	Valid	14.2K	—	4,562 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,236.0	—	13,979 ms	—	—	1	—	—
Prefill	251K	1	Valid	5,392.0	—	46,928 ms	—	—	1	—	—
Prefill	256K	1	Valid	5,341.0	—	48,318 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	168.7	170.4	193 ms	5.9 ms	1.0	1	5,060	30.0 s
Sustained decode	8K	2	Valid	326.2	165.8	308 ms	6.0 ms	2.0	2	9,784	30.0 s
Sustained decode	8K	4	Valid	606.0	155.0	390 ms	6.5 ms	4.0	4	18,177	30.0 s
Sustained decode	8K	8	Valid	1,038.6	135.5	639 ms	7.4 ms	7.9	8	31,155	30.0 s
Sustained decode	8K	16	Valid	1,126.9	104.8	1,712 ms	9.5 ms	16.0	0	33,795	30.0 s
Sustained decode	8K	32	Valid	2,079.1	69.0	4,446 ms	14.5 ms	32.0	0	62,370	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	159.2	161.0	210 ms	6.2 ms	1.0	1	4,776	30.0 s
Sustained decode	16K	2	Valid	301.9	153.7	315 ms	6.5 ms	2.0	2	9,056	30.0 s
Sustained decode	16K	4	Valid	538.1	138.1	438 ms	7.2 ms	4.0	4	16,140	30.0 s
Sustained decode	16K	8	Valid	674.0	110.3	1,199 ms	9.1 ms	8.0	0	20,218	30.0 s
Sustained decode	16K	16	Valid	1,174.1	79.8	1,953 ms	12.5 ms	16.0	0	35,220	30.0 s
Sustained decode	16K	32	Valid	1,527.5	49.4	4,006 ms	20.2 ms	32.0	0	45,830	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	143.9	145.8	234 ms	6.9 ms	1.0	1	4,318	30.0 s
Sustained decode	32K	2	Valid	261.0	133.4	350 ms	7.5 ms	2.0	2	7,834	30.0 s
Sustained decode	32K	4	Valid	330.2	111.7	578 ms	9.0 ms	4.0	0	9,174	27.8 s
Sustained decode	32K	8	Valid	643.4	82.0	1,241 ms	12.2 ms	8.0	0	19,303	30.0 s
Sustained decode	32K	16	Valid	855.6	54.4	2,269 ms	18.4 ms	16.0	0	25,657	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	122.2	122.3	232 ms	8.2 ms	1.0	0	2,657	21.7 s
Sustained decode	64K	2	Valid	150.1	106.7	359 ms	9.4 ms	2.0	0	4,031	26.8 s
Sustained decode	64K	4	Valid	317.1	82.2	615 ms	12.2 ms	4.0	0	9,512	30.0 s
Sustained decode	64K	8	Valid	427.4	53.7	1,318 ms	18.6 ms	8.0	0	12,816	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	92.8	92.9	336 ms	10.8 ms	1.0	0	2,785	30.0 s
Sustained decode	128K	2	Valid	151.4	75.8	473 ms	13.2 ms	2.0	0	4,544	30.0 s
Sustained decode	128K	4	Valid	207.3	51.9	779 ms	19.3 ms	4.0	0	6,220	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	63.4	63.4	446 ms	15.8 ms	1.0	0	1,903	30.0 s
Sustained decode	251K	2	Valid	96.7	48.4	627 ms	20.7 ms	2.0	0	2,902	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.