Speed test matrix

Ministral-3-3B-Instruct-2512 on H200 NVL 141GB

sglang · 0.5.6.post3.dev9335+gdd15fb57b · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 19, 2026, 14:51 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9335+gdd15fb57b
Quantization: FP8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: dd15fb57b5ef

Run

Served model: mistral
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-mistralai-sglang/bin/sglang" serve --model-path mistralai/Ministral-3-3B-Instruct-2512 --served-model-name mistral --trust-remote-code --tp 1 --tool-call-parser mistral --context-length 262144 --mem-fraction-static 0.859148 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --load-format mistral --model-config-parser mistral

Prefill

prompt tok/s by context length

8K context62.1K132 ms TTFT

16K context57.4K283 ms TTFT

32K context44.1K733 ms TTFT

64K context28.6K2,253 ms TTFT

128K context16.5K7,795 ms TTFT

251K context9,09327,775 ms TTFT

256K context9,01628,569 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	376.850 ms TTFT	688.566 ms TTFT	1,236.877 ms TTFT	2,007.2125 ms TTFT	2,261.9205 ms TTFT	2,812.0453 ms TTFT	4,140.9904 ms TTFT	Over capacity
16K	347.471 ms TTFT	139.496 ms TTFT	1,057.090 ms TTFT	1,333.0175 ms TTFT	1,617.0426 ms TTFT	2,617.2821 ms TTFT	Over capacity	Over capacity
32K	302.997 ms TTFT	269.5136 ms TTFT	557.1192 ms TTFT	722.5408 ms TTFT	1,252.9782 ms TTFT	Over capacity	Over capacity	Over capacity
64K	242.5135 ms TTFT	168.7242 ms TTFT	503.4399 ms TTFT	417.1765 ms TTFT	447.81,480 ms TTFT	Over capacity	Over capacity	Over capacity
128K	173.1225 ms TTFT	54.1428 ms TTFT	109.8768 ms TTFT	148.21,467 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	111.8406 ms TTFT	28.4785 ms TTFT	58.11,450 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	62.1K	—	132 ms	—	—	1	—	—
Prefill	16K	1	Valid	57.4K	—	283 ms	—	—	1	—	—
Prefill	32K	1	Valid	44.1K	—	733 ms	—	—	1	—	—
Prefill	64K	1	Valid	28.6K	—	2,253 ms	—	—	1	—	—
Prefill	128K	1	Valid	16.5K	—	7,795 ms	—	—	1	—	—
Prefill	251K	1	Valid	9,093.0	—	27,775 ms	—	—	1	—	—
Prefill	256K	1	Valid	9,016.0	—	28,569 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	376.8	378.9	50 ms	2.6 ms	1.0	3	11,303	30.0 s
Sustained decode	8K	2	Valid	688.5	346.7	66 ms	2.9 ms	2.0	4	20,655	30.0 s
Sustained decode	8K	4	Valid	1,236.8	311.1	77 ms	3.2 ms	4.0	8	37,101	30.0 s
Sustained decode	8K	8	Valid	2,007.2	253.8	125 ms	3.9 ms	8.0	16	60,217	30.0 s
Sustained decode	8K	16	Valid	2,261.9	171.2	205 ms	6.1 ms	21.3	16	67,861	30.0 s
Sustained decode	8K	32	Valid	2,812.0	89.8	453 ms	11.1 ms	46.4	0	84,352	30.0 s
Sustained decode	8K	64	Valid	4,140.9	63.7	904 ms	15.7 ms	75.7	0	124,217	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	347.4	349.1	71 ms	2.9 ms	1.0	2	10,422	30.0 s
Sustained decode	16K	2	Valid	139.4	68.1	96 ms	14.7 ms	61.7	0	4,181	30.0 s
Sustained decode	16K	4	Valid	1,057.0	267.0	90 ms	3.7 ms	4.1	8	31,707	30.0 s
Sustained decode	16K	8	Valid	1,333.0	181.0	175 ms	5.7 ms	10.0	8	39,988	30.0 s
Sustained decode	16K	16	Valid	1,617.0	100.1	426 ms	10.0 ms	21.1	0	48,512	30.0 s
Sustained decode	16K	32	Valid	2,617.2	79.6	821 ms	12.6 ms	33.0	0	78,495	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	302.9	305.0	97 ms	3.3 ms	1.0	2	9,087	30.0 s
Sustained decode	32K	2	Valid	269.5	193.9	136 ms	6.0 ms	21.2	2	8,086	30.0 s
Sustained decode	32K	4	Valid	557.1	171.1	192 ms	6.2 ms	5.5	4	16,712	30.0 s
Sustained decode	32K	8	Valid	722.5	90.8	408 ms	11.0 ms	11.7	0	21,672	30.0 s
Sustained decode	32K	16	Valid	1,252.9	75.6	782 ms	13.2 ms	18.1	0	37,584	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	242.5	243.8	135 ms	4.1 ms	1.0	1	7,274	30.0 s
Sustained decode	64K	2	Valid	168.7	80.9	242 ms	12.4 ms	14.3	0	5,062	30.0 s
Sustained decode	64K	4	Valid	503.4	119.7	399 ms	8.4 ms	4.5	1	15,059	29.9 s
Sustained decode	64K	8	Valid	417.1	52.9	765 ms	18.9 ms	9.0	0	12,504	30.0 s
Sustained decode	64K	16	Valid	447.8	28.5	1,480 ms	35.1 ms	25.0	0	13,423	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	173.1	174.8	225 ms	5.7 ms	1.0	1	5,191	30.0 s
Sustained decode	128K	2	Valid	54.1	27.1	428 ms	37.0 ms	26.4	0	1,622	30.0 s
Sustained decode	128K	4	Valid	109.8	26.6	768 ms	37.6 ms	22.3	0	3,292	30.0 s
Sustained decode	128K	8	Valid	148.2	18.8	1,467 ms	53.2 ms	30.0	0	4,440	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	111.8	112.0	406 ms	8.9 ms	1.0	0	3,354	30.0 s
Sustained decode	251K	2	Valid	28.4	14.2	785 ms	70.2 ms	28.5	0	852	30.0 s
Sustained decode	251K	4	Valid	58.1	14.7	1,450 ms	68.0 ms	20.0	0	1,740	29.9 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.