Speed test matrix

Ministral-3-3B-Base-2512 on H200 NVL 141GB

sglang · 0.5.6.post3.dev9335+gdd15fb57b · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 19, 2026, 14:28 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9335+gdd15fb57b
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: dd15fb57b5ef

Run

Served model: mistral
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-mistralai-sglang/bin/sglang" serve --model-path mistralai/Ministral-3-3B-Base-2512 --served-model-name mistral --trust-remote-code --tp 1 --context-length 262144 --mem-fraction-static 0.860748 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context15.6K527 ms TTFT

16K context47.5K342 ms TTFT

32K context37.9K853 ms TTFT

64K context24.7K2,610 ms TTFT

128K context15.6K8,268 ms TTFT

251K context8,82628,614 ms TTFT

256K context8,76129,403 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	264.952 ms TTFT	507.367 ms TTFT	933.990 ms TTFT	1,612.4126 ms TTFT	2,456.3204 ms TTFT	2,526.0464 ms TTFT	3,375.0873 ms TTFT	Over capacity
16K	249.772 ms TTFT	102.798 ms TTFT	720.1118 ms TTFT	1,170.2196 ms TTFT	1,233.5420 ms TTFT	1,957.2822 ms TTFT	Over capacity	Over capacity
32K	226.092 ms TTFT	101.9160 ms TTFT	549.5222 ms TTFT	645.2410 ms TTFT	959.0785 ms TTFT	Over capacity	Over capacity	Over capacity
64K	189.9141 ms TTFT	93.5236 ms TTFT	266.2405 ms TTFT	385.7771 ms TTFT	436.01,481 ms TTFT	Over capacity	Over capacity	Over capacity
128K	144.0225 ms TTFT	51.1449 ms TTFT	91.5759 ms TTFT	140.51,491 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	99.1398 ms TTFT	26.6800 ms TTFT	52.31,437 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	15.6K	—	527 ms	—	—	1	—	—
Prefill	16K	1	Valid	47.5K	—	342 ms	—	—	1	—	—
Prefill	32K	1	Valid	37.9K	—	853 ms	—	—	1	—	—
Prefill	64K	1	Valid	24.7K	—	2,610 ms	—	—	1	—	—
Prefill	128K	1	Valid	15.6K	—	8,268 ms	—	—	1	—	—
Prefill	251K	1	Valid	8,826.0	—	28,614 ms	—	—	1	—	—
Prefill	256K	1	Valid	8,761.0	—	29,403 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	264.9	265.9	52 ms	3.8 ms	1.0	2	7,948	30.0 s
Sustained decode	8K	2	Valid	507.3	235.5	67 ms	4.3 ms	2.0	2	15,220	30.0 s
Sustained decode	8K	4	Valid	933.9	235.0	90 ms	4.3 ms	4.0	4	28,015	30.0 s
Sustained decode	8K	8	Valid	1,612.4	204.3	126 ms	4.9 ms	8.0	8	48,368	30.0 s
Sustained decode	8K	16	Valid	2,456.3	158.0	204 ms	6.4 ms	16.5	16	73,684	30.0 s
Sustained decode	8K	32	Valid	2,526.0	80.6	464 ms	12.4 ms	47.0	0	75,776	30.0 s
Sustained decode	8K	64	Valid	3,375.0	52.6	873 ms	19.0 ms	84.3	0	101,248	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	249.7	251.2	72 ms	4.0 ms	1.0	2	7,490	30.0 s
Sustained decode	16K	2	Valid	102.7	51.9	98 ms	19.3 ms	66.0	0	3,078	30.0 s
Sustained decode	16K	4	Valid	720.1	181.2	118 ms	5.7 ms	7.1	4	21,601	30.0 s
Sustained decode	16K	8	Valid	1,170.2	156.9	196 ms	6.5 ms	9.7	8	35,103	30.0 s
Sustained decode	16K	16	Valid	1,233.5	78.1	420 ms	12.8 ms	24.0	0	36,992	30.0 s
Sustained decode	16K	32	Valid	1,957.2	60.9	822 ms	16.4 ms	41.6	0	58,719	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	226.0	227.0	92 ms	4.4 ms	1.0	1	6,779	30.0 s
Sustained decode	32K	2	Valid	101.9	51.3	160 ms	19.5 ms	34.0	0	3,056	30.0 s
Sustained decode	32K	4	Valid	549.5	151.8	222 ms	6.7 ms	5.0	4	16,485	30.0 s
Sustained decode	32K	8	Valid	645.2	81.3	410 ms	12.3 ms	12.0	0	19,352	30.0 s
Sustained decode	32K	16	Valid	959.0	58.9	785 ms	17.0 ms	20.9	0	28,770	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	189.9	191.1	141 ms	5.2 ms	1.0	1	5,696	30.0 s
Sustained decode	64K	2	Valid	93.5	46.9	236 ms	21.3 ms	18.0	0	2,806	30.0 s
Sustained decode	64K	4	Valid	266.2	63.7	405 ms	15.7 ms	6.5	0	7,984	30.0 s
Sustained decode	64K	8	Valid	385.7	48.6	771 ms	20.6 ms	12.0	0	11,568	30.0 s
Sustained decode	64K	16	Valid	436.0	27.8	1,481 ms	36.0 ms	24.0	0	13,072	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	144.0	145.5	225 ms	6.9 ms	1.0	1	4,321	30.0 s
Sustained decode	128K	2	Valid	51.1	25.6	449 ms	39.0 ms	26.0	0	1,532	30.0 s
Sustained decode	128K	4	Valid	91.5	22.5	759 ms	44.5 ms	25.4	0	2,744	30.0 s
Sustained decode	128K	8	Valid	140.5	17.8	1,491 ms	56.1 ms	30.0	0	4,208	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	99.1	99.1	398 ms	10.1 ms	1.0	0	2,972	30.0 s
Sustained decode	251K	2	Valid	26.6	13.4	800 ms	74.9 ms	32.0	0	796	29.9 s
Sustained decode	251K	4	Valid	52.3	13.0	1,437 ms	77.0 ms	25.0	0	1,568	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.