Speed test matrix

Mistral-Small-4-119B-2603 on H200 NVL 141GB ×2

sglang · 0.5.6.post3.dev9335+gdd15fb57b · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 21, 2026, 02:42 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9335+gdd15fb57b
Quantization: FP8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: dd15fb57b5ef

Run

Served model: mistral
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-mistralai-sglang/bin/sglang" serve --model-path mistralai/Mistral-Small-4-119B-2603 --served-model-name mistral --trust-remote-code --tp 2 --reasoning-parser mistral --tool-call-parser mistral --context-length 262144 --mem-fraction-static 0.876739 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --speculative-algorithm EAGLE --speculative-draft-model-path mistralai/Mistral-Small-4-119B-2603-eagle --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4

Prefill

prompt tok/s by context length

8K context36.5K225 ms TTFT

16K context40.8K399 ms TTFT

32K context30.4K1,064 ms TTFT

64K context24.4K2,647 ms TTFT

128K context18.7K6,888 ms TTFT

251K context11.3K22,267 ms TTFT

256K context11.3K22,836 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	268.5100 ms TTFT	451.1140 ms TTFT	687.4168 ms TTFT	1,180.9155 ms TTFT	1,880.0257 ms TTFT	3,139.4528 ms TTFT	Not run	Not run
16K	245.8120 ms TTFT	400.7146 ms TTFT	711.7177 ms TTFT	1,070.9247 ms TTFT	1,775.2471 ms TTFT	2,636.5911 ms TTFT	Not run	Not run
32K	244.6141 ms TTFT	405.7199 ms TTFT	649.0249 ms TTFT	1,022.0482 ms TTFT	1,467.2839 ms TTFT	2,160.41,580 ms TTFT	Not run	Over capacity
64K	211.2197 ms TTFT	393.2215 ms TTFT	558.4320 ms TTFT	861.5763 ms TTFT	1,180.91,463 ms TTFT	1,564.92,907 ms TTFT	Over capacity	Over capacity
128K	185.8315 ms TTFT	308.2365 ms TTFT	443.0758 ms TTFT	593.41,456 ms TTFT	793.82,855 ms TTFT	Over capacity	Over capacity	Over capacity
251K	126.0597 ms TTFT	214.0855 ms TTFT	286.01,459 ms TTFT	357.92,852 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	36.5K	—	225 ms	—	—	1	—	—
Prefill	16K	1	Valid	40.8K	—	399 ms	—	—	1	—	—
Prefill	32K	1	Valid	30.4K	—	1,064 ms	—	—	1	—	—
Prefill	64K	1	Valid	24.4K	—	2,647 ms	—	—	1	—	—
Prefill	128K	1	Valid	18.7K	—	6,888 ms	—	—	1	—	—
Prefill	251K	1	Valid	11.3K	—	22,267 ms	—	—	1	—	—
Prefill	256K	1	Valid	11.3K	—	22,836 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	268.5	273.7	100 ms	3.7 ms	1.0	2	8,044	30.0 s
Sustained decode	8K	2	Valid	451.1	222.8	140 ms	4.5 ms	2.0	2	13,532	30.0 s
Sustained decode	8K	4	Valid	687.4	168.4	168 ms	5.9 ms	3.9	4	20,614	30.0 s
Sustained decode	8K	8	Valid	1,180.9	149.9	155 ms	6.7 ms	7.9	8	35,426	30.0 s
Sustained decode	8K	16	Valid	1,880.0	118.9	257 ms	8.4 ms	16.0	8	56,283	29.9 s
Sustained decode	8K	32	Valid	3,139.4	98.1	528 ms	10.2 ms	32.0	1	94,149	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	245.8	256.8	120 ms	3.9 ms	1.0	2	7,376	30.0 s
Sustained decode	16K	2	Valid	400.7	200.1	146 ms	5.0 ms	2.0	2	12,028	30.0 s
Sustained decode	16K	4	Valid	711.7	184.2	177 ms	5.4 ms	4.0	4	21,345	30.0 s
Sustained decode	16K	8	Valid	1,070.9	133.1	247 ms	7.5 ms	7.9	8	32,091	30.0 s
Sustained decode	16K	16	Valid	1,775.2	111.4	471 ms	9.0 ms	16.0	0	53,263	30.0 s
Sustained decode	16K	32	Valid	2,636.5	83.0	911 ms	12.0 ms	32.0	0	79,053	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	244.6	244.0	141 ms	4.1 ms	1.0	1	7,328	30.0 s
Sustained decode	32K	2	Valid	405.7	203.9	199 ms	4.9 ms	1.9	2	12,168	30.0 s
Sustained decode	32K	4	Valid	649.0	160.8	249 ms	6.2 ms	3.9	4	19,452	30.0 s
Sustained decode	32K	8	Valid	1,022.0	123.2	482 ms	8.1 ms	7.9	7	30,660	30.0 s
Sustained decode	32K	16	Valid	1,467.2	91.9	839 ms	10.9 ms	16.0	0	43,977	30.0 s
Sustained decode	32K	32	Valid	2,160.4	67.1	1,580 ms	14.9 ms	32.0	0	64,775	30.0 s
Sustained decode	32K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	211.2	212.1	197 ms	4.8 ms	1.0	1	6,337	30.0 s
Sustained decode	64K	2	Valid	393.2	191.9	215 ms	5.2 ms	1.9	2	11,784	30.0 s
Sustained decode	64K	4	Valid	558.4	138.3	320 ms	7.2 ms	4.0	4	16,751	30.0 s
Sustained decode	64K	8	Valid	861.5	106.5	763 ms	9.4 ms	8.0	0	25,818	30.0 s
Sustained decode	64K	16	Valid	1,180.9	73.7	1,463 ms	13.6 ms	16.0	0	35,381	30.0 s
Sustained decode	64K	32	Valid	1,564.9	49.8	2,907 ms	20.1 ms	32.0	0	46,926	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	185.8	186.8	315 ms	5.4 ms	1.0	1	5,575	30.0 s
Sustained decode	128K	2	Valid	308.2	152.4	365 ms	6.6 ms	2.0	2	9,235	30.0 s
Sustained decode	128K	4	Valid	443.0	111.4	758 ms	9.0 ms	4.0	0	13,288	30.0 s
Sustained decode	128K	8	Valid	593.4	76.6	1,456 ms	13.1 ms	8.0	0	17,789	30.0 s
Sustained decode	128K	16	Valid	793.8	51.9	2,855 ms	19.3 ms	16.0	0	23,778	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	126.0	133.7	597 ms	7.5 ms	1.0	1	3,779	30.0 s
Sustained decode	251K	2	Valid	214.0	107.2	855 ms	9.4 ms	2.0	0	6,422	30.0 s
Sustained decode	251K	4	Valid	286.0	71.2	1,459 ms	14.0 ms	4.0	0	8,571	30.0 s
Sustained decode	251K	8	Valid	357.9	44.4	2,852 ms	22.5 ms	8.0	0	10,735	30.0 s
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.