Speed test matrix

Shieldstral-1.0-3B on H200 NVL 141GB

sglang · 0.5.6.post3.dev9335+gdd15fb57b · BF163 contexts (8K–27K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 27K, 32K

by @keennaylmx dev importMeasured Sep 19, 2026, 17:14 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9335+gdd15fb57b
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: dd15fb57b5ef

Run

Served model: shieldstral
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-mistralai-sglang/bin/sglang" serve --model-path mistralai/Shieldstral-1.0-3B --served-model-name shieldstral --tp 1 --context-length 32768 --mem-fraction-static 0.886040 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context51.7K158 ms TTFT

16K context47K346 ms TTFT

27K context40.4K676 ms TTFT

32K context17.6K1,840 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	268.252 ms TTFT	505.866 ms TTFT	867.084 ms TTFT	1,608.5127 ms TTFT	2,418.5193 ms TTFT	2,514.1445 ms TTFT	3,378.6853 ms TTFT	Over capacity
16K	253.169 ms TTFT	102.9100 ms TTFT	723.8118 ms TTFT	1,177.7205 ms TTFT	1,235.7425 ms TTFT	1,972.7824 ms TTFT	Over capacity	Over capacity
27K	235.668 ms TTFT	102.1133 ms TTFT	617.5197 ms TTFT	794.1352 ms TTFT	1,378.2678 ms TTFT	1,238.71,296 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	51.7K	—	158 ms	—	—	1	—	—
Prefill	16K	1	Valid	47K	—	346 ms	—	—	1	—	—
Prefill	27K	1	Valid	40.4K	—	676 ms	—	—	1	—	—
Prefill	32K	1	Valid	17.6K	—	1,840 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	268.2	269.2	52 ms	3.7 ms	1.0	2	8,046	30.0 s
Sustained decode	8K	2	Valid	505.8	254.3	66 ms	3.9 ms	2.0	4	15,173	30.0 s
Sustained decode	8K	4	Valid	867.0	218.3	84 ms	4.6 ms	4.7	4	26,008	30.0 s
Sustained decode	8K	8	Valid	1,608.5	202.6	127 ms	4.9 ms	8.0	8	48,255	30.0 s
Sustained decode	8K	16	Valid	2,418.5	156.4	193 ms	6.4 ms	16.5	16	72,551	30.0 s
Sustained decode	8K	32	Valid	2,514.1	80.3	445 ms	12.4 ms	47.5	0	75,424	30.0 s
Sustained decode	8K	64	Valid	3,378.6	52.7	853 ms	19.0 ms	84.3	0	101,350	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	253.1	254.2	69 ms	3.9 ms	1.0	2	7,594	30.0 s
Sustained decode	16K	2	Valid	102.9	52.0	100 ms	19.2 ms	66.0	0	3,086	30.0 s
Sustained decode	16K	4	Valid	723.8	181.6	118 ms	5.7 ms	7.1	4	21,713	30.0 s
Sustained decode	16K	8	Valid	1,177.7	157.1	205 ms	6.4 ms	9.6	8	35,327	30.0 s
Sustained decode	16K	16	Valid	1,235.7	78.2	425 ms	12.8 ms	24.0	0	37,072	30.0 s
Sustained decode	16K	32	Valid	1,972.7	61.3	824 ms	16.3 ms	41.1	0	59,184	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	27K	1	Valid	235.6	236.4	68 ms	4.2 ms	1.0	1	7,068	30.0 s
Sustained decode	27K	2	Valid	102.1	51.3	133 ms	19.5 ms	34.0	0	3,062	30.0 s
Sustained decode	27K	4	Valid	617.5	163.9	197 ms	6.2 ms	4.8	4	18,525	30.0 s
Sustained decode	27K	8	Valid	794.1	98.8	352 ms	10.1 ms	11.1	0	23,824	30.0 s
Sustained decode	27K	16	Valid	1,378.2	82.9	678 ms	12.1 ms	16.7	0	41,344	30.0 s
Sustained decode	27K	32	Valid	1,238.7	39.1	1,296 ms	25.6 ms	45.1	0	37,152	30.0 s
Sustained decode	27K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	27K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.