Speed test matrix

Ling-3.0-flash on H200 SXM 141GB ×4

sglang · 0.5.6.post3.dev8719+ge1a24a189 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 10, 2026, 19:59 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev8719+ge1a24a189
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: e1a24a189fe4

Run

Served model: ling-3.0-flash
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-flash --served-model-name ling-3.0-flash --trust-remote-code --tp 4 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 262144 --mem-fraction-static 0.835056 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --speculative-algorithm NEXTN

Prefill

prompt tok/s by context length

8K context32.3K254 ms TTFT

16K context18.4K883 ms TTFT

32K context53.7K600 ms TTFT

64K context53.3K1,206 ms TTFT

128K context48.9K2,627 ms TTFT

251K context40.3K6,241 ms TTFT

256K context38.7K6,638 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	386.4134 ms TTFT	726.5130 ms TTFT	1,112.3144 ms TTFT	1,756.4217 ms TTFT	2,651.8276 ms TTFT	3,881.5553 ms TTFT	Not run	Not run
16K	331.7153 ms TTFT	726.3168 ms TTFT	1,148.1158 ms TTFT	1,671.2232 ms TTFT	2,497.9393 ms TTFT	3,768.41,204 ms TTFT	Not run	Not run
32K	406.4190 ms TTFT	688.4200 ms TTFT	1,148.4245 ms TTFT	1,583.0420 ms TTFT	2,466.5695 ms TTFT	3,647.22,138 ms TTFT	Not run	Not run
64K	400.9257 ms TTFT	686.2280 ms TTFT	1,120.8268 ms TTFT	1,584.9779 ms TTFT	2,290.51,730 ms TTFT	3,315.44,737 ms TTFT	Not run	Over capacity
128K	397.0394 ms TTFT	671.0395 ms TTFT	1,005.3503 ms TTFT	1,456.01,522 ms TTFT	2,007.12,956 ms TTFT	2,821.48,658 ms TTFT	Over capacity	Over capacity
251K	356.5708 ms TTFT	574.1705 ms TTFT	916.31,906 ms TTFT	1,258.22,899 ms TTFT	1,599.19,306 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	32.3K	—	254 ms	—	—	1	—	—
Prefill	16K	1	Valid	18.4K	—	883 ms	—	—	1	—	—
Prefill	32K	1	Valid	53.7K	—	600 ms	—	—	1	—	—
Prefill	64K	1	Valid	53.3K	—	1,206 ms	—	—	1	—	—
Prefill	128K	1	Valid	48.9K	—	2,627 ms	—	—	1	—	—
Prefill	251K	1	Valid	40.3K	—	6,241 ms	—	—	1	—	—
Prefill	256K	1	Valid	38.7K	—	6,638 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	386.4	401.8	134 ms	2.5 ms	—	3	11,590	30.0 s
Sustained decode	8K	2	Valid	726.5	368.2	130 ms	2.7 ms	—	6	21,793	30.0 s
Sustained decode	8K	4	Valid	1,112.3	296.2	144 ms	3.4 ms	—	8	33,365	30.0 s
Sustained decode	8K	8	Valid	1,756.4	225.3	217 ms	4.4 ms	—	8	52,688	30.0 s
Sustained decode	8K	16	Valid	2,651.8	170.0	276 ms	5.9 ms	—	16	79,557	30.0 s
Sustained decode	8K	32	Valid	3,881.5	120.4	553 ms	8.3 ms	—	23	116,230	29.9 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	331.7	372.9	153 ms	2.7 ms	—	2	9,951	30.0 s
Sustained decode	16K	2	Valid	726.3	370.3	168 ms	2.7 ms	—	5	21,786	30.0 s
Sustained decode	16K	4	Valid	1,148.1	295.5	158 ms	3.4 ms	—	8	34,444	30.0 s
Sustained decode	16K	8	Valid	1,671.2	212.5	232 ms	4.7 ms	—	8	50,137	30.0 s
Sustained decode	16K	16	Valid	2,497.9	161.0	393 ms	6.2 ms	—	16	74,942	30.0 s
Sustained decode	16K	32	Valid	3,768.4	120.2	1,204 ms	8.3 ms	—	11	112,896	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	406.4	414.7	190 ms	2.4 ms	—	3	12,191	30.0 s
Sustained decode	32K	2	Valid	688.4	354.1	200 ms	2.8 ms	—	4	20,652	30.0 s
Sustained decode	32K	4	Valid	1,148.4	290.1	245 ms	3.4 ms	—	8	34,447	30.0 s
Sustained decode	32K	8	Valid	1,583.0	213.5	420 ms	4.7 ms	—	8	47,478	30.0 s
Sustained decode	32K	16	Valid	2,466.5	159.4	695 ms	6.3 ms	—	16	73,991	30.0 s
Sustained decode	32K	32	Valid	3,647.2	117.5	2,138 ms	8.5 ms	—	2	109,415	30.0 s
Sustained decode	32K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	400.9	411.1	257 ms	2.4 ms	—	3	12,025	30.0 s
Sustained decode	64K	2	Valid	686.2	351.1	280 ms	2.8 ms	—	4	20,585	30.0 s
Sustained decode	64K	4	Valid	1,120.8	290.7	268 ms	3.4 ms	—	8	33,624	30.0 s
Sustained decode	64K	8	Valid	1,584.9	207.1	779 ms	4.8 ms	—	8	47,536	30.0 s
Sustained decode	64K	16	Valid	2,290.5	151.4	1,730 ms	6.6 ms	—	16	68,708	30.0 s
Sustained decode	64K	32	Valid	3,315.4	110.2	4,737 ms	9.1 ms	—	0	99,414	30.0 s
Sustained decode	64K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	397.0	414.7	394 ms	2.4 ms	—	3	11,909	30.0 s
Sustained decode	128K	2	Valid	671.0	355.3	395 ms	2.8 ms	—	4	20,129	30.0 s
Sustained decode	128K	4	Valid	1,005.3	270.8	503 ms	3.7 ms	—	8	30,160	30.0 s
Sustained decode	128K	8	Valid	1,456.0	192.5	1,522 ms	5.2 ms	—	8	43,671	30.0 s
Sustained decode	128K	16	Valid	2,007.1	135.2	2,956 ms	7.4 ms	—	16	60,207	30.0 s
Sustained decode	128K	32	Valid	2,821.4	98.8	8,658 ms	10.1 ms	—	0	84,590	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	356.5	373.2	708 ms	2.7 ms	—	2	10,694	30.0 s
Sustained decode	251K	2	Valid	574.1	311.5	705 ms	3.2 ms	—	4	17,219	30.0 s
Sustained decode	251K	4	Valid	916.3	247.1	1,906 ms	4.0 ms	—	5	27,483	30.0 s
Sustained decode	251K	8	Valid	1,258.2	171.9	2,899 ms	5.8 ms	—	8	37,741	30.0 s
Sustained decode	251K	16	Valid	1,599.1	117.4	9,306 ms	8.5 ms	—	5	47,809	29.9 s
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.