Speed test matrix

Ling-3.0-flash-fp4 on H200 NVL 141GB

sglang · 0.5.6.post3.dev8719+ge1a24a189 · MXFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 19, 2026, 08:49 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev8719+ge1a24a189
Quantization: MXFP4+FP8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: e1a24a189fe4

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-flash-fp4 --served-model-name ling --trust-remote-code --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 262144 --mem-fraction-static 0.877124 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --moe-runner-backend flashinfer_mxfp4

Prefill

prompt tok/s by context length

8K context14.1K581 ms TTFT

16K context15.7K1,036 ms TTFT

32K context15.8K2,047 ms TTFT

64K context15K4,284 ms TTFT

128K context13.4K9,602 ms TTFT

251K context11K22,841 ms TTFT

256K context10.9K23,556 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	151.884 ms TTFT	234.1136 ms TTFT	490.8138 ms TTFT	662.6188 ms TTFT	1,066.1249 ms TTFT	1,375.2399 ms TTFT	1,874.2Requests queued2,370 ms TTFT	1,914.4Requests queued30,142 ms TTFT
16K	151.0103 ms TTFT	73.2166 ms TTFT	206.3180 ms TTFT	637.1252 ms TTFT	1,018.4415 ms TTFT	1,297.4710 ms TTFT	1,668.0Requests queued7,953 ms TTFT	1,799.7Requests queued35,741 ms TTFT
32K	149.9134 ms TTFT	61.8219 ms TTFT	200.6287 ms TTFT	516.1448 ms TTFT	938.9770 ms TTFT	1,210.51,479 ms TTFT	1,265.7Requests queued3,014 ms TTFT	Over capacity
64K	148.3188 ms TTFT	57.8305 ms TTFT	160.9486 ms TTFT	342.2904 ms TTFT	727.71,733 ms TTFT	992.73,139 ms TTFT	Over capacity	Over capacity
128K	144.6285 ms TTFT	59.7489 ms TTFT	154.2835 ms TTFT	433.91,623 ms TTFT	656.33,208 ms TTFT	Over capacity	Over capacity	Over capacity
251K	137.4561 ms TTFT	85.8957 ms TTFT	199.41,650 ms TTFT	461.93,357 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	14.1K	—	581 ms	—	—	1	—	—
Prefill	16K	1	Valid	15.7K	—	1,036 ms	—	—	1	—	—
Prefill	32K	1	Valid	15.8K	—	2,047 ms	—	—	1	—	—
Prefill	64K	1	Valid	15K	—	4,284 ms	—	—	1	—	—
Prefill	128K	1	Valid	13.4K	—	9,602 ms	—	—	1	—	—
Prefill	251K	1	Valid	11K	—	22,841 ms	—	—	1	—	—
Prefill	256K	1	Valid	10.9K	—	23,556 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	151.8	152.4	84 ms	6.6 ms	1.0	1	4,554	30.0 s
Sustained decode	8K	2	Valid	234.1	116.3	136 ms	8.6 ms	2.8	0	7,024	30.0 s
Sustained decode	8K	4	Valid	490.8	125.4	138 ms	8.0 ms	3.9	4	14,724	30.0 s
Sustained decode	8K	8	Valid	662.6	83.1	188 ms	12.0 ms	12.0	0	19,872	30.0 s
Sustained decode	8K	16	Valid	1,066.1	65.2	249 ms	15.3 ms	20.5	0	31,985	30.0 s
Sustained decode	8K	32	Valid	1,375.2	43.4	399 ms	23.1 ms	48.0	0	41,248	30.0 s
Sustained decode	8K	64	Requests queued	1,874.2	33.7	2,370 ms	29.6 ms	70.7	0	56,229	30.0 s
Sustained decode	8K	128	Requests queued	1,914.4	33.5	30,142 ms	29.8 ms	77.0	0	57,420	30.0 s
Sustained decode	16K	1	Valid	151.0	151.6	103 ms	6.6 ms	1.0	1	4,530	30.0 s
Sustained decode	16K	2	Valid	73.2	36.3	166 ms	27.5 ms	60.5	0	2,196	30.0 s
Sustained decode	16K	4	Valid	206.3	51.7	180 ms	19.3 ms	25.0	0	6,188	30.0 s
Sustained decode	16K	8	Valid	637.1	75.3	252 ms	13.3 ms	12.7	0	19,112	30.0 s
Sustained decode	16K	16	Valid	1,018.4	63.2	415 ms	15.8 ms	20.8	0	30,544	30.0 s
Sustained decode	16K	32	Valid	1,297.4	40.9	710 ms	24.5 ms	48.0	0	38,911	30.0 s
Sustained decode	16K	64	Requests queued	1,668.0	32.5	7,953 ms	30.7 ms	73.1	0	50,038	30.0 s
Sustained decode	16K	128	Requests queued	1,799.7	31.3	35,741 ms	32.0 ms	77.0	0	53,998	30.0 s
Sustained decode	32K	1	Valid	149.9	150.6	134 ms	6.6 ms	1.0	1	4,498	30.0 s
Sustained decode	32K	2	Valid	61.8	30.9	219 ms	32.3 ms	66.0	0	1,852	30.0 s
Sustained decode	32K	4	Valid	200.6	48.7	287 ms	20.6 ms	25.5	0	6,016	30.0 s
Sustained decode	32K	8	Valid	516.1	62.1	448 ms	16.1 ms	20.8	0	15,480	30.0 s
Sustained decode	32K	16	Valid	938.9	58.7	770 ms	17.1 ms	23.2	0	28,160	30.0 s
Sustained decode	32K	32	Valid	1,210.5	38.0	1,479 ms	26.3 ms	48.0	0	36,288	30.0 s
Sustained decode	32K	64	Requests queued	1,265.7	29.0	3,014 ms	34.5 ms	77.0	0	37,935	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	148.3	149.2	188 ms	6.7 ms	1.0	1	4,448	30.0 s
Sustained decode	64K	2	Valid	57.8	28.9	305 ms	34.6 ms	66.0	0	1,734	30.0 s
Sustained decode	64K	4	Valid	160.9	38.8	486 ms	25.8 ms	40.0	0	4,824	30.0 s
Sustained decode	64K	8	Valid	342.2	42.8	904 ms	23.4 ms	33.0	0	10,264	30.0 s
Sustained decode	64K	16	Valid	727.7	45.7	1,733 ms	21.9 ms	26.7	0	21,824	30.0 s
Sustained decode	64K	32	Valid	992.7	31.8	3,139 ms	31.5 ms	53.4	0	29,769	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	144.6	145.9	285 ms	6.9 ms	1.0	1	4,337	30.0 s
Sustained decode	128K	2	Valid	59.7	29.9	489 ms	33.4 ms	50.0	0	1,790	30.0 s
Sustained decode	128K	4	Valid	154.2	38.0	835 ms	26.3 ms	38.0	0	4,624	30.0 s
Sustained decode	128K	8	Valid	433.9	52.1	1,623 ms	19.2 ms	22.0	0	13,016	30.0 s
Sustained decode	128K	16	Valid	656.3	41.6	3,208 ms	24.0 ms	25.9	0	19,680	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	137.4	140.1	561 ms	7.1 ms	1.0	1	4,122	30.0 s
Sustained decode	251K	2	Valid	85.8	42.0	957 ms	23.8 ms	21.1	0	2,572	30.0 s
Sustained decode	251K	4	Valid	199.4	48.9	1,650 ms	20.5 ms	16.9	0	5,980	30.0 s
Sustained decode	251K	8	Valid	461.9	58.8	3,357 ms	17.0 ms	12.4	0	13,856	30.0 s
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.