Speed test matrix

Ling-3.0-tiny-base-midtrain on H200 NVL 141GB

sglang · 0.5.6.post3.dev8719+ge1a24a189 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 19, 2026, 10:15 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev8719+ge1a24a189
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: e1a24a189fe4

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny-base-midtrain --served-model-name ling --trust-remote-code --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 262144 --mem-fraction-static 0.869695 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context64K128 ms TTFT

16K context38K426 ms TTFT

32K context78.4K411 ms TTFT

64K context71K907 ms TTFT

128K context59K2,180 ms TTFT

251K context42.2K5,978 ms TTFT

256K context41.5K6,200 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	403.663 ms TTFT	754.874 ms TTFT	1,240.388 ms TTFT	1,983.5143 ms TTFT	2,989.4220 ms TTFT	5,062.8392 ms TTFT	7,260.1759 ms TTFT	12.6K1,332 ms TTFT
16K	399.177 ms TTFT	712.1110 ms TTFT	1,257.499 ms TTFT	1,923.4232 ms TTFT	2,984.9405 ms TTFT	4,546.3733 ms TTFT	5,973.01,348 ms TTFT	10.4K2,652 ms TTFT
32K	391.7108 ms TTFT	519.4150 ms TTFT	1,215.6150 ms TTFT	1,892.4380 ms TTFT	2,799.6551 ms TTFT	3,888.31,478 ms TTFT	6,491.92,985 ms TTFT	8,981.25,715 ms TTFT
64K	379.8159 ms TTFT	326.3257 ms TTFT	1,099.4263 ms TTFT	1,621.4681 ms TTFT	2,478.01,285 ms TTFT	2,391.73,221 ms TTFT	4,324.16,372 ms TTFT	Over capacity
128K	362.3266 ms TTFT	238.5459 ms TTFT	1,017.0437 ms TTFT	1,342.71,120 ms TTFT	1,874.23,207 ms TTFT	1,878.76,348 ms TTFT	2,161.812,624 ms TTFT	Over capacity
251K	329.5510 ms TTFT	69.8891 ms TTFT	141.71,638 ms TTFT	980.83,247 ms TTFT	1,275.96,161 ms TTFT	1,602.112,879 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	64K	—	128 ms	—	—	1	—	—
Prefill	16K	1	Valid	38K	—	426 ms	—	—	1	—	—
Prefill	32K	1	Valid	78.4K	—	411 ms	—	—	1	—	—
Prefill	64K	1	Valid	71K	—	907 ms	—	—	1	—	—
Prefill	128K	1	Valid	59K	—	2,180 ms	—	—	1	—	—
Prefill	251K	1	Valid	42.2K	—	5,978 ms	—	—	1	—	—
Prefill	256K	1	Valid	41.5K	—	6,200 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	403.6	405.9	63 ms	2.5 ms	1.0	3	12,107	30.0 s
Sustained decode	8K	2	Valid	754.8	380.1	74 ms	2.6 ms	2.0	6	22,644	30.0 s
Sustained decode	8K	4	Valid	1,240.3	322.3	88 ms	3.1 ms	4.3	8	37,210	30.0 s
Sustained decode	8K	8	Valid	1,983.5	252.3	143 ms	4.0 ms	8.1	16	59,504	30.0 s
Sustained decode	8K	16	Valid	2,989.4	189.9	220 ms	5.3 ms	19.7	16	89,680	30.0 s
Sustained decode	8K	32	Valid	5,062.8	159.3	392 ms	6.3 ms	34.1	32	151,888	30.0 s
Sustained decode	8K	64	Valid	7,260.1	113.0	759 ms	8.8 ms	84.3	0	217,808	30.0 s
Sustained decode	8K	128	Valid	12.6K	98.4	1,332 ms	10.2 ms	128.0	0	377,854	30.0 s
Sustained decode	16K	1	Valid	399.1	402.0	77 ms	2.5 ms	1.0	3	11,974	30.0 s
Sustained decode	16K	2	Valid	712.1	375.0	110 ms	2.7 ms	10.8	4	21,362	30.0 s
Sustained decode	16K	4	Valid	1,257.4	317.1	99 ms	3.2 ms	4.0	8	37,749	30.0 s
Sustained decode	16K	8	Valid	1,923.4	239.2	232 ms	4.2 ms	7.7	8	57,697	30.0 s
Sustained decode	16K	16	Valid	2,984.9	189.7	405 ms	5.3 ms	15.5	16	89,539	30.0 s
Sustained decode	16K	32	Valid	4,546.3	150.3	733 ms	6.7 ms	35.8	32	136,386	30.0 s
Sustained decode	16K	64	Valid	5,973.0	93.6	1,348 ms	10.7 ms	94.9	0	179,199	30.0 s
Sustained decode	16K	128	Valid	10.4K	79.2	2,652 ms	12.6 ms	142.9	0	312,832	30.0 s
Sustained decode	32K	1	Valid	391.7	396.3	108 ms	2.5 ms	1.0	3	11,752	30.0 s
Sustained decode	32K	2	Valid	519.4	271.7	150 ms	4.2 ms	48.9	2	15,582	30.0 s
Sustained decode	32K	4	Valid	1,215.6	306.9	150 ms	3.3 ms	3.9	8	36,469	30.0 s
Sustained decode	32K	8	Valid	1,892.4	234.7	380 ms	4.3 ms	8.1	8	56,773	30.0 s
Sustained decode	32K	16	Valid	2,799.6	177.2	551 ms	5.6 ms	15.9	16	83,985	30.0 s
Sustained decode	32K	32	Valid	3,888.3	120.3	1,478 ms	8.3 ms	39.4	0	116,639	30.0 s
Sustained decode	32K	64	Valid	6,491.9	104.5	2,985 ms	9.6 ms	64.0	0	194,767	30.0 s
Sustained decode	32K	128	Valid	8,981.2	69.5	5,715 ms	14.4 ms	128.6	0	269,439	30.0 s
Sustained decode	64K	1	Valid	379.8	386.2	159 ms	2.6 ms	0.9	3	11,392	30.0 s
Sustained decode	64K	2	Valid	326.3	243.6	257 ms	5.2 ms	84.5	2	9,788	30.0 s
Sustained decode	64K	4	Valid	1,099.4	287.5	263 ms	3.5 ms	4.3	8	32,982	30.0 s
Sustained decode	64K	8	Valid	1,621.4	207.4	681 ms	4.8 ms	9.1	8	48,630	30.0 s
Sustained decode	64K	16	Valid	2,478.0	159.7	1,285 ms	6.3 ms	16.1	16	74,337	30.0 s
Sustained decode	64K	32	Valid	2,391.7	77.8	3,221 ms	12.8 ms	46.9	0	71,742	30.0 s
Sustained decode	64K	64	Valid	4,324.1	70.3	6,372 ms	14.2 ms	77.3	0	129,728	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	362.3	368.4	266 ms	2.7 ms	1.0	2	10,870	30.0 s
Sustained decode	128K	2	Valid	238.5	111.7	459 ms	9.0 ms	49.5	0	7,156	30.0 s
Sustained decode	128K	4	Valid	1,017.0	262.1	437 ms	3.8 ms	3.9	8	30,510	30.0 s
Sustained decode	128K	8	Valid	1,342.7	178.8	1,120 ms	5.6 ms	9.8	8	40,278	30.0 s
Sustained decode	128K	16	Valid	1,874.2	120.4	3,207 ms	8.3 ms	19.4	5	56,077	29.9 s
Sustained decode	128K	32	Valid	1,878.7	65.5	6,348 ms	15.3 ms	37.0	0	56,352	30.0 s
Sustained decode	128K	64	Valid	2,161.8	39.0	12,624 ms	25.6 ms	96.0	0	64,832	30.0 s
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	329.5	341.1	510 ms	2.9 ms	0.9	2	9,884	30.0 s
Sustained decode	251K	2	Valid	69.8	34.9	891 ms	28.6 ms	66.4	0	2,094	30.0 s
Sustained decode	251K	4	Valid	141.7	35.8	1,638 ms	28.0 ms	69.0	0	4,248	30.0 s
Sustained decode	251K	8	Valid	980.8	126.7	3,247 ms	7.9 ms	10.8	6	29,001	29.6 s
Sustained decode	251K	16	Valid	1,275.9	85.7	6,161 ms	11.7 ms	22.0	0	38,272	30.0 s
Sustained decode	251K	32	Valid	1,602.1	54.4	12,879 ms	18.4 ms	36.8	0	48,064	30.0 s
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.