Speed test matrix

Ling-3.0-tiny-base on H200 NVL 141GB

sglang · 0.5.6.post3.dev8719+ge1a24a189 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 19, 2026, 10:45 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev8719+ge1a24a189
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: e1a24a189fe4

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny-base --served-model-name ling --trust-remote-code --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 262144 --mem-fraction-static 0.869695 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context67.5K121 ms TTFT

16K context74K219 ms TTFT

32K context76.6K421 ms TTFT

64K context72.2K891 ms TTFT

128K context58.5K2,197 ms TTFT

251K context42K5,996 ms TTFT

256K context41.4K6,211 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	403.167 ms TTFT	750.088 ms TTFT	1,245.191 ms TTFT	1,993.3146 ms TTFT	3,034.7203 ms TTFT	5,105.5341 ms TTFT	7,369.3696 ms TTFT	12.6K1,244 ms TTFT
16K	399.077 ms TTFT	733.4108 ms TTFT	1,253.8100 ms TTFT	1,933.9205 ms TTFT	2,814.8361 ms TTFT	4,406.7703 ms TTFT	5,919.81,295 ms TTFT	10.6K2,459 ms TTFT
32K	392.1105 ms TTFT	523.1170 ms TTFT	1,139.9171 ms TTFT	1,796.8422 ms TTFT	2,815.9602 ms TTFT	3,911.01,665 ms TTFT	4,903.33,214 ms TTFT	6,950.46,191 ms TTFT
64K	379.6167 ms TTFT	185.3268 ms TTFT	1,145.2250 ms TTFT	1,684.1663 ms TTFT	2,513.91,098 ms TTFT	2,508.83,388 ms TTFT	4,602.46,920 ms TTFT	Over capacity
128K	364.1244 ms TTFT	290.4466 ms TTFT	957.4682 ms TTFT	1,500.61,063 ms TTFT	1,951.53,446 ms TTFT	1,541.16,633 ms TTFT	2,146.714,575 ms TTFT	Over capacity
251K	328.6515 ms TTFT	69.3957 ms TTFT	143.31,767 ms TTFT	963.93,481 ms TTFT	1,166.46,694 ms TTFT	1,273.513,588 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	67.5K	—	121 ms	—	—	1	—	—
Prefill	16K	1	Valid	74K	—	219 ms	—	—	1	—	—
Prefill	32K	1	Valid	76.6K	—	421 ms	—	—	1	—	—
Prefill	64K	1	Valid	72.2K	—	891 ms	—	—	1	—	—
Prefill	128K	1	Valid	58.5K	—	2,197 ms	—	—	1	—	—
Prefill	251K	1	Valid	42K	—	5,996 ms	—	—	1	—	—
Prefill	256K	1	Valid	41.4K	—	6,211 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	403.1	405.9	67 ms	2.5 ms	1.0	3	12,093	30.0 s
Sustained decode	8K	2	Valid	750.0	377.8	88 ms	2.6 ms	2.0	6	22,500	30.0 s
Sustained decode	8K	4	Valid	1,245.1	323.3	91 ms	3.1 ms	4.3	8	37,350	30.0 s
Sustained decode	8K	8	Valid	1,993.3	257.2	146 ms	3.9 ms	8.0	16	59,800	30.0 s
Sustained decode	8K	16	Valid	3,034.7	192.5	203 ms	5.2 ms	19.5	16	91,040	30.0 s
Sustained decode	8K	32	Valid	5,105.5	165.0	341 ms	6.1 ms	34.4	32	153,168	30.0 s
Sustained decode	8K	64	Valid	7,369.3	114.9	696 ms	8.7 ms	83.2	0	221,088	30.0 s
Sustained decode	8K	128	Valid	12.6K	99.3	1,244 ms	10.1 ms	128.0	0	378,369	30.0 s
Sustained decode	16K	1	Valid	399.0	402.0	77 ms	2.5 ms	1.0	3	11,971	30.0 s
Sustained decode	16K	2	Valid	733.4	377.2	108 ms	2.7 ms	6.2	4	22,001	30.0 s
Sustained decode	16K	4	Valid	1,253.8	316.2	100 ms	3.2 ms	4.0	8	37,614	30.0 s
Sustained decode	16K	8	Valid	1,933.9	239.0	205 ms	4.2 ms	8.0	16	58,016	30.0 s
Sustained decode	16K	16	Valid	2,814.8	181.3	361 ms	5.5 ms	20.3	16	84,449	30.0 s
Sustained decode	16K	32	Valid	4,406.7	150.3	703 ms	6.7 ms	37.7	32	132,194	30.0 s
Sustained decode	16K	64	Valid	5,919.8	92.9	1,295 ms	10.8 ms	96.0	0	177,601	30.0 s
Sustained decode	16K	128	Valid	10.6K	79.6	2,459 ms	12.6 ms	145.1	0	316,624	30.0 s
Sustained decode	32K	1	Valid	392.1	396.4	105 ms	2.5 ms	1.0	3	11,763	30.0 s
Sustained decode	32K	2	Valid	523.1	366.5	170 ms	2.7 ms	48.9	4	15,690	30.0 s
Sustained decode	32K	4	Valid	1,139.9	305.6	171 ms	3.3 ms	4.6	8	34,195	30.0 s
Sustained decode	32K	8	Valid	1,796.8	225.3	422 ms	4.4 ms	8.7	8	53,904	30.0 s
Sustained decode	32K	16	Valid	2,815.9	179.4	602 ms	5.6 ms	15.9	16	84,473	30.0 s
Sustained decode	32K	32	Valid	3,911.0	121.8	1,665 ms	8.2 ms	38.5	11	117,122	29.9 s
Sustained decode	32K	64	Valid	4,903.3	79.4	3,214 ms	12.6 ms	75.0	0	147,072	30.0 s
Sustained decode	32K	128	Valid	6,950.4	55.7	6,191 ms	18.0 ms	170.0	0	208,512	30.0 s
Sustained decode	64K	1	Valid	379.6	386.3	167 ms	2.6 ms	1.0	3	11,388	30.0 s
Sustained decode	64K	2	Valid	185.3	89.3	268 ms	11.2 ms	115.1	0	5,558	30.0 s
Sustained decode	64K	4	Valid	1,145.2	291.7	250 ms	3.4 ms	4.0	8	34,353	30.0 s
Sustained decode	64K	8	Valid	1,684.1	213.5	663 ms	4.7 ms	8.6	8	50,523	30.0 s
Sustained decode	64K	16	Valid	2,513.9	159.5	1,098 ms	6.3 ms	15.8	16	75,415	30.0 s
Sustained decode	64K	32	Valid	2,508.8	81.6	3,388 ms	12.3 ms	45.9	0	75,264	30.0 s
Sustained decode	64K	64	Valid	4,602.4	74.8	6,920 ms	13.4 ms	73.8	0	138,048	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	364.1	370.0	244 ms	2.7 ms	1.0	2	10,923	30.0 s
Sustained decode	128K	2	Valid	290.4	229.4	466 ms	5.4 ms	42.5	2	8,713	30.0 s
Sustained decode	128K	4	Valid	957.4	242.6	682 ms	4.1 ms	4.6	4	28,720	30.0 s
Sustained decode	128K	8	Valid	1,500.6	192.7	1,063 ms	5.2 ms	7.9	8	45,015	30.0 s
Sustained decode	128K	16	Valid	1,951.5	123.5	3,446 ms	8.1 ms	17.0	15	58,844	30.2 s
Sustained decode	128K	32	Valid	1,541.1	53.9	6,633 ms	18.6 ms	47.0	0	46,208	30.0 s
Sustained decode	128K	64	Valid	2,146.7	39.4	14,575 ms	25.4 ms	106.4	0	64,384	30.0 s
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	328.6	340.4	515 ms	2.9 ms	0.9	2	9,857	30.0 s
Sustained decode	251K	2	Valid	69.3	34.6	957 ms	28.9 ms	74.1	0	2,076	30.0 s
Sustained decode	251K	4	Valid	143.3	36.2	1,767 ms	27.6 ms	68.9	0	4,296	30.0 s
Sustained decode	251K	8	Valid	963.9	125.6	3,481 ms	8.0 ms	10.7	8	29,286	30.4 s
Sustained decode	251K	16	Valid	1,166.4	78.9	6,694 ms	12.7 ms	24.0	0	34,993	30.0 s
Sustained decode	251K	32	Valid	1,273.5	45.4	13,588 ms	22.0 ms	42.0	0	38,208	30.0 s
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.