Speed test matrix

Ling-3.0-tiny-base on RTX PRO 6000 Blackwell Server Edition 96GB

sglang · 0.5.22.dev236+g1093c501d · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Oct 12, 2026, 08:51 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.22.dev236+g1093c501d
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1093c501dfef

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.7.6
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env MAX_JOBS=2 SGLANG_BATCH_INVARIANT_OPS_ENABLE_MM_DEEPGEMM=0 "$HOME/env_rtxpro6k-inclusionai-sglang-1093c501dfef/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny-base --served-model-name ling --trust-remote-code --tp 1 --context-length 262144 --mem-fraction-static 0.72 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-deterministic-inference --attention-backend triton --moe-runner-backend triton

Prefill

prompt tok/s by context length

8K context26.3K312 ms TTFT

16K context21.3K762 ms TTFT

32K context15K2,151 ms TTFT

64K context9,3736,868 ms TTFT

128K context4,64127,710 ms TTFT

251K context2,199114,599 ms TTFT

256K context2,152119,467 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	97.461 ms TTFT	194.1211 ms TTFT	381.1105 ms TTFT	746.3175 ms TTFT	1,360.3221 ms TTFT	2,569.7377 ms TTFT	4,584.5757 ms TTFT	7,051.41,556 ms TTFT
16K	96.890 ms TTFT	194.1120 ms TTFT	373.4147 ms TTFT	726.8207 ms TTFT	1,311.0387 ms TTFT	2,413.6758 ms TTFT	4,031.51,519 ms TTFT	5,966.73,052 ms TTFT
32K	95.6114 ms TTFT	184.9141 ms TTFT	349.8251 ms TTFT	652.8449 ms TTFT	1,046.8897 ms TTFT	1,752.81,821 ms TTFT	2,688.73,704 ms TTFT	Over capacity
64K	92.2201 ms TTFT	177.4275 ms TTFT	314.3421 ms TTFT	538.9834 ms TTFT	810.41,699 ms TTFT	1,295.13,411 ms TTFT	Over capacity	Over capacity
128K	88.7375 ms TTFT	160.7474 ms TTFT	274.9877 ms TTFT	449.81,604 ms TTFT	660.63,157 ms TTFT	973.36,467 ms TTFT	Over capacity	Over capacity
251K	75.0733 ms TTFT	137.81,007 ms TTFT	224.11,657 ms TTFT	355.23,293 ms TTFT	499.26,530 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	26.3K	—	312 ms	—	—	1	—	—
Prefill	16K	1	Valid	21.3K	—	762 ms	—	—	1	—	—
Prefill	32K	1	Valid	15K	—	2,151 ms	—	—	1	—	—
Prefill	64K	1	Valid	9,373.0	—	6,868 ms	—	—	1	—	—
Prefill	128K	1	Valid	4,641.0	—	27,710 ms	—	—	1	—	—
Prefill	251K	1	Valid	2,199.0	—	114,599 ms	—	—	1	—	—
Prefill	256K	1	Valid	2,152.0	—	119,467 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	97.4	97.5	61 ms	10.3 ms	1.0	0	2,923	30.0 s
Sustained decode	8K	2	Valid	194.1	96.7	211 ms	10.3 ms	2.0	0	5,822	30.0 s
Sustained decode	8K	4	Valid	381.1	95.3	105 ms	10.5 ms	4.0	0	11,432	30.0 s
Sustained decode	8K	8	Valid	746.3	93.3	175 ms	10.7 ms	8.0	0	22,384	30.0 s
Sustained decode	8K	16	Valid	1,360.3	84.9	221 ms	11.8 ms	16.0	0	40,800	30.0 s
Sustained decode	8K	32	Valid	2,569.7	80.2	377 ms	12.5 ms	32.0	0	77,088	30.0 s
Sustained decode	8K	64	Valid	4,584.5	71.8	757 ms	13.9 ms	64.0	0	137,536	30.0 s
Sustained decode	8K	128	Valid	7,051.4	55.4	1,556 ms	18.0 ms	128.0	0	211,499	30.0 s
Sustained decode	16K	1	Valid	96.8	96.8	90 ms	10.3 ms	1.0	0	2,904	30.0 s
Sustained decode	16K	2	Valid	194.1	97.2	120 ms	10.3 ms	2.0	0	5,822	30.0 s
Sustained decode	16K	4	Valid	373.4	93.4	147 ms	10.7 ms	4.0	0	11,200	30.0 s
Sustained decode	16K	8	Valid	726.8	90.8	207 ms	11.0 ms	8.0	0	21,800	30.0 s
Sustained decode	16K	16	Valid	1,311.0	81.8	387 ms	12.2 ms	16.0	0	39,328	30.0 s
Sustained decode	16K	32	Valid	2,413.6	75.3	758 ms	13.3 ms	32.0	0	72,384	30.0 s
Sustained decode	16K	64	Valid	4,031.5	63.1	1,519 ms	15.9 ms	64.0	0	120,896	30.0 s
Sustained decode	16K	128	Valid	5,966.7	46.7	3,052 ms	21.4 ms	128.0	0	178,944	30.0 s
Sustained decode	32K	1	Valid	95.6	95.6	114 ms	10.5 ms	1.0	0	2,867	30.0 s
Sustained decode	32K	2	Valid	184.9	92.6	141 ms	10.8 ms	2.0	0	5,548	30.0 s
Sustained decode	32K	4	Valid	349.8	87.4	251 ms	11.4 ms	4.0	0	10,492	30.0 s
Sustained decode	32K	8	Valid	652.8	81.4	449 ms	12.3 ms	8.0	0	19,592	30.0 s
Sustained decode	32K	16	Valid	1,046.8	65.6	897 ms	15.2 ms	16.0	0	31,392	30.0 s
Sustained decode	32K	32	Valid	1,752.8	55.0	1,821 ms	18.2 ms	32.0	0	52,576	30.0 s
Sustained decode	32K	64	Valid	2,688.7	42.2	3,704 ms	23.7 ms	64.0	0	80,640	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	92.2	92.3	201 ms	10.8 ms	1.0	0	2,766	30.0 s
Sustained decode	64K	2	Valid	177.4	88.8	275 ms	11.3 ms	2.0	0	5,321	30.0 s
Sustained decode	64K	4	Valid	314.3	78.3	421 ms	12.8 ms	4.0	0	9,427	30.0 s
Sustained decode	64K	8	Valid	538.9	67.3	834 ms	14.9 ms	8.0	0	16,168	30.0 s
Sustained decode	64K	16	Valid	810.4	50.9	1,699 ms	19.6 ms	16.0	0	24,304	30.0 s
Sustained decode	64K	32	Valid	1,295.1	40.9	3,411 ms	24.5 ms	32.0	0	38,848	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	88.7	89.0	375 ms	11.2 ms	1.0	0	2,662	30.0 s
Sustained decode	128K	2	Valid	160.7	80.3	474 ms	12.5 ms	2.0	0	4,821	30.0 s
Sustained decode	128K	4	Valid	274.9	68.6	877 ms	14.6 ms	4.0	0	8,243	30.0 s
Sustained decode	128K	8	Valid	449.8	56.2	1,604 ms	17.8 ms	8.0	0	13,488	30.0 s
Sustained decode	128K	16	Valid	660.6	41.5	3,157 ms	24.1 ms	16.0	0	19,808	30.0 s
Sustained decode	128K	32	Valid	973.3	31.0	6,467 ms	32.2 ms	32.0	0	29,184	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	75.0	75.1	733 ms	13.3 ms	1.0	0	2,251	30.0 s
Sustained decode	251K	2	Valid	137.8	68.7	1,007 ms	14.6 ms	2.0	0	4,132	30.0 s
Sustained decode	251K	4	Valid	224.1	55.8	1,657 ms	17.9 ms	4.0	0	6,719	30.0 s
Sustained decode	251K	8	Valid	355.2	44.3	3,293 ms	22.6 ms	8.0	0	10,649	30.0 s
Sustained decode	251K	16	Valid	499.2	31.7	6,530 ms	31.5 ms	16.0	0	14,961	30.0 s
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.