Speed test matrix

Ling-3.0-flash-fp4 on RTX PRO 6000 Blackwell Server Edition 96GB

sglang · 0.5.22.dev236+g1093c501d · MXFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Oct 12, 2026, 06:47 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.22.dev236+g1093c501d
Quantization: MXFP4+FP8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 1093c501dfef

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.7.6
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env MAX_JOBS=2 "$HOME/env_rtxpro6k-inclusionai-sglang-1093c501dfef/bin/sglang" serve --model-path inclusionAI/Ling-3.0-flash-fp4 --served-model-name ling --trust-remote-code --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 262144 --mem-fraction-static 0.81 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton --fp8-gemm-backend cutlass --moe-runner-backend flashinfer_mxfp4

Prefill

prompt tok/s by context length

8K context19.9K413 ms TTFT

16K context13.7K1,182 ms TTFT

32K context8,0833,994 ms TTFT

64K context4,55014,150 ms TTFT

128K context2,08261,764 ms TTFT

251K context963.0261,722 ms TTFT

256K context939.0273,603 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	149.575 ms TTFT	247.4132 ms TTFT	414.1148 ms TTFT	610.5191 ms TTFT	888.4254 ms TTFT	874.4Requests queued38,730 ms TTFT	887.4Requests queued38,347 ms TTFT	Over capacity
16K	146.897 ms TTFT	242.6228 ms TTFT	398.5210 ms TTFT	593.2274 ms TTFT	826.2400 ms TTFT	836.2Requests queued40,193 ms TTFT	Over capacity	Over capacity
32K	142.3131 ms TTFT	232.6199 ms TTFT	375.2295 ms TTFT	534.2411 ms TTFT	743.3827 ms TTFT	Over capacity	Over capacity	Over capacity
64K	134.6228 ms TTFT	215.8287 ms TTFT	333.2460 ms TTFT	460.2895 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	122.5444 ms TTFT	190.7541 ms TTFT	280.5841 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	107.1Requests queued834 ms TTFT	154.6Requests queued1,049 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	19.9K	—	413 ms	—	—	1	—	—
Prefill	16K	1	Valid	13.7K	—	1,182 ms	—	—	1	—	—
Prefill	32K	1	Valid	8,083.0	—	3,994 ms	—	—	1	—	—
Prefill	64K	1	Valid	4,550.0	—	14,150 ms	—	—	1	—	—
Prefill	128K	1	Valid	2,082.0	—	61,764 ms	—	—	1	—	—
Prefill	251K	1	Valid	963.0	—	261,722 ms	—	—	1	—	—
Prefill	256K	1	Valid	939.0	—	273,603 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	149.5	150.0	75 ms	6.7 ms	1.0	1	4,483	30.0 s
Sustained decode	8K	2	Valid	247.4	124.9	132 ms	8.0 ms	2.0	2	7,422	30.0 s
Sustained decode	8K	4	Valid	414.1	103.7	148 ms	9.6 ms	4.0	0	12,420	30.0 s
Sustained decode	8K	8	Valid	610.5	76.9	191 ms	13.0 ms	8.0	0	18,312	30.0 s
Sustained decode	8K	16	Valid	888.4	56.2	254 ms	17.8 ms	16.0	0	26,639	30.0 s
Sustained decode	8K	32	Requests queued	874.4	56.3	38,730 ms	17.8 ms	16.0	16	26,224	30.0 s
Sustained decode	8K	64	Requests queued	887.4	56.7	38,347 ms	17.7 ms	16.0	16	26,623	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	146.8	147.6	97 ms	6.8 ms	1.0	1	4,403	30.0 s
Sustained decode	16K	2	Valid	242.6	121.4	228 ms	8.2 ms	2.0	0	7,276	30.0 s
Sustained decode	16K	4	Valid	398.5	99.7	210 ms	10.0 ms	4.0	0	11,952	30.0 s
Sustained decode	16K	8	Valid	593.2	74.5	274 ms	13.4 ms	8.0	0	17,791	30.0 s
Sustained decode	16K	16	Valid	826.2	52.2	400 ms	19.2 ms	16.0	0	24,785	30.0 s
Sustained decode	16K	32	Requests queued	836.2	53.7	40,193 ms	18.6 ms	15.5	16	25,087	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	142.3	143.1	131 ms	7.0 ms	1.0	1	4,272	30.0 s
Sustained decode	32K	2	Valid	232.6	116.4	199 ms	8.6 ms	2.0	0	6,976	30.0 s
Sustained decode	32K	4	Valid	375.2	93.8	295 ms	10.7 ms	4.0	0	11,256	30.0 s
Sustained decode	32K	8	Valid	534.2	66.9	411 ms	14.9 ms	8.0	0	16,024	30.0 s
Sustained decode	32K	16	Valid	743.3	46.7	827 ms	21.4 ms	16.0	0	22,293	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	134.6	135.9	228 ms	7.4 ms	1.0	1	4,039	30.0 s
Sustained decode	64K	2	Valid	215.8	107.8	287 ms	9.3 ms	2.0	0	6,472	30.0 s
Sustained decode	64K	4	Valid	333.2	83.1	460 ms	12.0 ms	4.0	0	9,996	30.0 s
Sustained decode	64K	8	Valid	460.2	57.6	895 ms	17.4 ms	8.0	0	13,800	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	122.5	124.8	444 ms	8.0 ms	1.0	1	3,674	30.0 s
Sustained decode	128K	2	Valid	190.7	95.2	541 ms	10.5 ms	2.0	0	5,722	30.0 s
Sustained decode	128K	4	Valid	280.5	69.8	841 ms	14.3 ms	4.0	0	8,412	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Requests queued	107.1	108.3	834 ms	9.2 ms	1.0	0	3,213	30.0 s
Sustained decode	251K	2	Requests queued	154.6	78.7	1,049 ms	12.7 ms	1.9	0	4,636	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.