Speed test matrix

LFM2.5-8B-A1B on RTX PRO 6000 Blackwell Server Edition 96GB

sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K

by @keennaylmx dev importMeasured Sep 22, 2026, 08:07 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9265+g05c584c44
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 05c584c44fb0

Run

Served model: lfm
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_rtxpro6k-liquidai-sglang/bin/sglang" serve --model-path LiquidAI/LFM2.5-8B-A1B --served-model-name lfm --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser lfm2 --context-length 128000 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context16.1K508 ms TTFT

16K context31.9K510 ms TTFT

32K context31.7K1,019 ms TTFT

64K context42.2K1,528 ms TTFT

120K context39.5K3,060 ms TTFT

125K context41K3,073 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	261.037 ms TTFT	384.993 ms TTFT	596.9106 ms TTFT	912.5120 ms TTFT	1,505.5192 ms TTFT	2,496.5376 ms TTFT	3,503.1694 ms TTFT	6,034.81,275 ms TTFT
16K	248.8155 ms TTFT	375.1276 ms TTFT	582.7516 ms TTFT	917.4428 ms TTFT	1,440.3382 ms TTFT	2,338.2673 ms TTFT	3,222.41,310 ms TTFT	5,455.92,604 ms TTFT
32K	242.4192 ms TTFT	365.6333 ms TTFT	568.2559 ms TTFT	878.3356 ms TTFT	1,354.9714 ms TTFT	2,065.91,410 ms TTFT	2,854.92,911 ms TTFT	Over capacity
64K	233.5259 ms TTFT	346.3427 ms TTFT	515.5730 ms TTFT	820.7917 ms TTFT	1,160.81,426 ms TTFT	1,700.22,857 ms TTFT	Over capacity	Over capacity
120K	210.8377 ms TTFT	320.4592 ms TTFT	487.51,015 ms TTFT	727.81,393 ms TTFT	984.02,613 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	16.1K	—	508 ms	—	—	1	—	—
Prefill	16K	1	Valid	31.9K	—	510 ms	—	—	1	—	—
Prefill	32K	1	Valid	31.7K	—	1,019 ms	—	—	1	—	—
Prefill	64K	1	Valid	42.2K	—	1,528 ms	—	—	1	—	—
Prefill	120K	1	Valid	39.5K	—	3,060 ms	—	—	1	—	—
Prefill	125K	1	Valid	41K	—	3,073 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	261.0	261.7	37 ms	3.8 ms	1.0	2	7,829	30.0 s
Sustained decode	8K	2	Valid	384.9	193.4	93 ms	5.2 ms	2.0	2	11,548	30.0 s
Sustained decode	8K	4	Valid	596.9	152.2	106 ms	6.6 ms	4.0	4	17,904	30.0 s
Sustained decode	8K	8	Valid	912.5	115.9	120 ms	8.6 ms	8.0	0	27,376	30.0 s
Sustained decode	8K	16	Valid	1,505.5	95.5	192 ms	10.5 ms	16.0	0	45,152	30.0 s
Sustained decode	8K	32	Valid	2,496.5	79.8	376 ms	12.5 ms	32.0	0	74,896	30.0 s
Sustained decode	8K	64	Valid	3,503.1	56.4	694 ms	17.7 ms	64.0	0	105,088	30.0 s
Sustained decode	8K	128	Valid	6,034.8	48.6	1,275 ms	20.6 ms	128.0	0	180,992	30.0 s
Sustained decode	16K	1	Valid	248.8	251.5	155 ms	4.0 ms	1.0	2	7,464	30.0 s
Sustained decode	16K	2	Valid	375.1	189.9	276 ms	5.3 ms	2.0	2	11,252	30.0 s
Sustained decode	16K	4	Valid	582.7	150.4	516 ms	6.6 ms	4.0	4	17,480	30.0 s
Sustained decode	16K	8	Valid	917.4	115.8	428 ms	8.6 ms	8.0	0	27,520	30.0 s
Sustained decode	16K	16	Valid	1,440.3	91.3	382 ms	11.0 ms	16.0	0	43,200	30.0 s
Sustained decode	16K	32	Valid	2,338.2	74.6	673 ms	13.4 ms	32.0	0	70,144	30.0 s
Sustained decode	16K	64	Valid	3,222.4	51.6	1,310 ms	19.4 ms	64.0	0	96,640	30.0 s
Sustained decode	16K	128	Valid	5,455.9	43.8	2,604 ms	22.8 ms	128.0	0	163,584	30.0 s
Sustained decode	32K	1	Valid	242.4	244.0	192 ms	4.1 ms	1.0	1	7,273	30.0 s
Sustained decode	32K	2	Valid	365.6	185.3	333 ms	5.4 ms	1.9	2	10,966	30.0 s
Sustained decode	32K	4	Valid	568.2	147.7	559 ms	6.8 ms	4.0	4	17,044	30.0 s
Sustained decode	32K	8	Valid	878.3	110.5	356 ms	9.0 ms	8.0	0	26,344	30.0 s
Sustained decode	32K	16	Valid	1,354.9	85.5	714 ms	11.7 ms	16.0	0	40,640	30.0 s
Sustained decode	32K	32	Valid	2,065.9	65.7	1,410 ms	15.2 ms	32.0	0	61,952	30.0 s
Sustained decode	32K	64	Valid	2,854.9	45.5	2,911 ms	22.0 ms	64.0	0	85,632	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	233.5	235.6	259 ms	4.2 ms	1.0	1	7,004	30.0 s
Sustained decode	64K	2	Valid	346.3	175.7	427 ms	5.7 ms	1.9	2	10,386	30.0 s
Sustained decode	64K	4	Valid	515.5	140.2	730 ms	7.2 ms	3.9	4	15,464	30.0 s
Sustained decode	64K	8	Valid	820.7	103.6	917 ms	9.7 ms	8.0	0	24,616	30.0 s
Sustained decode	64K	16	Valid	1,160.8	74.2	1,426 ms	13.5 ms	16.0	0	34,816	30.0 s
Sustained decode	64K	32	Valid	1,700.2	55.1	2,857 ms	18.1 ms	32.0	0	50,976	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	1	Valid	210.8	213.6	377 ms	4.7 ms	1.0	1	6,324	30.0 s
Sustained decode	120K	2	Valid	320.4	164.3	592 ms	6.1 ms	1.9	2	9,612	30.0 s
Sustained decode	120K	4	Valid	487.5	156.1	1,015 ms	6.6 ms	3.9	4	14,624	30.0 s
Sustained decode	120K	8	Valid	727.8	92.2	1,393 ms	10.8 ms	8.0	0	21,835	30.0 s
Sustained decode	120K	16	Valid	984.0	63.7	2,613 ms	15.7 ms	16.0	0	29,504	30.0 s
Sustained decode	120K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.