Speed test matrix

Ling-3.0-flash-fp4 on H200 NVL 141GB ×2

vllm · 0.27.2rc1.dev113+g5cecfc013 · MXFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 3, 2026, 14:23 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.27.2rc1.dev113+g5cecfc013
Quantization: MXFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 5cecfc013750

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-3.0-flash-fp4 --served-model-name ling --trust-remote-code --tensor-parallel-size 2 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 262144 --gpu-memory-utilization 0.872320 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align

Prefill

prompt tok/s by context length

8K context26.5K310 ms TTFT

16K context26.1K620 ms TTFT

32K context27.7K1,165 ms TTFT

64K context28.2K2,280 ms TTFT

128K context25.9K4,972 ms TTFT

251K context21.7K11,593 ms TTFT

256K context21.5K11,974 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	164.162 ms TTFT	284.685 ms TTFT	534.9102 ms TTFT	937.9166 ms TTFT	1,621.8207 ms TTFT	2,544.8360 ms TTFT	4,001.2529 ms TTFT	5,912.7845 ms TTFT
16K	162.1148 ms TTFT	281.9178 ms TTFT	528.2273 ms TTFT	929.0343 ms TTFT	1,573.7711 ms TTFT	2,536.0819 ms TTFT	3,809.71,363 ms TTFT	5,527.82,292 ms TTFT
32K	159.5167 ms TTFT	277.4209 ms TTFT	520.4286 ms TTFT	901.5408 ms TTFT	1,520.4613 ms TTFT	2,386.4980 ms TTFT	3,538.21,534 ms TTFT	4,919.32,649 ms TTFT
64K	154.6231 ms TTFT	269.5284 ms TTFT	498.7380 ms TTFT	858.2597 ms TTFT	1,428.4888 ms TTFT	2,115.21,558 ms TTFT	3,003.82,705 ms TTFT	3,990.05,119 ms TTFT
128K	145.0351 ms TTFT	255.1390 ms TTFT	482.8706 ms TTFT	790.21,116 ms TTFT	1,250.51,960 ms TTFT	1,743.33,652 ms TTFT	2,304.86,741 ms TTFT	Over capacity
251K	129.4582 ms TTFT	234.3838 ms TTFT	438.81,163 ms TTFT	636.31,795 ms TTFT	1,013.63,313 ms TTFT	1,313.16,137 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	26.5K	—	310 ms	—	—	1	—	—
Prefill	16K	1	Valid	26.1K	—	620 ms	—	—	1	—	—
Prefill	32K	1	Valid	27.7K	—	1,165 ms	—	—	1	—	—
Prefill	64K	1	Valid	28.2K	—	2,280 ms	—	—	1	—	—
Prefill	128K	1	Valid	25.9K	—	4,972 ms	—	—	1	—	—
Prefill	251K	1	Valid	21.7K	—	11,593 ms	—	—	1	—	—
Prefill	256K	1	Valid	21.5K	—	11,974 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	164.1	164.6	62 ms	6.1 ms	1.0	1	4,925	30.0 s
Sustained decode	8K	2	Valid	284.6	142.9	85 ms	7.0 ms	2.0	2	8,536	30.0 s
Sustained decode	8K	4	Valid	534.9	134.2	102 ms	7.5 ms	4.0	4	16,047	30.0 s
Sustained decode	8K	8	Valid	937.9	117.6	166 ms	8.5 ms	8.0	0	28,136	30.0 s
Sustained decode	8K	16	Valid	1,621.8	101.6	207 ms	9.8 ms	16.0	0	48,656	30.0 s
Sustained decode	8K	32	Valid	2,544.8	81.4	360 ms	12.3 ms	32.0	0	76,342	30.0 s
Sustained decode	8K	64	Valid	4,001.2	63.0	529 ms	15.9 ms	64.0	0	120,011	30.0 s
Sustained decode	8K	128	Valid	5,912.7	46.3	845 ms	21.6 ms	128.0	0	177,409	30.0 s
Sustained decode	16K	1	Valid	162.1	163.2	148 ms	6.1 ms	1.0	1	4,864	30.0 s
Sustained decode	16K	2	Valid	281.9	141.8	178 ms	7.1 ms	2.0	2	8,456	30.0 s
Sustained decode	16K	4	Valid	528.2	133.3	273 ms	7.5 ms	4.0	4	15,845	30.0 s
Sustained decode	16K	8	Valid	929.0	116.3	343 ms	8.6 ms	8.0	0	27,872	30.0 s
Sustained decode	16K	16	Valid	1,573.7	98.8	711 ms	10.1 ms	16.0	0	47,200	30.0 s
Sustained decode	16K	32	Valid	2,536.0	79.0	819 ms	12.7 ms	32.0	0	76,064	30.0 s
Sustained decode	16K	64	Valid	3,809.7	58.9	1,363 ms	17.0 ms	64.0	0	114,309	30.0 s
Sustained decode	16K	128	Valid	5,527.8	41.7	2,292 ms	24.0 ms	128.0	0	165,837	30.0 s
Sustained decode	32K	1	Valid	159.5	160.8	167 ms	6.2 ms	1.0	1	4,785	30.0 s
Sustained decode	32K	2	Valid	277.4	139.7	209 ms	7.2 ms	2.0	2	8,326	30.0 s
Sustained decode	32K	4	Valid	520.4	131.2	286 ms	7.6 ms	4.0	4	15,611	30.0 s
Sustained decode	32K	8	Valid	901.5	112.8	408 ms	8.9 ms	8.0	0	27,040	30.0 s
Sustained decode	32K	16	Valid	1,520.4	95.0	613 ms	10.5 ms	16.0	0	45,600	30.0 s
Sustained decode	32K	32	Valid	2,386.4	74.2	980 ms	13.5 ms	32.0	0	71,584	30.0 s
Sustained decode	32K	64	Valid	3,538.2	54.3	1,534 ms	18.4 ms	64.0	0	106,170	30.0 s
Sustained decode	32K	128	Valid	4,919.3	36.9	2,649 ms	27.1 ms	128.0	0	147,465	30.0 s
Sustained decode	64K	1	Valid	154.6	156.3	231 ms	6.4 ms	1.0	1	4,639	30.0 s
Sustained decode	64K	2	Valid	269.5	136.1	284 ms	7.3 ms	2.0	2	8,086	30.0 s
Sustained decode	64K	4	Valid	498.7	126.1	380 ms	7.9 ms	3.9	4	14,961	30.0 s
Sustained decode	64K	8	Valid	858.2	106.8	597 ms	9.4 ms	8.0	0	25,744	30.0 s
Sustained decode	64K	16	Valid	1,428.4	88.0	888 ms	11.4 ms	16.0	0	42,851	30.0 s
Sustained decode	64K	32	Valid	2,115.2	64.7	1,558 ms	15.5 ms	32.0	0	63,456	30.0 s
Sustained decode	64K	64	Valid	3,003.8	45.0	2,705 ms	22.2 ms	64.0	0	90,124	30.0 s
Sustained decode	64K	128	Valid	3,990.0	28.4	5,119 ms	35.2 ms	128.0	0	119,694	30.0 s
Sustained decode	128K	1	Valid	145.0	147.7	351 ms	6.8 ms	1.0	1	4,350	30.0 s
Sustained decode	128K	2	Valid	255.1	129.5	390 ms	7.7 ms	2.0	2	7,655	30.0 s
Sustained decode	128K	4	Valid	482.8	120.7	706 ms	8.3 ms	4.0	1	14,483	30.0 s
Sustained decode	128K	8	Valid	790.2	98.4	1,116 ms	10.2 ms	8.0	0	23,704	30.0 s
Sustained decode	128K	16	Valid	1,250.5	77.3	1,960 ms	12.9 ms	16.0	0	37,536	30.0 s
Sustained decode	128K	32	Valid	1,743.3	53.4	3,652 ms	18.7 ms	32.0	0	52,292	30.0 s
Sustained decode	128K	64	Valid	2,304.8	34.5	6,741 ms	29.0 ms	64.0	0	69,122	30.0 s
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	129.4	136.3	582 ms	7.3 ms	1.0	1	3,883	30.0 s
Sustained decode	251K	2	Valid	234.3	117.7	838 ms	8.5 ms	2.0	0	7,028	30.0 s
Sustained decode	251K	4	Valid	438.8	110.0	1,163 ms	9.1 ms	4.0	0	13,164	30.0 s
Sustained decode	251K	8	Valid	636.3	85.4	1,795 ms	11.7 ms	8.0	0	19,087	30.0 s
Sustained decode	251K	16	Valid	1,013.6	63.7	3,313 ms	15.7 ms	16.0	0	30,400	30.0 s
Sustained decode	251K	32	Valid	1,313.1	41.2	6,137 ms	24.3 ms	32.0	0	39,392	30.0 s
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.