Speed test matrix

Ling-3.0-flash-int4 on H200 NVL 141GB ×2

vllm · 0.27.2rc1.dev113+g5cecfc013 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 3, 2026, 17:22 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.27.2rc1.dev113+g5cecfc013
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 5cecfc013750

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-3.0-flash-int4 --served-model-name ling --trust-remote-code --tensor-parallel-size 2 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 262144 --gpu-memory-utilization 0.872072 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align

Prefill

prompt tok/s by context length

8K context26.1K314 ms TTFT

16K context26.3K618 ms TTFT

32K context27.9K1,156 ms TTFT

64K context28.3K2,276 ms TTFT

128K context26K4,952 ms TTFT

251K context21.7K11,605 ms TTFT

256K context21.2K12,130 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	173.960 ms TTFT	305.582 ms TTFT	560.1107 ms TTFT	990.5160 ms TTFT	1,684.3243 ms TTFT	2,743.8336 ms TTFT	4,228.9539 ms TTFT	6,337.6851 ms TTFT
16K	172.0150 ms TTFT	302.2171 ms TTFT	557.0237 ms TTFT	966.7305 ms TTFT	1,651.4650 ms TTFT	2,723.51,001 ms TTFT	4,082.91,314 ms TTFT	5,865.42,326 ms TTFT
32K	169.0161 ms TTFT	297.8194 ms TTFT	544.2289 ms TTFT	962.1324 ms TTFT	1,591.8609 ms TTFT	2,523.3913 ms TTFT	3,703.21,490 ms TTFT	5,166.12,641 ms TTFT
64K	163.1238 ms TTFT	288.3251 ms TTFT	526.3328 ms TTFT	909.2548 ms TTFT	1,496.8908 ms TTFT	2,205.51,567 ms TTFT	3,112.42,930 ms TTFT	4,016.15,491 ms TTFT
128K	152.7360 ms TTFT	271.1438 ms TTFT	495.0490 ms TTFT	738.81,109 ms TTFT	1,274.61,760 ms TTFT	1,825.23,408 ms TTFT	2,391.86,607 ms TTFT	Over capacity
251K	135.9550 ms TTFT	244.7634 ms TTFT	458.21,116 ms TTFT	709.31,886 ms TTFT	1,039.73,402 ms TTFT	1,359.36,428 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	26.1K	—	314 ms	—	—	1	—	—
Prefill	16K	1	Valid	26.3K	—	618 ms	—	—	1	—	—
Prefill	32K	1	Valid	27.9K	—	1,156 ms	—	—	1	—	—
Prefill	64K	1	Valid	28.3K	—	2,276 ms	—	—	1	—	—
Prefill	128K	1	Valid	26K	—	4,952 ms	—	—	1	—	—
Prefill	251K	1	Valid	21.7K	—	11,605 ms	—	—	1	—	—
Prefill	256K	1	Valid	21.2K	—	12,130 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	173.9	174.4	60 ms	5.7 ms	1.0	1	5,216	30.0 s
Sustained decode	8K	2	Valid	305.5	153.2	82 ms	6.5 ms	2.0	2	9,165	30.0 s
Sustained decode	8K	4	Valid	560.1	140.7	107 ms	7.1 ms	4.0	4	16,800	30.0 s
Sustained decode	8K	8	Valid	990.5	125.7	160 ms	8.0 ms	7.8	8	29,715	30.0 s
Sustained decode	8K	16	Valid	1,684.3	105.8	243 ms	9.4 ms	16.0	0	50,531	30.0 s
Sustained decode	8K	32	Valid	2,743.8	86.5	336 ms	11.6 ms	32.0	0	82,307	30.0 s
Sustained decode	8K	64	Valid	4,228.9	66.7	539 ms	15.0 ms	64.0	0	126,881	30.0 s
Sustained decode	8K	128	Valid	6,337.6	49.6	851 ms	20.2 ms	128.0	0	190,086	30.0 s
Sustained decode	16K	1	Valid	172.0	173.2	150 ms	5.8 ms	1.0	1	5,161	30.0 s
Sustained decode	16K	2	Valid	302.2	151.9	171 ms	6.6 ms	2.0	2	9,066	30.0 s
Sustained decode	16K	4	Valid	557.0	140.4	237 ms	7.1 ms	4.0	4	16,712	30.0 s
Sustained decode	16K	8	Valid	966.7	122.2	305 ms	8.2 ms	7.9	8	28,997	30.0 s
Sustained decode	16K	16	Valid	1,651.4	103.7	650 ms	9.6 ms	16.0	0	49,537	30.0 s
Sustained decode	16K	32	Valid	2,723.5	85.3	1,001 ms	11.7 ms	32.0	0	81,698	30.0 s
Sustained decode	16K	64	Valid	4,082.9	63.0	1,314 ms	15.9 ms	64.0	0	122,497	30.0 s
Sustained decode	16K	128	Valid	5,865.4	44.3	2,326 ms	22.6 ms	128.0	0	175,998	30.0 s
Sustained decode	32K	1	Valid	169.0	170.3	161 ms	5.9 ms	1.0	1	5,070	30.0 s
Sustained decode	32K	2	Valid	297.8	149.9	194 ms	6.7 ms	2.0	2	8,934	30.0 s
Sustained decode	32K	4	Valid	544.2	137.4	289 ms	7.3 ms	4.0	4	16,327	30.0 s
Sustained decode	32K	8	Valid	962.1	121.4	324 ms	8.2 ms	7.7	8	28,864	30.0 s
Sustained decode	32K	16	Valid	1,591.8	99.9	609 ms	10.0 ms	16.0	0	47,741	30.0 s
Sustained decode	32K	32	Valid	2,523.3	78.4	913 ms	12.8 ms	32.0	0	75,683	30.0 s
Sustained decode	32K	64	Valid	3,703.2	57.0	1,490 ms	17.6 ms	64.0	0	111,104	30.0 s
Sustained decode	32K	128	Valid	5,166.1	38.8	2,641 ms	25.8 ms	128.0	0	154,927	30.0 s
Sustained decode	64K	1	Valid	163.1	164.9	238 ms	6.1 ms	1.0	1	4,892	30.0 s
Sustained decode	64K	2	Valid	288.3	145.3	251 ms	6.9 ms	2.0	2	8,650	30.0 s
Sustained decode	64K	4	Valid	526.3	132.9	328 ms	7.5 ms	4.0	4	15,792	30.0 s
Sustained decode	64K	8	Valid	909.2	113.3	548 ms	8.8 ms	8.0	0	27,271	30.0 s
Sustained decode	64K	16	Valid	1,496.8	92.6	908 ms	10.8 ms	16.0	0	44,897	30.0 s
Sustained decode	64K	32	Valid	2,205.5	67.6	1,567 ms	14.8 ms	32.0	0	66,145	30.0 s
Sustained decode	64K	64	Valid	3,112.4	46.3	2,930 ms	21.6 ms	64.0	0	93,385	30.0 s
Sustained decode	64K	128	Valid	4,016.1	29.0	5,491 ms	34.5 ms	128.0	0	120,492	30.0 s
Sustained decode	128K	1	Valid	152.7	155.4	360 ms	6.4 ms	1.0	1	4,580	30.0 s
Sustained decode	128K	2	Valid	271.1	138.1	438 ms	7.2 ms	2.0	2	8,135	30.0 s
Sustained decode	128K	4	Valid	495.0	125.6	490 ms	8.0 ms	4.0	4	14,849	30.0 s
Sustained decode	128K	8	Valid	738.8	103.4	1,109 ms	9.7 ms	8.0	0	22,163	30.0 s
Sustained decode	128K	16	Valid	1,274.6	79.0	1,760 ms	12.7 ms	16.0	0	38,240	30.0 s
Sustained decode	128K	32	Valid	1,825.2	56.0	3,408 ms	17.8 ms	32.0	0	54,755	30.0 s
Sustained decode	128K	64	Valid	2,391.8	36.1	6,607 ms	27.7 ms	64.0	0	71,745	30.0 s
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	135.9	140.2	550 ms	7.1 ms	1.0	1	4,077	30.0 s
Sustained decode	251K	2	Valid	244.7	134.3	634 ms	7.5 ms	2.0	2	7,340	30.0 s
Sustained decode	251K	4	Valid	458.2	114.9	1,116 ms	8.7 ms	4.0	0	13,744	30.0 s
Sustained decode	251K	8	Valid	709.3	89.2	1,886 ms	11.2 ms	8.0	0	21,272	30.0 s
Sustained decode	251K	16	Valid	1,039.7	65.6	3,402 ms	15.3 ms	16.0	0	31,186	30.0 s
Sustained decode	251K	32	Valid	1,359.3	42.7	6,428 ms	23.4 ms	32.0	0	40,769	30.0 s
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.