Speed test matrix

Ling-3.0-tiny-int4 on H200 NVL 141GB

vllm · 0.27.2rc1.dev113+g5cecfc013 · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 3, 2026, 20:37 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.27.2rc1.dev113+g5cecfc013
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 5cecfc013750

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-3.0-tiny-int4 --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --reasoning-parser ling3 --enable-auto-tool-choice --tool-call-parser ling3 --max-model-len 131072 --gpu-memory-utilization 0.884376 --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-cache-mode align

Prefill

prompt tok/s by context length

8K context57.2K143 ms TTFT

16K context52.2K310 ms TTFT

32K context60.9K530 ms TTFT

64K context59.5K1,082 ms TTFT

123K context52.2K2,365 ms TTFT

128K context50K2,573 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	327.868 ms TTFT	591.2106 ms TTFT	1,122.0125 ms TTFT	1,605.4158 ms TTFT	3,784.6194 ms TTFT	6,321.1248 ms TTFT	9,848.7291 ms TTFT	15K816 ms TTFT
16K	322.1109 ms TTFT	581.5152 ms TTFT	1,101.2217 ms TTFT	1,562.5314 ms TTFT	3,594.4373 ms TTFT	5,869.3491 ms TTFT	8,731.2797 ms TTFT	12.9K1,401 ms TTFT
32K	312.9127 ms TTFT	567.5211 ms TTFT	1,067.0280 ms TTFT	1,503.0340 ms TTFT	3,338.7449 ms TTFT	5,154.0669 ms TTFT	7,581.71,282 ms TTFT	10.2K2,330 ms TTFT
64K	296.1193 ms TTFT	539.2215 ms TTFT	1,011.1215 ms TTFT	1,407.2341 ms TTFT	2,888.2519 ms TTFT	4,100.2645 ms TTFT	5,750.52,673 ms TTFT	7,120.04,700 ms TTFT
123K	268.0321 ms TTFT	491.5390 ms TTFT	935.3476 ms TTFT	1,251.5490 ms TTFT	2,323.0473 ms TTFT	3,239.72,831 ms TTFT	3,947.65,381 ms TTFT	4,456.210,552 ms TTFT

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	57.2K	—	143 ms	—	—	1	—	—
Prefill	16K	1	Valid	52.2K	—	310 ms	—	—	1	—	—
Prefill	32K	1	Valid	60.9K	—	530 ms	—	—	1	—	—
Prefill	64K	1	Valid	59.5K	—	1,082 ms	—	—	1	—	—
Prefill	123K	1	Valid	52.2K	—	2,365 ms	—	—	1	—	—
Prefill	128K	1	Valid	50K	—	2,573 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	327.8	329.5	68 ms	3.0 ms	1.0	2	9,832	30.0 s
Sustained decode	8K	2	Valid	591.2	297.7	106 ms	3.4 ms	2.0	4	17,734	30.0 s
Sustained decode	8K	4	Valid	1,122.0	283.0	125 ms	3.5 ms	4.0	8	33,659	30.0 s
Sustained decode	8K	8	Valid	1,605.4	201.9	158 ms	5.0 ms	8.0	8	48,158	30.0 s
Sustained decode	8K	16	Valid	3,784.6	238.2	194 ms	4.2 ms	16.0	16	113,540	30.0 s
Sustained decode	8K	32	Valid	6,321.1	199.4	248 ms	5.0 ms	32.0	32	189,617	30.0 s
Sustained decode	8K	64	Valid	9,848.7	155.0	291 ms	6.5 ms	63.1	64	295,449	30.0 s
Sustained decode	8K	128	Valid	15K	116.7	816 ms	8.6 ms	128.0	0	449,936	30.0 s
Sustained decode	16K	1	Valid	322.1	324.7	109 ms	3.1 ms	1.0	2	9,662	30.0 s
Sustained decode	16K	2	Valid	581.5	293.9	152 ms	3.4 ms	2.0	4	17,444	30.0 s
Sustained decode	16K	4	Valid	1,101.2	279.4	217 ms	3.6 ms	4.0	8	33,056	30.0 s
Sustained decode	16K	8	Valid	1,562.5	198.0	314 ms	5.1 ms	8.0	8	46,877	30.0 s
Sustained decode	16K	16	Valid	3,594.4	227.4	373 ms	4.4 ms	16.0	16	107,888	30.0 s
Sustained decode	16K	32	Valid	5,869.3	186.4	491 ms	5.4 ms	31.5	32	176,070	30.0 s
Sustained decode	16K	64	Valid	8,731.2	139.5	797 ms	7.2 ms	62.7	64	261,929	30.0 s
Sustained decode	16K	128	Valid	12.9K	99.3	1,401 ms	10.1 ms	128.0	0	388,358	30.0 s
Sustained decode	32K	1	Valid	312.9	315.8	127 ms	3.2 ms	1.0	2	9,388	30.0 s
Sustained decode	32K	2	Valid	567.5	287.9	211 ms	3.5 ms	1.9	4	17,032	30.0 s
Sustained decode	32K	4	Valid	1,067.0	272.0	280 ms	3.7 ms	4.0	8	32,008	30.0 s
Sustained decode	32K	8	Valid	1,503.0	190.3	340 ms	5.3 ms	7.7	8	45,088	30.0 s
Sustained decode	32K	16	Valid	3,338.7	211.4	449 ms	4.7 ms	15.8	16	100,153	30.0 s
Sustained decode	32K	32	Valid	5,154.0	163.9	669 ms	6.1 ms	31.4	32	154,614	30.0 s
Sustained decode	32K	64	Valid	7,581.7	116.4	1,282 ms	8.6 ms	64.0	0	227,534	30.0 s
Sustained decode	32K	128	Valid	10.2K	76.8	2,330 ms	13.0 ms	128.0	0	306,448	30.0 s
Sustained decode	64K	1	Valid	296.1	300.4	193 ms	3.3 ms	1.0	2	8,882	30.0 s
Sustained decode	64K	2	Valid	539.2	273.8	215 ms	3.7 ms	2.0	4	16,175	30.0 s
Sustained decode	64K	4	Valid	1,011.1	256.4	215 ms	3.9 ms	4.0	8	30,331	30.0 s
Sustained decode	64K	8	Valid	1,407.2	177.2	341 ms	5.6 ms	8.0	8	42,213	30.0 s
Sustained decode	64K	16	Valid	2,888.2	182.5	519 ms	5.5 ms	15.8	16	86,641	30.0 s
Sustained decode	64K	32	Valid	4,100.2	130.9	645 ms	7.6 ms	31.4	32	123,004	30.0 s
Sustained decode	64K	64	Valid	5,750.5	87.1	2,673 ms	11.5 ms	64.0	0	172,485	30.0 s
Sustained decode	64K	128	Valid	7,120.0	52.0	4,700 ms	19.2 ms	128.0	0	213,643	30.0 s
Sustained decode	123K	1	Valid	268.0	274.1	321 ms	3.6 ms	1.0	2	8,045	30.0 s
Sustained decode	123K	2	Valid	491.5	252.8	390 ms	4.0 ms	2.0	4	14,753	30.0 s
Sustained decode	123K	4	Valid	935.3	237.1	476 ms	4.2 ms	3.9	4	28,059	30.0 s
Sustained decode	123K	8	Valid	1,251.5	158.7	490 ms	6.3 ms	8.0	8	37,540	30.0 s
Sustained decode	123K	16	Valid	2,323.0	146.9	473 ms	6.8 ms	15.8	16	69,680	30.0 s
Sustained decode	123K	32	Valid	3,239.7	99.7	2,831 ms	10.0 ms	32.0	0	97,187	30.0 s
Sustained decode	123K	64	Valid	3,947.6	59.9	5,381 ms	16.7 ms	64.0	0	118,409	30.0 s
Sustained decode	123K	128	Valid	4,456.2	33.6	10,552 ms	29.8 ms	128.0	0	133,767	30.0 s