Speed test matrix

Ling-2.6-flash-int4 on H200 NVL 141GB

vllm · 0.27.2rc1.dev113+g5cecfc013 · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 3, 2026, 12:04 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.27.2rc1.dev113+g5cecfc013
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 5cecfc013750

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-2.6-flash-int4 --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.884685 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context16.1K510 ms TTFT

16K context16K1,011 ms TTFT

32K context16.4K1,969 ms TTFT

64K context16.3K3,952 ms TTFT

123K context15.3K8,076 ms TTFT

128K context15.2K8,471 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	153.475 ms TTFT	268.1109 ms TTFT	472.9152 ms TTFT	774.9225 ms TTFT	1,214.7282 ms TTFT	2,027.0495 ms TTFT	3,059.5679 ms TTFT	4,238.61,428 ms TTFT
16K	152.4170 ms TTFT	264.8253 ms TTFT	462.6375 ms TTFT	769.7636 ms TTFT	1,241.91,101 ms TTFT	1,968.21,993 ms TTFT	3,062.42,638 ms TTFT	4,154.34,629 ms TTFT
32K	151.4179 ms TTFT	265.1277 ms TTFT	456.6385 ms TTFT	761.3616 ms TTFT	1,198.3993 ms TTFT	1,935.31,759 ms TTFT	2,884.12,928 ms TTFT	3,894.84,080 ms TTFT
64K	148.7246 ms TTFT	258.7379 ms TTFT	450.9574 ms TTFT	720.6789 ms TTFT	1,117.91,315 ms TTFT	1,765.02,147 ms TTFT	2,584.73,343 ms TTFT	3,364.66,007 ms TTFT
123K	143.5380 ms TTFT	249.8419 ms TTFT	438.5703 ms TTFT	681.21,224 ms TTFT	1,036.02,011 ms TTFT	1,588.92,955 ms TTFT	2,226.15,138 ms TTFT	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	16.1K	—	510 ms	—	—	1	—	—
Prefill	16K	1	Valid	16K	—	1,011 ms	—	—	1	—	—
Prefill	32K	1	Valid	16.4K	—	1,969 ms	—	—	1	—	—
Prefill	64K	1	Valid	16.3K	—	3,952 ms	—	—	1	—	—
Prefill	123K	1	Valid	15.3K	—	8,076 ms	—	—	1	—	—
Prefill	128K	1	Valid	15.2K	—	8,471 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	153.4	154.0	75 ms	6.5 ms	1.0	1	4,603	30.0 s
Sustained decode	8K	2	Valid	268.1	135.7	109 ms	7.4 ms	2.0	2	8,042	30.0 s
Sustained decode	8K	4	Valid	472.9	118.6	152 ms	8.4 ms	4.0	0	14,184	30.0 s
Sustained decode	8K	8	Valid	774.9	97.4	225 ms	10.3 ms	8.0	0	23,256	30.0 s
Sustained decode	8K	16	Valid	1,214.7	76.8	282 ms	13.0 ms	16.0	0	36,432	30.0 s
Sustained decode	8K	32	Valid	2,027.0	64.2	495 ms	15.6 ms	32.0	0	60,800	30.0 s
Sustained decode	8K	64	Valid	3,059.5	48.3	679 ms	20.7 ms	64.0	0	91,776	30.0 s
Sustained decode	8K	128	Valid	4,238.6	33.3	1,428 ms	30.0 ms	128.0	0	127,104	30.0 s
Sustained decode	16K	1	Valid	152.4	153.7	170 ms	6.5 ms	1.0	1	4,575	30.0 s
Sustained decode	16K	2	Valid	264.8	135.0	253 ms	7.4 ms	1.9	2	7,948	30.0 s
Sustained decode	16K	4	Valid	462.6	116.0	375 ms	8.6 ms	4.0	0	13,884	30.0 s
Sustained decode	16K	8	Valid	769.7	96.6	636 ms	10.3 ms	8.0	0	23,104	30.0 s
Sustained decode	16K	16	Valid	1,241.9	78.5	1,101 ms	12.7 ms	16.0	0	37,248	30.0 s
Sustained decode	16K	32	Valid	1,968.2	62.3	1,993 ms	16.0 ms	32.0	0	59,040	30.0 s
Sustained decode	16K	64	Valid	3,062.4	46.9	2,638 ms	21.3 ms	64.0	0	91,840	30.0 s
Sustained decode	16K	128	Valid	4,154.3	30.3	4,629 ms	33.0 ms	128.0	0	124,672	30.0 s
Sustained decode	32K	1	Valid	151.4	152.8	179 ms	6.5 ms	1.0	1	4,545	30.0 s
Sustained decode	32K	2	Valid	265.1	135.2	277 ms	7.4 ms	1.9	2	7,954	30.0 s
Sustained decode	32K	4	Valid	456.6	114.5	385 ms	8.7 ms	4.0	0	13,697	30.0 s
Sustained decode	32K	8	Valid	761.3	95.7	616 ms	10.5 ms	8.0	0	22,841	30.0 s
Sustained decode	32K	16	Valid	1,198.3	75.5	993 ms	13.2 ms	16.0	0	35,936	30.0 s
Sustained decode	32K	32	Valid	1,935.3	61.3	1,759 ms	16.3 ms	32.0	0	58,048	30.0 s
Sustained decode	32K	64	Valid	2,884.1	45.5	2,928 ms	22.0 ms	64.0	0	86,527	30.0 s
Sustained decode	32K	128	Valid	3,894.8	29.3	4,080 ms	34.1 ms	128.0	0	116,864	30.0 s
Sustained decode	64K	1	Valid	148.7	150.6	246 ms	6.6 ms	1.0	1	4,460	30.0 s
Sustained decode	64K	2	Valid	258.7	132.6	379 ms	7.5 ms	2.0	2	7,764	30.0 s
Sustained decode	64K	4	Valid	450.9	113.1	574 ms	8.8 ms	4.0	0	13,532	30.0 s
Sustained decode	64K	8	Valid	720.6	90.4	789 ms	11.1 ms	8.0	0	21,616	30.0 s
Sustained decode	64K	16	Valid	1,117.9	70.1	1,315 ms	14.3 ms	16.0	0	33,536	30.0 s
Sustained decode	64K	32	Valid	1,765.0	54.8	2,147 ms	18.3 ms	32.0	0	52,927	30.0 s
Sustained decode	64K	64	Valid	2,584.7	38.9	3,343 ms	25.7 ms	64.0	0	77,504	30.0 s
Sustained decode	64K	128	Valid	3,364.6	23.9	6,007 ms	41.8 ms	128.0	0	100,864	30.0 s
Sustained decode	123K	1	Valid	143.5	146.5	380 ms	6.8 ms	1.0	1	4,307	30.0 s
Sustained decode	123K	2	Valid	249.8	126.8	419 ms	7.9 ms	2.0	2	7,493	30.0 s
Sustained decode	123K	4	Valid	438.5	109.7	703 ms	9.1 ms	4.0	0	13,156	30.0 s
Sustained decode	123K	8	Valid	681.2	85.0	1,224 ms	11.8 ms	8.0	0	20,432	30.0 s
Sustained decode	123K	16	Valid	1,036.0	64.0	2,011 ms	15.6 ms	16.0	0	31,071	30.0 s
Sustained decode	123K	32	Valid	1,588.9	48.4	2,955 ms	20.7 ms	32.0	0	47,648	30.0 s
Sustained decode	123K	64	Valid	2,226.1	32.7	5,138 ms	30.6 ms	64.0	0	66,752	30.0 s
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.