Speed test matrix

Ling-2.6-flash-base on H200 NVL 141GB

vllm · 0.27.2rc1.dev113+g5cecfc013 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 3, 2026, 10:29 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.27.2rc1.dev113+g5cecfc013
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 5cecfc013750

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-2.6-flash-base --served-model-name ling --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 262144 --gpu-memory-utilization 0.884765 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context22.2K369 ms TTFT

16K context4,6743,469 ms TTFT

32K context12.4K2,600 ms TTFT

64K context22.4K2,871 ms TTFT

128K context20.5K6,258 ms TTFT

251K context17.5K14,428 ms TTFT

256K context12.2K20,982 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	170.568 ms TTFT	282.4118 ms TTFT	458.6160 ms TTFT	661.1230 ms TTFT	1,124.0321 ms TTFT	1,715.6492 ms TTFT	2,743.9836 ms TTFT	3,665.71,346 ms TTFT
16K	169.2167 ms TTFT	280.6240 ms TTFT	461.5321 ms TTFT	653.52,095 ms TTFT	1,110.7874 ms TTFT	1,756.41,536 ms TTFT	2,759.02,019 ms TTFT	3,572.63,595 ms TTFT
32K	167.5181 ms TTFT	277.0272 ms TTFT	456.43,106 ms TTFT	659.8574 ms TTFT	1,097.2855 ms TTFT	1,752.81,209 ms TTFT	2,592.41,893 ms TTFT	Over capacity
64K	164.4242 ms TTFT	273.2367 ms TTFT	449.8464 ms TTFT	635.6760 ms TTFT	1,114.81,101 ms TTFT	1,609.61,663 ms TTFT	Over capacity	Over capacity
128K	157.8370 ms TTFT	264.1391 ms TTFT	437.3702 ms TTFT	615.81,166 ms TTFT	1,017.21,873 ms TTFT	1,531.63,453 ms TTFT	Over capacity	Over capacity
251K	146.7583 ms TTFT	248.3638 ms TTFT	417.11,173 ms TTFT	572.31,947 ms TTFT	918.13,489 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	22.2K	—	369 ms	—	—	1	—	—
Prefill	16K	1	Valid	4,674.0	—	3,469 ms	—	—	1	—	—
Prefill	32K	1	Valid	12.4K	—	2,600 ms	—	—	1	—	—
Prefill	64K	1	Valid	22.4K	—	2,871 ms	—	—	1	—	—
Prefill	128K	1	Valid	20.5K	—	6,258 ms	—	—	1	—	—
Prefill	251K	1	Valid	17.5K	—	14,428 ms	—	—	1	—	—
Prefill	256K	1	Valid	12.2K	—	20,982 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	170.5	171.1	68 ms	5.8 ms	1.0	1	5,113	30.0 s
Sustained decode	8K	2	Valid	282.4	142.3	118 ms	7.0 ms	2.0	2	8,470	30.0 s
Sustained decode	8K	4	Valid	458.6	114.8	160 ms	8.7 ms	4.0	0	13,756	30.0 s
Sustained decode	8K	8	Valid	661.1	82.9	230 ms	12.1 ms	8.0	0	19,833	30.0 s
Sustained decode	8K	16	Valid	1,124.0	70.9	321 ms	14.1 ms	16.0	0	33,712	30.0 s
Sustained decode	8K	32	Valid	1,715.6	54.3	492 ms	18.4 ms	32.0	0	51,456	30.0 s
Sustained decode	8K	64	Valid	2,743.9	43.4	836 ms	23.0 ms	64.0	0	82,302	30.0 s
Sustained decode	8K	128	Valid	3,665.7	28.9	1,346 ms	34.6 ms	128.0	0	109,952	30.0 s
Sustained decode	16K	1	Valid	169.2	170.4	167 ms	5.9 ms	1.0	1	5,074	30.0 s
Sustained decode	16K	2	Valid	280.6	141.9	240 ms	7.0 ms	2.0	2	8,418	30.0 s
Sustained decode	16K	4	Valid	461.5	115.6	321 ms	8.7 ms	4.0	0	13,844	30.0 s
Sustained decode	16K	8	Valid	653.5	82.0	2,095 ms	12.2 ms	8.0	0	19,600	30.0 s
Sustained decode	16K	16	Valid	1,110.7	70.1	874 ms	14.3 ms	16.0	0	33,312	30.0 s
Sustained decode	16K	32	Valid	1,756.4	55.5	1,536 ms	18.0 ms	32.0	0	52,672	30.0 s
Sustained decode	16K	64	Valid	2,759.0	42.7	2,019 ms	23.4 ms	64.0	0	82,752	30.0 s
Sustained decode	16K	128	Valid	3,572.6	26.8	3,595 ms	37.3 ms	128.0	0	107,136	30.0 s
Sustained decode	32K	1	Valid	167.5	168.8	181 ms	5.9 ms	1.0	1	5,023	30.0 s
Sustained decode	32K	2	Valid	277.0	140.9	272 ms	7.1 ms	2.0	2	8,310	30.0 s
Sustained decode	32K	4	Valid	456.4	114.3	3,106 ms	8.7 ms	4.0	0	13,692	30.0 s
Sustained decode	32K	8	Valid	659.8	82.6	574 ms	12.1 ms	8.0	0	19,792	30.0 s
Sustained decode	32K	16	Valid	1,097.2	69.2	855 ms	14.4 ms	16.0	0	32,912	30.0 s
Sustained decode	32K	32	Valid	1,752.8	55.1	1,209 ms	18.1 ms	32.0	0	52,576	30.0 s
Sustained decode	32K	64	Valid	2,592.4	38.5	1,893 ms	26.0 ms	64.0	0	77,759	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	164.4	166.2	242 ms	6.0 ms	1.0	1	4,935	30.0 s
Sustained decode	64K	2	Valid	273.2	139.4	367 ms	7.2 ms	1.9	2	8,196	30.0 s
Sustained decode	64K	4	Valid	449.8	112.5	464 ms	8.9 ms	4.0	0	13,492	30.0 s
Sustained decode	64K	8	Valid	635.6	79.5	760 ms	12.6 ms	8.0	0	19,064	30.0 s
Sustained decode	64K	16	Valid	1,114.8	69.5	1,101 ms	14.4 ms	16.0	0	33,440	30.0 s
Sustained decode	64K	32	Valid	1,609.6	48.2	1,663 ms	20.7 ms	32.0	0	48,289	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	157.8	160.4	370 ms	6.2 ms	1.0	1	4,735	30.0 s
Sustained decode	128K	2	Valid	264.1	133.9	391 ms	7.5 ms	1.9	2	7,921	30.0 s
Sustained decode	128K	4	Valid	437.3	109.2	702 ms	9.2 ms	4.0	0	13,116	30.0 s
Sustained decode	128K	8	Valid	615.8	76.6	1,166 ms	13.1 ms	8.0	0	18,472	30.0 s
Sustained decode	128K	16	Valid	1,017.2	62.8	1,873 ms	15.9 ms	16.0	0	30,512	30.0 s
Sustained decode	128K	32	Valid	1,531.6	46.7	3,453 ms	21.4 ms	32.0	0	45,952	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	146.7	150.9	583 ms	6.6 ms	1.0	1	4,402	30.0 s
Sustained decode	251K	2	Valid	248.3	131.7	638 ms	7.6 ms	1.9	2	7,450	30.0 s
Sustained decode	251K	4	Valid	417.1	104.6	1,173 ms	9.6 ms	4.0	0	12,512	30.0 s
Sustained decode	251K	8	Valid	572.3	71.8	1,947 ms	13.9 ms	8.0	0	17,168	30.0 s
Sustained decode	251K	16	Valid	918.1	57.5	3,489 ms	17.4 ms	16.0	0	27,553	30.0 s
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.