Speed test matrix

Ling-2.6-flash on H200 NVL 141GB ×2

vllm · 0.27.2rc1.dev113+g5cecfc013 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 3, 2026, 13:12 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.27.2rc1.dev113+g5cecfc013
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 5cecfc013750

Run

Served model: ling
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-inclusionai-vllm/bin/vllm" serve inclusionAI/Ling-2.6-flash --served-model-name ling --trust-remote-code --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser hermes --max-model-len 131072 --gpu-memory-utilization 0.868619 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context28.8K284 ms TTFT

16K context6,3832,540 ms TTFT

32K context31.3K1,029 ms TTFT

64K context31.7K2,028 ms TTFT

123K context30K4,116 ms TTFT

128K context19.8K6,478 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	176.857 ms TTFT	292.777 ms TTFT	462.9110 ms TTFT	678.2183 ms TTFT	1,136.0273 ms TTFT	1,801.9406 ms TTFT	2,925.0556 ms TTFT	4,663.2957 ms TTFT
16K	175.3116 ms TTFT	292.4118 ms TTFT	460.2562 ms TTFT	670.7317 ms TTFT	1,181.4429 ms TTFT	1,878.6679 ms TTFT	2,853.91,096 ms TTFT	4,525.11,792 ms TTFT
32K	173.7119 ms TTFT	289.8134 ms TTFT	460.5190 ms TTFT	660.9310 ms TTFT	1,099.5468 ms TTFT	1,704.3773 ms TTFT	2,659.61,246 ms TTFT	Over capacity
64K	170.1188 ms TTFT	282.9234 ms TTFT	448.4304 ms TTFT	644.8480 ms TTFT	1,048.0797 ms TTFT	1,598.11,258 ms TTFT	Over capacity	Over capacity
123K	163.7306 ms TTFT	274.5340 ms TTFT	439.7543 ms TTFT	623.4861 ms TTFT	979.81,522 ms TTFT	1,475.62,735 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	28.8K	—	284 ms	—	—	1	—	—
Prefill	16K	1	Valid	6,383.0	—	2,540 ms	—	—	1	—	—
Prefill	32K	1	Valid	31.3K	—	1,029 ms	—	—	1	—	—
Prefill	64K	1	Valid	31.7K	—	2,028 ms	—	—	1	—	—
Prefill	123K	1	Valid	30K	—	4,116 ms	—	—	1	—	—
Prefill	128K	1	Valid	19.8K	—	6,478 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	176.8	177.3	57 ms	5.6 ms	1.0	1	5,304	30.0 s
Sustained decode	8K	2	Valid	292.7	147.0	77 ms	6.8 ms	2.0	2	8,785	30.0 s
Sustained decode	8K	4	Valid	462.9	115.9	110 ms	8.6 ms	4.0	0	13,884	30.0 s
Sustained decode	8K	8	Valid	678.2	84.8	183 ms	11.8 ms	8.0	0	20,352	30.0 s
Sustained decode	8K	16	Valid	1,136.0	71.7	273 ms	13.9 ms	16.0	0	34,080	30.0 s
Sustained decode	8K	32	Valid	1,801.9	57.4	406 ms	17.4 ms	32.0	0	54,048	30.0 s
Sustained decode	8K	64	Valid	2,925.0	46.3	556 ms	21.6 ms	64.0	0	87,742	30.0 s
Sustained decode	8K	128	Valid	4,663.2	36.7	957 ms	27.3 ms	128.0	0	139,902	30.0 s
Sustained decode	16K	1	Valid	175.3	176.2	116 ms	5.7 ms	1.0	1	5,259	30.0 s
Sustained decode	16K	2	Valid	292.4	146.7	118 ms	6.8 ms	2.0	2	8,778	30.0 s
Sustained decode	16K	4	Valid	460.2	115.1	562 ms	8.7 ms	4.0	0	13,804	30.0 s
Sustained decode	16K	8	Valid	670.7	83.9	317 ms	11.9 ms	8.0	0	20,120	30.0 s
Sustained decode	16K	16	Valid	1,181.4	74.2	429 ms	13.5 ms	16.0	0	35,440	30.0 s
Sustained decode	16K	32	Valid	1,878.6	59.2	679 ms	16.9 ms	32.0	0	56,352	30.0 s
Sustained decode	16K	64	Valid	2,853.9	44.9	1,096 ms	22.3 ms	64.0	0	85,632	30.0 s
Sustained decode	16K	128	Valid	4,525.1	35.2	1,792 ms	28.4 ms	128.0	0	135,808	30.0 s
Sustained decode	32K	1	Valid	173.7	174.6	119 ms	5.7 ms	1.0	1	5,212	30.0 s
Sustained decode	32K	2	Valid	289.8	145.7	134 ms	6.9 ms	2.0	2	8,693	30.0 s
Sustained decode	32K	4	Valid	460.5	115.1	190 ms	8.7 ms	4.0	0	13,820	30.0 s
Sustained decode	32K	8	Valid	660.9	82.7	310 ms	12.1 ms	8.0	0	19,824	30.0 s
Sustained decode	32K	16	Valid	1,099.5	69.3	468 ms	14.4 ms	16.0	0	32,976	30.0 s
Sustained decode	32K	32	Valid	1,704.3	53.7	773 ms	18.6 ms	32.0	0	51,136	30.0 s
Sustained decode	32K	64	Valid	2,659.6	41.8	1,246 ms	23.9 ms	64.0	0	79,808	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	170.1	171.6	188 ms	5.8 ms	1.0	1	5,103	30.0 s
Sustained decode	64K	2	Valid	282.9	143.0	234 ms	7.0 ms	1.9	2	8,485	30.0 s
Sustained decode	64K	4	Valid	448.4	112.2	304 ms	8.9 ms	4.0	0	13,452	30.0 s
Sustained decode	64K	8	Valid	644.8	80.5	480 ms	12.4 ms	8.0	0	19,344	30.0 s
Sustained decode	64K	16	Valid	1,048.0	65.8	797 ms	15.2 ms	16.0	0	31,440	30.0 s
Sustained decode	64K	32	Valid	1,598.1	50.0	1,258 ms	20.0 ms	32.0	0	47,936	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	163.7	166.0	306 ms	6.0 ms	1.0	1	4,914	30.0 s
Sustained decode	123K	2	Valid	274.5	139.8	340 ms	7.2 ms	1.9	2	8,238	30.0 s
Sustained decode	123K	4	Valid	439.7	110.2	543 ms	9.1 ms	4.0	0	13,192	30.0 s
Sustained decode	123K	8	Valid	623.4	78.0	861 ms	12.8 ms	8.0	0	18,712	30.0 s
Sustained decode	123K	16	Valid	979.8	62.0	1,522 ms	16.1 ms	16.0	0	29,392	30.0 s
Sustained decode	123K	32	Valid	1,475.6	46.6	2,735 ms	21.5 ms	32.0	0	44,255	30.0 s
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.