Speed test matrix

Kimi-K2.5 on H200 SXM 141GB ×8

vllm · 0.29.0 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 16, 2026, 23:52 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 8
VRAM: 1128 GB (8 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: moonshotai/Kimi-K2.5
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve moonshotai/Kimi-K2.5 --served-model-name moonshotai/Kimi-K2.5 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --max-num-batched-tokens 16384 --kernel-config.enable_flashinfer_autotune=False

Prefill

prompt tok/s by context length

8K context17.8K460 ms TTFT

16K context18.8K863 ms TTFT

32K context17.6K1,835 ms TTFT

64K context15.4K4,175 ms TTFT

128K context12.4K10,411 ms TTFT

251K context8,92228,290 ms TTFT

256K context8,86429,041 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	10.8128 ms TTFT	21.6237 ms TTFT	43.1344 ms TTFT	86.1354 ms TTFT	172.4472 ms TTFT	340.4707 ms TTFT	Over capacity	Over capacity
16K	10.8151 ms TTFT	21.6271 ms TTFT	43.1359 ms TTFT	86.3461 ms TTFT	172.5626 ms TTFT	Over capacity	Over capacity	Over capacity
32K	10.8193 ms TTFT	21.7308 ms TTFT	43.1450 ms TTFT	86.3658 ms TTFT	172.7971 ms TTFT	Over capacity	Over capacity	Over capacity
64K	10.8279 ms TTFT	21.6436 ms TTFT	43.0640 ms TTFT	86.1995 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
128K	10.8472 ms TTFT	21.6670 ms TTFT	43.11,132 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	10.8802 ms TTFT	21.61,185 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	17.8K	—	460 ms	—	—	1	—	—
Prefill	16K	1	Valid	18.8K	—	863 ms	—	—	1	—	—
Prefill	32K	1	Valid	17.6K	—	1,835 ms	—	—	1	—	—
Prefill	64K	1	Valid	15.4K	—	4,175 ms	—	—	1	—	—
Prefill	128K	1	Valid	12.4K	—	10,411 ms	—	—	1	—	—
Prefill	251K	1	Valid	8,922.0	—	28,290 ms	—	—	1	—	—
Prefill	256K	1	Valid	8,864.0	—	29,041 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	10.8	10.8	128 ms	92.7 ms	1.0	0	324	30.0 s
Sustained decode	8K	2	Valid	21.6	10.8	237 ms	92.8 ms	2.0	0	646	30.0 s
Sustained decode	8K	4	Valid	43.1	10.8	344 ms	92.9 ms	4.0	0	1,293	30.0 s
Sustained decode	8K	8	Valid	86.1	10.8	354 ms	92.9 ms	8.0	0	2,584	30.0 s
Sustained decode	8K	16	Valid	172.4	10.8	472 ms	93.0 ms	16.0	0	5,168	30.0 s
Sustained decode	8K	32	Valid	340.4	10.6	707 ms	94.2 ms	32.0	0	10,208	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	10.8	10.8	151 ms	92.9 ms	1.0	0	323	30.0 s
Sustained decode	16K	2	Valid	21.6	10.8	271 ms	92.5 ms	2.0	0	648	30.0 s
Sustained decode	16K	4	Valid	43.1	10.8	359 ms	92.9 ms	4.0	0	1,293	30.0 s
Sustained decode	16K	8	Valid	86.3	10.8	461 ms	92.9 ms	8.0	0	2,584	29.9 s
Sustained decode	16K	16	Valid	172.5	10.7	626 ms	93.0 ms	16.0	0	5,167	30.0 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	10.8	10.8	193 ms	92.5 ms	1.0	0	324	30.0 s
Sustained decode	32K	2	Valid	21.7	10.8	308 ms	92.7 ms	2.0	0	649	30.0 s
Sustained decode	32K	4	Valid	43.1	10.8	450 ms	93.0 ms	4.0	0	1,293	30.0 s
Sustained decode	32K	8	Valid	86.3	10.8	658 ms	93.0 ms	8.0	0	2,584	30.0 s
Sustained decode	32K	16	Valid	172.7	10.8	971 ms	92.8 ms	16.0	0	5,170	29.9 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	10.8	10.8	279 ms	92.6 ms	1.0	0	324	30.0 s
Sustained decode	64K	2	Valid	21.6	10.8	436 ms	92.8 ms	2.0	0	648	30.0 s
Sustained decode	64K	4	Valid	43.0	10.7	640 ms	93.1 ms	4.0	0	1,289	30.0 s
Sustained decode	64K	8	Valid	86.1	10.7	995 ms	93.2 ms	8.0	0	2,576	29.9 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	10.8	10.8	472 ms	92.3 ms	1.0	0	325	30.0 s
Sustained decode	128K	2	Valid	21.6	10.8	670 ms	92.7 ms	2.0	0	646	29.9 s
Sustained decode	128K	4	Valid	43.1	10.8	1,132 ms	93.0 ms	4.0	0	1,292	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	10.8	10.8	802 ms	92.7 ms	1.0	0	324	30.0 s
Sustained decode	251K	2	Valid	21.6	10.8	1,185 ms	92.8 ms	2.0	0	648	30.0 s
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.