Speed test matrix

Kimi-K2.5 on H200 SXM 141GB ×8

vllm · 0.29.0 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 16, 2026, 23:15 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 8
VRAM: 1128 GB (8 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: z-lab/Kimi-K2.5-DFlash
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve moonshotai/Kimi-K2.5 --served-model-name z-lab/Kimi-K2.5-DFlash --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --max-num-batched-tokens 16384 --speculative-config '{"method":"dflash","model":"z-lab/Kimi-K2.5-DFlash","num_speculative_tokens":8}'

Prefill

prompt tok/s by context length

8K context17.8K459 ms TTFT

16K context18.8K864 ms TTFT

32K context17.4K1,858 ms TTFT

64K context15.2K4,244 ms TTFT

128K context12.2K10,528 ms TTFT

251K context8,82628,597 ms TTFT

256K context8,79229,282 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	26.2135 ms TTFT	47.3256 ms TTFT	106.2379 ms TTFT	200.6389 ms TTFT	401.3511 ms TTFT	Over capacity	Over capacity	Over capacity
16K	26.5158 ms TTFT	52.4282 ms TTFT	104.4390 ms TTFT	200.0504 ms TTFT	410.4649 ms TTFT	Over capacity	Over capacity	Over capacity
32K	26.5199 ms TTFT	53.4319 ms TTFT	95.7485 ms TTFT	194.6662 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
64K	26.8283 ms TTFT	52.4454 ms TTFT	98.9666 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
128K	28.5453 ms TTFT	52.2674 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	26.1787 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	17.8K	—	459 ms	—	—	1	—	—
Prefill	16K	1	Valid	18.8K	—	864 ms	—	—	1	—	—
Prefill	32K	1	Valid	17.4K	—	1,858 ms	—	—	1	—	—
Prefill	64K	1	Valid	15.2K	—	4,244 ms	—	—	1	—	—
Prefill	128K	1	Valid	12.2K	—	10,528 ms	—	—	1	—	—
Prefill	251K	1	Valid	8,826.0	—	28,597 ms	—	—	1	—	—
Prefill	256K	1	Valid	8,792.0	—	29,282 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	26.2	27.5	135 ms	36.4 ms	1.0	0	786	30.0 s
Sustained decode	8K	2	Valid	47.3	25.3	256 ms	39.5 ms	2.0	0	1,413	29.9 s
Sustained decode	8K	4	Valid	106.2	28.2	379 ms	35.5 ms	4.0	0	3,183	30.0 s
Sustained decode	8K	8	Valid	200.6	25.6	389 ms	39.1 ms	8.0	0	6,003	29.9 s
Sustained decode	8K	16	Valid	401.3	26.4	511 ms	37.9 ms	16.0	0	12,009	29.9 s
Sustained decode	8K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	26.5	28.0	158 ms	35.7 ms	1.0	0	792	29.9 s
Sustained decode	16K	2	Valid	52.4	27.3	282 ms	36.6 ms	2.0	0	1,569	29.9 s
Sustained decode	16K	4	Valid	104.4	28.2	390 ms	35.4 ms	4.0	0	3,129	30.0 s
Sustained decode	16K	8	Valid	200.0	25.8	504 ms	38.8 ms	8.0	0	5,980	29.9 s
Sustained decode	16K	16	Valid	410.4	26.5	649 ms	37.8 ms	16.0	0	12,291	29.9 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	26.5	28.1	199 ms	35.6 ms	1.0	0	792	29.9 s
Sustained decode	32K	2	Valid	53.4	27.9	319 ms	35.9 ms	2.0	0	1,596	29.9 s
Sustained decode	32K	4	Valid	95.7	25.3	485 ms	39.6 ms	4.0	0	2,862	29.9 s
Sustained decode	32K	8	Valid	194.6	25.6	662 ms	39.1 ms	8.0	0	5,827	29.9 s
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	26.8	28.6	283 ms	34.9 ms	1.0	0	803	29.9 s
Sustained decode	64K	2	Valid	52.4	27.4	454 ms	36.5 ms	2.0	0	1,567	29.9 s
Sustained decode	64K	4	Valid	98.9	25.6	666 ms	39.1 ms	4.0	0	2,961	29.9 s
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	28.5	29.3	453 ms	34.1 ms	1.0	0	853	29.9 s
Sustained decode	128K	2	Valid	52.2	27.5	674 ms	36.4 ms	2.0	0	1,562	29.9 s
Sustained decode	128K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	26.1	27.7	787 ms	36.1 ms	1.0	0	781	30.0 s
Sustained decode	251K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.