Speed test matrix

Kimi-K2.7-Code-NVFP4 on H200 SXM 141GB ×8

vllm · 0.29.0 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 17, 2026, 01:46 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 8
VRAM: 1128 GB (8 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: NVFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: nvidia/Kimi-K2.7-Code-NVFP4
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve nvidia/Kimi-K2.7-Code-NVFP4 --served-model-name nvidia/Kimi-K2.7-Code-NVFP4 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --max-num-batched-tokens 16384 --kernel-config.enable_flashinfer_autotune=False

Prefill

prompt tok/s by context length

8K context17.6K466 ms TTFT

16K context18.7K869 ms TTFT

32K context17.5K1,843 ms TTFT

64K context15.4K4,195 ms TTFT

128K context12.4K10,420 ms TTFT

251K context8,93028,263 ms TTFT

256K context8,87928,995 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	10.2138 ms TTFT	20.4262 ms TTFT	40.8396 ms TTFT	81.9397 ms TTFT	163.8538 ms TTFT	323.0726 ms TTFT	620.11,144 ms TTFT	Over capacity
16K	10.2161 ms TTFT	20.3315 ms TTFT	40.8425 ms TTFT	81.7562 ms TTFT	163.5716 ms TTFT	322.41,097 ms TTFT	Over capacity	Over capacity
32K	10.2199 ms TTFT	20.4333 ms TTFT	40.7531 ms TTFT	81.4740 ms TTFT	163.41,038 ms TTFT	321.51,707 ms TTFT	Over capacity	Over capacity
64K	10.2285 ms TTFT	20.4476 ms TTFT	40.8697 ms TTFT	81.11,084 ms TTFT	162.81,654 ms TTFT	Over capacity	Over capacity	Over capacity
128K	10.2439 ms TTFT	20.4699 ms TTFT	41.01,100 ms TTFT	82.21,702 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	10.2765 ms TTFT	20.51,145 ms TTFT	41.21,889 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	17.6K	—	466 ms	—	—	1	—	—
Prefill	16K	1	Valid	18.7K	—	869 ms	—	—	1	—	—
Prefill	32K	1	Valid	17.5K	—	1,843 ms	—	—	1	—	—
Prefill	64K	1	Valid	15.4K	—	4,195 ms	—	—	1	—	—
Prefill	128K	1	Valid	12.4K	—	10,420 ms	—	—	1	—	—
Prefill	251K	1	Valid	8,930.0	—	28,263 ms	—	—	1	—	—
Prefill	256K	1	Valid	8,879.0	—	28,995 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	10.2	10.2	138 ms	98.2 ms	1.0	0	306	30.0 s
Sustained decode	8K	2	Valid	20.4	10.2	262 ms	98.2 ms	2.0	0	612	30.0 s
Sustained decode	8K	4	Valid	40.8	10.2	396 ms	98.0 ms	4.0	0	1,223	30.0 s
Sustained decode	8K	8	Valid	81.9	10.2	397 ms	98.0 ms	8.0	0	2,456	30.0 s
Sustained decode	8K	16	Valid	163.8	10.2	538 ms	97.9 ms	16.0	0	4,910	30.0 s
Sustained decode	8K	32	Valid	323.0	10.0	726 ms	99.5 ms	32.0	0	9,664	29.9 s
Sustained decode	8K	64	Valid	620.1	9.6	1,144 ms	104.1 ms	64.0	0	18,559	29.9 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	10.2	10.2	161 ms	98.2 ms	1.0	0	306	29.9 s
Sustained decode	16K	2	Valid	20.3	10.1	315 ms	98.7 ms	2.0	0	608	29.9 s
Sustained decode	16K	4	Valid	40.8	10.2	425 ms	98.2 ms	4.0	0	1,224	30.0 s
Sustained decode	16K	8	Valid	81.7	10.2	562 ms	98.2 ms	8.0	0	2,448	30.0 s
Sustained decode	16K	16	Valid	163.5	10.2	716 ms	98.3 ms	16.0	0	4,895	29.9 s
Sustained decode	16K	32	Valid	322.4	10.0	1,097 ms	100.0 ms	32.0	0	9,663	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	10.2	10.2	199 ms	98.2 ms	1.0	0	305	29.9 s
Sustained decode	32K	2	Valid	20.4	10.2	333 ms	98.5 ms	2.0	0	610	30.0 s
Sustained decode	32K	4	Valid	40.7	10.2	531 ms	98.4 ms	4.0	0	1,220	30.0 s
Sustained decode	32K	8	Valid	81.4	10.1	740 ms	98.5 ms	8.0	0	2,439	30.0 s
Sustained decode	32K	16	Valid	163.4	10.1	1,038 ms	98.6 ms	16.0	0	4,895	30.0 s
Sustained decode	32K	32	Valid	321.5	9.9	1,707 ms	100.6 ms	32.0	0	9,630	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	10.2	10.2	285 ms	98.2 ms	1.0	0	305	29.9 s
Sustained decode	64K	2	Valid	20.4	10.2	476 ms	98.5 ms	2.0	0	610	30.0 s
Sustained decode	64K	4	Valid	40.8	10.2	697 ms	98.5 ms	4.0	0	1,220	29.9 s
Sustained decode	64K	8	Valid	81.1	10.1	1,084 ms	98.9 ms	8.0	0	2,430	30.0 s
Sustained decode	64K	16	Valid	162.8	10.1	1,654 ms	99.1 ms	16.0	0	4,880	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	10.2	10.2	439 ms	98.0 ms	1.0	0	306	29.9 s
Sustained decode	128K	2	Valid	20.4	10.1	699 ms	98.7 ms	2.0	0	610	30.0 s
Sustained decode	128K	4	Valid	41.0	10.2	1,100 ms	97.7 ms	4.0	0	1,227	29.9 s
Sustained decode	128K	8	Valid	82.2	10.2	1,702 ms	98.0 ms	8.0	0	2,464	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	10.2	10.2	765 ms	98.1 ms	1.0	0	305	29.9 s
Sustained decode	251K	2	Valid	20.5	10.3	1,145 ms	97.5 ms	2.0	0	614	29.9 s
Sustained decode	251K	4	Valid	41.2	10.2	1,889 ms	97.9 ms	4.0	0	1,232	29.9 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.