Speed test matrix

Kimi-K2.7-Code on RTX PRO 6000 Blackwell Server Edition 96GB ×8

vllm · 0.30.1rc1.dev452+g73a583112 · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Oct 13, 2026, 10:08 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
GPU count: 8
VRAM: 768 GB (8 × 96 GB)
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev452+g73a583112
Quantization: INT4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 73a5831127a9

Run

Served model: moonshotai/Kimi-K2.7-Code
Revision: main
Benchmark tool: llm-inference-bench v0.7.9
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_V2_MODEL_RUNNER=1 FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_rtxpro6k-moonshotai-vllm-73a5831127a9/bin/vllm" serve moonshotai/Kimi-K2.7-Code --served-model-name moonshotai/Kimi-K2.7-Code --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 131072 --gpu-memory-utilization 0.91 --host 0.0.0.0 --port 8000 --api-key <redacted> --mm-encoder-tp-mode data --kernel-config.enable_flashinfer_autotune=False --attention-backend TRITON_MLA --moe-backend marlin

Prefill

prompt tok/s by context length

8K context5,6631,446 ms TTFT

16K context5,5292,935 ms TTFT

32K context5,2626,140 ms TTFT

64K context4,79313,455 ms TTFT

123K context4,07830,356 ms TTFT

128K context4,03831,898 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	63.187 ms TTFT	109.0406 ms TTFT	183.0227 ms TTFT	283.8232 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
16K	60.4127 ms TTFT	105.9422 ms TTFT	174.2286 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
32K	56.9227 ms TTFT	99.6557 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
64K	51.1404 ms TTFT	89.4810 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
123K	42.9753 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	5,663.0	—	1,446 ms	—	—	1	—	—
Prefill	16K	1	Valid	5,529.0	—	2,935 ms	—	—	1	—	—
Prefill	32K	1	Valid	5,262.0	—	6,140 ms	—	—	1	—	—
Prefill	64K	1	Valid	4,793.0	—	13,455 ms	—	—	1	—	—
Prefill	123K	1	Valid	4,078.0	—	30,356 ms	—	—	1	—	—
Prefill	128K	1	Valid	4,038.0	—	31,898 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	63.1	63.1	87 ms	15.9 ms	1.0	0	1,892	30.0 s
Sustained decode	8K	2	Valid	109.0	54.1	406 ms	18.5 ms	2.0	0	3,270	30.0 s
Sustained decode	8K	4	Valid	183.0	45.8	227 ms	21.8 ms	4.0	0	5,488	30.0 s
Sustained decode	8K	8	Valid	283.8	35.5	232 ms	28.2 ms	8.0	0	8,512	30.0 s
Sustained decode	8K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	60.4	60.4	127 ms	16.5 ms	1.0	0	1,810	30.0 s
Sustained decode	16K	2	Valid	105.9	52.7	422 ms	19.0 ms	2.0	0	3,176	30.0 s
Sustained decode	16K	4	Valid	174.2	43.7	286 ms	22.9 ms	4.0	0	5,224	30.0 s
Sustained decode	16K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	56.9	57.0	227 ms	17.5 ms	1.0	0	1,707	30.0 s
Sustained decode	32K	2	Valid	99.6	49.4	557 ms	20.2 ms	2.0	0	2,986	30.0 s
Sustained decode	32K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	51.1	51.1	404 ms	19.6 ms	1.0	0	1,532	30.0 s
Sustained decode	64K	2	Valid	89.4	44.2	810 ms	22.6 ms	2.0	0	2,680	30.0 s
Sustained decode	64K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	42.9	42.9	753 ms	23.3 ms	1.0	0	1,287	30.0 s
Sustained decode	123K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.