Speed test matrix

Kimi-K2.6 on RTX PRO 6000 Blackwell Server Edition 96GB ×8

sglang · 0.5.6.post3.dev11412+g964c45cf3 · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Oct 13, 2026, 21:01 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
GPU count: 8
VRAM: 768 GB (8 × 96 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev11412+g964c45cf3
Quantization: INT4
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 964c45cf31e8

Run

Served model: moonshotai/Kimi-K2.6
Revision: main
Benchmark tool: llm-inference-bench v0.7.9
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_rtxpro6k-nvidia-sglang-964c45cf3/bin/sglang" serve --model-path moonshotai/Kimi-K2.6 --served-model-name moonshotai/Kimi-K2.6 --trust-remote-code --tp 8 --reasoning-parser kimi_k2 --tool-call-parser kimi_k2 --context-length 131072 --mem-fraction-static 0.88 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton --moe-runner-backend marlin

Prefill

prompt tok/s by context length

8K context4,5141,814 ms TTFT

16K context3,6514,445 ms TTFT

32K context2,60512,401 ms TTFT

64K context1,67238,550 ms TTFT

123K context904.0136,884 ms TTFT

128K context867.0148,577 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	72.3118 ms TTFT	135.4190 ms TTFT	235.0787 ms TTFT	376.1233 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
16K	68.1216 ms TTFT	128.2350 ms TTFT	220.7451 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
32K	61.3425 ms TTFT	109.7740 ms TTFT	173.7897 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
64K	52.6900 ms TTFT	92.91,539 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
123K	42.3Requests queued1,756 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	4,514.0	—	1,814 ms	—	—	1	—	—
Prefill	16K	1	Valid	3,651.0	—	4,445 ms	—	—	1	—	—
Prefill	32K	1	Valid	2,605.0	—	12,401 ms	—	—	1	—	—
Prefill	64K	1	Valid	1,672.0	—	38,550 ms	—	—	1	—	—
Prefill	123K	1	Valid	904.0	—	136,884 ms	—	—	1	—	—
Prefill	128K	1	Valid	867.0	—	148,577 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	72.3	72.6	118 ms	13.8 ms	1.0	0	2,168	30.0 s
Sustained decode	8K	2	Valid	135.4	67.9	190 ms	14.7 ms	2.0	0	4,066	30.0 s
Sustained decode	8K	4	Valid	235.0	59.1	787 ms	16.9 ms	4.0	0	7,057	30.0 s
Sustained decode	8K	8	Valid	376.1	47.5	233 ms	21.0 ms	8.0	0	11,278	30.0 s
Sustained decode	8K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	68.1	68.2	216 ms	14.7 ms	1.0	0	2,043	30.0 s
Sustained decode	16K	2	Valid	128.2	64.2	350 ms	15.6 ms	2.0	0	3,844	30.0 s
Sustained decode	16K	4	Valid	220.7	55.5	451 ms	18.0 ms	4.0	0	6,620	30.0 s
Sustained decode	16K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	61.3	61.3	425 ms	16.3 ms	1.0	0	1,840	30.0 s
Sustained decode	32K	2	Valid	109.7	54.8	740 ms	18.2 ms	2.0	0	3,296	30.0 s
Sustained decode	32K	4	Valid	173.7	43.6	897 ms	23.0 ms	4.0	0	5,208	30.0 s
Sustained decode	32K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	52.6	52.6	900 ms	19.0 ms	1.0	0	1,577	30.0 s
Sustained decode	64K	2	Valid	92.9	46.2	1,539 ms	21.7 ms	2.0	0	2,788	30.0 s
Sustained decode	64K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Requests queued	42.3	44.1	1,756 ms	22.7 ms	0.9	0	1,269	30.0 s
Sustained decode	123K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.