Speed test matrix

Kimi-K2.7-Code-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8

sglang · 0.5.6.post3.dev11412+g964c45cf3 · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Oct 14, 2026, 00:48 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
GPU count: 8
VRAM: 768 GB (8 × 96 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev11412+g964c45cf3
Quantization: NVFP4
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 964c45cf31e8

Run

Served model: Kimi-K2.7-Code-NVFP4
Revision: main
Benchmark tool: llm-inference-bench v0.7.9
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_rtxpro6k-nvidia-sglang-964c45cf3/bin/sglang" serve --model-path nvidia/Kimi-K2.7-Code-NVFP4 --served-model-name Kimi-K2.7-Code-NVFP4 --trust-remote-code --tp 8 --reasoning-parser kimi_k2 --tool-call-parser kimi_k2 --context-length 131072 --mem-fraction-static 0.86 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton --moe-runner-backend flashinfer_cutlass --quantization modelopt_fp4

Prefill

prompt tok/s by context length

8K context2,9892,740 ms TTFT

16K context2,5876,275 ms TTFT

32K context2,02115,993 ms TTFT

64K context1,40146,042 ms TTFT

123K context884.0140,000 ms TTFT

128K context864.0149,083 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	62.7125 ms TTFT	120.8214 ms TTFT	211.9247 ms TTFT	341.8243 ms TTFT	489.0251 ms TTFT	Over capacity	Over capacity	Over capacity
16K	59.8223 ms TTFT	117.7367 ms TTFT	202.9433 ms TTFT	323.3448 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
32K	57.1403 ms TTFT	109.9694 ms TTFT	186.0823 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
64K	53.3786 ms TTFT	100.11,377 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
123K	46.7Requests queued1,504 ms TTFT	87.62,601 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	2,989.0	—	2,740 ms	—	—	1	—	—
Prefill	16K	1	Valid	2,587.0	—	6,275 ms	—	—	1	—	—
Prefill	32K	1	Valid	2,021.0	—	15,993 ms	—	—	1	—	—
Prefill	64K	1	Valid	1,401.0	—	46,042 ms	—	—	1	—	—
Prefill	123K	1	Valid	884.0	—	140,000 ms	—	—	1	—	—
Prefill	128K	1	Valid	864.0	—	149,083 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	62.7	62.9	125 ms	15.9 ms	1.0	0	1,884	30.0 s
Sustained decode	8K	2	Valid	120.8	60.4	214 ms	16.6 ms	2.0	0	3,627	30.0 s
Sustained decode	8K	4	Valid	211.9	53.1	247 ms	18.8 ms	4.0	0	6,356	30.0 s
Sustained decode	8K	8	Valid	341.8	43.0	243 ms	23.2 ms	8.0	0	10,265	30.0 s
Sustained decode	8K	16	Valid	489.0	30.8	251 ms	32.4 ms	16.0	0	14,672	30.0 s
Sustained decode	8K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	59.8	60.0	223 ms	16.7 ms	1.0	0	1,794	30.0 s
Sustained decode	16K	2	Valid	117.7	58.9	367 ms	17.0 ms	2.0	0	3,534	30.0 s
Sustained decode	16K	4	Valid	202.9	50.8	433 ms	19.7 ms	4.0	0	6,092	30.0 s
Sustained decode	16K	8	Valid	323.3	40.6	448 ms	24.6 ms	8.0	0	9,688	30.0 s
Sustained decode	16K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	57.1	57.2	403 ms	17.5 ms	1.0	0	1,712	30.0 s
Sustained decode	32K	2	Valid	109.9	54.9	694 ms	18.2 ms	2.0	0	3,300	30.0 s
Sustained decode	32K	4	Valid	186.0	46.6	823 ms	21.5 ms	4.0	0	5,577	30.0 s
Sustained decode	32K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	53.3	53.4	786 ms	18.7 ms	1.0	0	1,599	30.0 s
Sustained decode	64K	2	Valid	100.1	49.9	1,377 ms	20.0 ms	2.0	0	3,006	30.0 s
Sustained decode	64K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Requests queued	46.7	48.2	1,504 ms	20.7 ms	0.9	0	1,401	30.0 s
Sustained decode	123K	2	Valid	87.6	43.5	2,601 ms	23.0 ms	2.0	0	2,628	30.0 s
Sustained decode	123K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.