Speed test matrix
Kimi-K2.7-Code-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
sglang · 0.5.6.post3.dev11412+g964c45cf3 · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11412+g964c45cf3
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
964c45cf31e8
Run
- Served model
- Kimi-K2.7-Code-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-sglang-964c45cf3/bin/sglang" serve --model-path nvidia/Kimi-K2.7-Code-NVFP4 --served-model-name Kimi-K2.7-Code-NVFP4 --trust-remote-code --tp 8 --reasoning-parser kimi_k2 --tool-call-parser kimi_k2 --context-length 131072 --mem-fraction-static 0.86 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton --moe-runner-backend flashinfer_cutlass --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context2,9892,740 ms TTFT
16K context2,5876,275 ms TTFT
32K context2,02115,993 ms TTFT
64K context1,40146,042 ms TTFT
123K context884.0140,000 ms TTFT
128K context864.0149,083 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 62.7125 ms TTFT | 120.8214 ms TTFT | 211.9247 ms TTFT | 341.8243 ms TTFT | 489.0251 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 59.8223 ms TTFT | 117.7367 ms TTFT | 202.9433 ms TTFT | 323.3448 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 57.1403 ms TTFT | 109.9694 ms TTFT | 186.0823 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 53.3786 ms TTFT | 100.11,377 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 46.7Requests queued1,504 ms TTFT | 87.62,601 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,989.0 | — | 2,740 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 2,587.0 | — | 6,275 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 2,021.0 | — | 15,993 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,401.0 | — | 46,042 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 884.0 | — | 140,000 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 864.0 | — | 149,083 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 62.7 | 62.9 | 125 ms | 15.9 ms | 1.0 | 0 | 1,884 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 120.8 | 60.4 | 214 ms | 16.6 ms | 2.0 | 0 | 3,627 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 211.9 | 53.1 | 247 ms | 18.8 ms | 4.0 | 0 | 6,356 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 341.8 | 43.0 | 243 ms | 23.2 ms | 8.0 | 0 | 10,265 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 489.0 | 30.8 | 251 ms | 32.4 ms | 16.0 | 0 | 14,672 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 59.8 | 60.0 | 223 ms | 16.7 ms | 1.0 | 0 | 1,794 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 117.7 | 58.9 | 367 ms | 17.0 ms | 2.0 | 0 | 3,534 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 202.9 | 50.8 | 433 ms | 19.7 ms | 4.0 | 0 | 6,092 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 323.3 | 40.6 | 448 ms | 24.6 ms | 8.0 | 0 | 9,688 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 57.1 | 57.2 | 403 ms | 17.5 ms | 1.0 | 0 | 1,712 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 109.9 | 54.9 | 694 ms | 18.2 ms | 2.0 | 0 | 3,300 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 186.0 | 46.6 | 823 ms | 21.5 ms | 4.0 | 0 | 5,577 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 53.3 | 53.4 | 786 ms | 18.7 ms | 1.0 | 0 | 1,599 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 100.1 | 49.9 | 1,377 ms | 20.0 ms | 2.0 | 0 | 3,006 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Requests queued | 46.7 | 48.2 | 1,504 ms | 20.7 ms | 0.9 | 0 | 1,401 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 87.6 | 43.5 | 2,601 ms | 23.0 ms | 2.0 | 0 | 2,628 | 30.0 s |
| Sustained decode | 123K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.