Speed test matrix
Kimi-K2.7-Code on RTX PRO 6000 Blackwell Server Edition 96GB ×8
sglang · 0.5.6.post3.dev11412+g964c45cf3 · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11412+g964c45cf3
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
964c45cf31e8
Run
- Served model
- moonshotai/Kimi-K2.7-Code
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-sglang-964c45cf3/bin/sglang" serve --model-path moonshotai/Kimi-K2.7-Code --served-model-name moonshotai/Kimi-K2.7-Code --trust-remote-code --tp 8 --reasoning-parser kimi_k2 --tool-call-parser kimi_k2 --context-length 131072 --mem-fraction-static 0.88 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton --moe-runner-backend marlin
Prefill
prompt tok/s by context length8K context4,4441,843 ms TTFT
16K context3,6004,508 ms TTFT
32K context2,58512,500 ms TTFT
64K context1,66338,765 ms TTFT
123K context901.0137,377 ms TTFT
128K context866.0148,809 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 72.4115 ms TTFT | 136.0184 ms TTFT | 236.4250 ms TTFT | 374.8229 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 16K | 67.9216 ms TTFT | 128.2352 ms TTFT | 219.5438 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 61.1427 ms TTFT | 110.0708 ms TTFT | 173.3883 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 52.6900 ms TTFT | 92.81,544 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 42.2Requests queued1,750 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,444.0 | — | 1,843 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 3,600.0 | — | 4,508 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 2,585.0 | — | 12,500 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,663.0 | — | 38,765 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 901.0 | — | 137,377 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 866.0 | — | 148,809 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 72.4 | 72.6 | 115 ms | 13.8 ms | 1.0 | 0 | 2,174 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 136.0 | 68.2 | 184 ms | 14.7 ms | 2.0 | 0 | 4,080 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 236.4 | 59.3 | 250 ms | 16.9 ms | 4.0 | 0 | 7,100 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 374.8 | 47.1 | 229 ms | 21.2 ms | 8.0 | 0 | 11,239 | 30.0 s |
| Sustained decode | 8K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 67.9 | 68.0 | 216 ms | 14.7 ms | 1.0 | 0 | 2,036 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 128.2 | 64.2 | 352 ms | 15.6 ms | 2.0 | 0 | 3,850 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 219.5 | 55.1 | 438 ms | 18.1 ms | 4.0 | 0 | 6,584 | 30.0 s |
| Sustained decode | 16K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 61.1 | 61.1 | 427 ms | 16.4 ms | 1.0 | 0 | 1,835 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 110.0 | 54.9 | 708 ms | 18.2 ms | 2.0 | 0 | 3,302 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 173.3 | 43.4 | 883 ms | 23.1 ms | 4.0 | 0 | 5,204 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 52.6 | 52.7 | 900 ms | 19.0 ms | 1.0 | 0 | 1,579 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 92.8 | 46.1 | 1,544 ms | 21.7 ms | 2.0 | 0 | 2,782 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Requests queued | 42.2 | 44.0 | 1,750 ms | 22.7 ms | 0.9 | 0 | 1,267 | 30.0 s |
| Sustained decode | 123K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.