Speed test matrix
Kimi-K2.7-Code on H200 SXM 141GB ×8
vllm · 0.29.0 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- moonshotai/Kimi-K2.7-Code
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve moonshotai/Kimi-K2.7-Code --served-model-name moonshotai/Kimi-K2.7-Code --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --kernel-config.enable_flashinfer_autotune=False
Prefill
prompt tok/s by context length8K context17.7K463 ms TTFT
16K context17.5K927 ms TTFT
32K context16.4K1,973 ms TTFT
64K context14.4K4,479 ms TTFT
128K context11.6K11,087 ms TTFT
251K context8,38730,089 ms TTFT
256K context8,33430,882 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 126.354 ms TTFT | 238.163 ms TTFT | 420.9105 ms TTFT | 695.7162 ms TTFT | 1,100.0272 ms TTFT | 1,560.9495 ms TTFT | Over capacity | Over capacity |
| 16K | 120.474 ms TTFT | 227.5119 ms TTFT | 406.4164 ms TTFT | 656.5259 ms TTFT | 980.9425 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 110.1118 ms TTFT | 209.4172 ms TTFT | 377.2261 ms TTFT | 575.7434 ms TTFT | 818.4780 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 94.2205 ms TTFT | 180.3318 ms TTFT | 321.0476 ms TTFT | 461.0800 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 72.4377 ms TTFT | 139.8557 ms TTFT | 250.1871 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 50.9719 ms TTFT | 99.31,042 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.7K | — | 463 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 17.5K | — | 927 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16.4K | — | 1,973 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.4K | — | 4,479 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.6K | — | 11,087 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,387.0 | — | 30,089 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,334.0 | — | 30,882 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 126.3 | 127.1 | 54 ms | 7.9 ms | 1.0 | 1 | 3,788 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 238.1 | 119.4 | 63 ms | 8.4 ms | 2.0 | 2 | 7,144 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 420.9 | 105.5 | 105 ms | 9.5 ms | 4.0 | 0 | 12,624 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 695.7 | 87.4 | 162 ms | 11.4 ms | 8.0 | 0 | 20,864 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,100.0 | 69.1 | 272 ms | 14.5 ms | 16.0 | 0 | 32,992 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,560.9 | 49.2 | 495 ms | 20.3 ms | 32.0 | 0 | 46,818 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 120.4 | 120.6 | 74 ms | 8.3 ms | 1.0 | 0 | 3,612 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 227.5 | 113.9 | 119 ms | 8.8 ms | 2.0 | 0 | 6,824 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 406.4 | 101.7 | 164 ms | 9.8 ms | 4.0 | 0 | 12,194 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 656.5 | 82.2 | 259 ms | 12.2 ms | 8.0 | 0 | 19,688 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 980.9 | 61.7 | 425 ms | 16.2 ms | 16.0 | 0 | 29,426 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 110.1 | 110.3 | 118 ms | 9.1 ms | 1.0 | 0 | 3,303 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 209.4 | 104.8 | 172 ms | 9.5 ms | 2.0 | 0 | 6,280 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 377.2 | 94.4 | 261 ms | 10.6 ms | 4.0 | 0 | 11,316 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 575.7 | 72.2 | 434 ms | 13.9 ms | 8.0 | 0 | 17,265 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 818.4 | 51.4 | 780 ms | 19.5 ms | 16.0 | 0 | 24,544 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 94.2 | 94.3 | 205 ms | 10.6 ms | 1.0 | 0 | 2,826 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 180.3 | 90.2 | 318 ms | 11.1 ms | 2.0 | 0 | 5,408 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 321.0 | 80.4 | 476 ms | 12.4 ms | 4.0 | 0 | 9,628 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 461.0 | 57.8 | 800 ms | 17.3 ms | 8.0 | 0 | 13,825 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 72.4 | 72.5 | 377 ms | 13.8 ms | 1.0 | 0 | 2,172 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 139.8 | 70.0 | 557 ms | 14.3 ms | 2.0 | 0 | 4,194 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 250.1 | 62.5 | 871 ms | 16.0 ms | 4.0 | 0 | 7,500 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 50.9 | 50.9 | 719 ms | 19.7 ms | 1.0 | 0 | 1,526 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 99.3 | 49.6 | 1,042 ms | 20.2 ms | 2.0 | 0 | 2,978 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.