Speed test matrix
Kimi-K2.6 on H200 SXM 141GB ×8
vllm · 0.29.0 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- moonshotai/Kimi-K2.6
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve moonshotai/Kimi-K2.6 --served-model-name moonshotai/Kimi-K2.6 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --kernel-config.enable_flashinfer_autotune=False
Prefill
prompt tok/s by context length8K context17.6K465 ms TTFT
16K context17.5K928 ms TTFT
32K context16.3K1,977 ms TTFT
64K context14.4K4,491 ms TTFT
128K context11.6K11,097 ms TTFT
251K context8,36530,174 ms TTFT
256K context8,32130,938 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 126.954 ms TTFT | 238.968 ms TTFT | 423.2106 ms TTFT | 707.4184 ms TTFT | 1,112.4266 ms TTFT | 1,576.9498 ms TTFT | Over capacity | Over capacity |
| 16K | 120.976 ms TTFT | 228.9122 ms TTFT | 406.8146 ms TTFT | Looped | 995.3448 ms TTFT | 1,366.5836 ms TTFT | Over capacity | Over capacity |
| 32K | 110.5123 ms TTFT | 210.7176 ms TTFT | 377.9259 ms TTFT | 586.7435 ms TTFT | 832.1796 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 94.5207 ms TTFT | 181.5303 ms TTFT | 324.3481 ms TTFT | 461.6804 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 72.6382 ms TTFT | 140.4556 ms TTFT | 250.0862 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 51.0728 ms TTFT | 99.41,065 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.6K | — | 465 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 17.5K | — | 928 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16.3K | — | 1,977 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.4K | — | 4,491 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.6K | — | 11,097 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,365.0 | — | 30,174 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,321.0 | — | 30,938 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 126.9 | 127.7 | 54 ms | 7.8 ms | 1.0 | 1 | 3,806 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 238.9 | 119.9 | 68 ms | 8.3 ms | 2.0 | 2 | 7,165 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 423.2 | 106.1 | 106 ms | 9.4 ms | 4.0 | 0 | 12,695 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 707.4 | 88.7 | 184 ms | 11.3 ms | 8.0 | 0 | 21,216 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,112.4 | 69.9 | 266 ms | 14.3 ms | 16.0 | 0 | 33,360 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,576.9 | 49.7 | 498 ms | 20.1 ms | 32.0 | 0 | 47,296 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 120.9 | 121.0 | 76 ms | 8.3 ms | 1.0 | 0 | 3,626 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 228.9 | 114.6 | 122 ms | 8.7 ms | 2.0 | 0 | 6,866 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 406.8 | 101.9 | 146 ms | 9.8 ms | 4.0 | 0 | 12,204 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Valid | 995.3 | 62.5 | 448 ms | 16.0 ms | 16.0 | 0 | 29,856 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,366.5 | 43.0 | 836 ms | 23.3 ms | 32.0 | 0 | 40,991 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 110.5 | 110.6 | 123 ms | 9.0 ms | 1.0 | 0 | 3,313 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 210.7 | 105.5 | 176 ms | 9.5 ms | 2.0 | 0 | 6,320 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 377.9 | 94.5 | 259 ms | 10.6 ms | 4.0 | 0 | 11,336 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 586.7 | 73.4 | 435 ms | 13.6 ms | 8.0 | 0 | 17,600 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 832.1 | 52.2 | 796 ms | 19.2 ms | 16.0 | 0 | 24,963 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 94.5 | 94.6 | 207 ms | 10.6 ms | 1.0 | 0 | 2,835 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 181.5 | 90.8 | 303 ms | 11.0 ms | 2.0 | 0 | 5,445 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 324.3 | 81.1 | 481 ms | 12.3 ms | 4.0 | 0 | 9,728 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 461.6 | 57.8 | 804 ms | 17.3 ms | 8.0 | 0 | 13,848 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 72.6 | 72.7 | 382 ms | 13.8 ms | 1.0 | 0 | 2,178 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 140.4 | 70.3 | 556 ms | 14.2 ms | 2.0 | 0 | 4,210 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 250.0 | 62.5 | 862 ms | 16.0 ms | 4.0 | 0 | 7,500 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 51.0 | 51.0 | 728 ms | 19.6 ms | 1.0 | 0 | 1,528 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 99.4 | 49.7 | 1,065 ms | 20.1 ms | 2.0 | 0 | 2,982 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.