Speed test matrix
Kimi-K2.7-Code on H200 SXM 141GB ×8
vllm · 0.29.0 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- nvidia/Kimi-K2.7-Code-DFlash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve moonshotai/Kimi-K2.7-Code --served-model-name nvidia/Kimi-K2.7-Code-DFlash --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --max-num-batched-tokens 16384 --speculative-config '{"method":"dflash","model":"nvidia/Kimi-K2.7-Code-DFlash","num_speculative_tokens":8}'Prefill
prompt tok/s by context length8K context17.7K462 ms TTFT
16K context18.7K867 ms TTFT
32K context17.4K1,861 ms TTFT
64K context15.2K4,250 ms TTFT
128K context12.2K10,562 ms TTFT
251K context8,82828,591 ms TTFT
256K context8,78329,311 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 24.3136 ms TTFT | 47.9258 ms TTFT | 91.1382 ms TTFT | 186.5393 ms TTFT | 366.5511 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 31.8161 ms TTFT | 46.8281 ms TTFT | 87.5392 ms TTFT | 188.1465 ms TTFT | 358.8700 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 23.9202 ms TTFT | 47.9327 ms TTFT | 102.1495 ms TTFT | 183.4670 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 20.2290 ms TTFT | 46.1464 ms TTFT | 105.9676 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 25.4468 ms TTFT | 46.6695 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 10.1821 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.7K | — | 462 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.7K | — | 867 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 17.4K | — | 1,861 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.2K | — | 4,250 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.2K | — | 10,562 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,828.0 | — | 28,591 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,783.0 | — | 29,311 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 24.3 | 25.5 | 136 ms | 39.2 ms | 1.0 | 0 | 728 | 29.9 s |
| Sustained decode | 8K | 2 | Valid | 47.9 | 25.0 | 258 ms | 40.0 ms | 2.0 | 0 | 1,434 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 91.1 | 24.8 | 382 ms | 40.3 ms | 4.0 | 0 | 2,728 | 29.9 s |
| Sustained decode | 8K | 8 | Valid | 186.5 | 24.7 | 393 ms | 40.5 ms | 8.0 | 0 | 5,588 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 366.5 | 23.9 | 511 ms | 41.9 ms | 16.0 | 0 | 10,958 | 29.9 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 31.8 | 32.0 | 161 ms | 31.3 ms | 1.0 | 0 | 951 | 29.9 s |
| Sustained decode | 16K | 2 | Valid | 46.8 | 24.1 | 281 ms | 41.5 ms | 2.0 | 0 | 1,402 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 87.5 | 23.1 | 392 ms | 43.6 ms | 4.0 | 0 | 2,618 | 29.9 s |
| Sustained decode | 16K | 8 | Valid | 188.1 | 23.9 | 465 ms | 41.9 ms | 8.0 | 0 | 5,625 | 29.9 s |
| Sustained decode | 16K | 16 | Valid | 358.8 | 24.1 | 700 ms | 41.5 ms | 16.0 | 0 | 10,731 | 29.9 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 23.9 | 24.7 | 202 ms | 40.5 ms | 1.0 | 0 | 715 | 29.9 s |
| Sustained decode | 32K | 2 | Valid | 47.9 | 25.2 | 327 ms | 39.7 ms | 2.0 | 0 | 1,431 | 29.9 s |
| Sustained decode | 32K | 4 | Valid | 102.1 | 25.1 | 495 ms | 39.8 ms | 4.0 | 0 | 3,055 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 183.4 | 24.6 | 670 ms | 40.7 ms | 8.0 | 0 | 5,486 | 29.9 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 20.2 | 21.3 | 290 ms | 46.9 ms | 1.0 | 0 | 605 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 46.1 | 24.1 | 464 ms | 41.6 ms | 2.0 | 0 | 1,379 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 105.9 | 27.6 | 676 ms | 36.3 ms | 4.0 | 0 | 3,167 | 29.9 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 25.4 | 26.3 | 468 ms | 38.0 ms | 1.0 | 0 | 759 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 46.6 | 24.2 | 695 ms | 42.0 ms | 2.0 | 0 | 1,395 | 29.9 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 10.1 | 10.1 | 821 ms | 99.4 ms | 1.0 | 0 | 302 | 29.9 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.