Speed test matrix
Kimi-K2.6 on H200 SXM 141GB ×8
vllm · 0.29.0 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- nvidia/Kimi-K2.6-Eagle3
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve moonshotai/Kimi-K2.6 --served-model-name nvidia/Kimi-K2.6-Eagle3 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.85 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --max-num-batched-tokens 16384 --kernel-config.enable_flashinfer_autotune=False --speculative-config '{"method":"eagle3","model":"nvidia/Kimi-K2.6-Eagle3","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context17.8K460 ms TTFT
16K context18.8K864 ms TTFT
32K context17.4K1,855 ms TTFT
64K context15.2K4,246 ms TTFT
128K context12.2K10,575 ms TTFT
251K context8,74428,859 ms TTFT
256K context8,70729,557 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 22.7140 ms TTFT | 47.7251 ms TTFT | 92.2372 ms TTFT | 183.3385 ms TTFT | 349.0519 ms TTFT | 686.5697 ms TTFT | Over capacity | Over capacity |
| 16K | 24.8156 ms TTFT | 43.0278 ms TTFT | 93.2391 ms TTFT | 179.8526 ms TTFT | 347.4637 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 23.9200 ms TTFT | 45.0324 ms TTFT | 94.8473 ms TTFT | 181.8660 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 24.8282 ms TTFT | 46.5451 ms TTFT | 94.0673 ms TTFT | 179.91,011 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 24.0457 ms TTFT | 45.6680 ms TTFT | 90.21,059 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 19.9799 ms TTFT | 40.71,181 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.8K | — | 460 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.8K | — | 864 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 17.4K | — | 1,855 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.2K | — | 4,246 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.2K | — | 10,575 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,744.0 | — | 28,859 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,707.0 | — | 29,557 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 22.7 | 23.8 | 140 ms | 42.0 ms | 1.0 | 0 | 681 | 29.9 s |
| Sustained decode | 8K | 2 | Valid | 47.7 | 24.5 | 251 ms | 40.9 ms | 2.0 | 0 | 1,431 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 92.2 | 23.8 | 372 ms | 42.1 ms | 4.0 | 0 | 2,761 | 29.9 s |
| Sustained decode | 8K | 8 | Valid | 183.3 | 23.8 | 385 ms | 42.1 ms | 8.0 | 0 | 5,497 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 349.0 | 22.6 | 519 ms | 44.2 ms | 16.0 | 0 | 10,452 | 29.9 s |
| Sustained decode | 8K | 32 | Valid | 686.5 | 22.1 | 697 ms | 45.3 ms | 32.0 | 0 | 20,535 | 29.9 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 24.8 | 25.2 | 156 ms | 39.7 ms | 1.0 | 0 | 743 | 29.9 s |
| Sustained decode | 16K | 2 | Valid | 43.0 | 22.2 | 278 ms | 45.0 ms | 2.0 | 0 | 1,287 | 29.9 s |
| Sustained decode | 16K | 4 | Valid | 93.2 | 24.2 | 391 ms | 41.4 ms | 4.0 | 0 | 2,789 | 29.9 s |
| Sustained decode | 16K | 8 | Valid | 179.8 | 23.3 | 526 ms | 42.9 ms | 8.0 | 0 | 5,391 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 347.4 | 22.4 | 637 ms | 44.7 ms | 16.0 | 0 | 10,415 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 23.9 | 24.5 | 200 ms | 40.8 ms | 1.0 | 0 | 717 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 45.0 | 23.2 | 324 ms | 43.0 ms | 2.0 | 0 | 1,349 | 29.9 s |
| Sustained decode | 32K | 4 | Valid | 94.8 | 24.0 | 473 ms | 41.7 ms | 4.0 | 0 | 2,843 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 181.8 | 23.6 | 660 ms | 42.4 ms | 8.0 | 0 | 5,438 | 29.9 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 24.8 | 25.0 | 282 ms | 40.1 ms | 1.0 | 0 | 743 | 29.9 s |
| Sustained decode | 64K | 2 | Valid | 46.5 | 23.8 | 451 ms | 42.1 ms | 2.0 | 0 | 1,394 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 94.0 | 24.2 | 673 ms | 41.3 ms | 4.0 | 0 | 2,814 | 29.9 s |
| Sustained decode | 64K | 8 | Valid | 179.9 | 23.1 | 1,011 ms | 43.4 ms | 8.0 | 0 | 5,388 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 24.0 | 24.7 | 457 ms | 40.4 ms | 1.0 | 0 | 717 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 45.6 | 23.3 | 680 ms | 43.0 ms | 2.0 | 0 | 1,368 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 90.2 | 23.2 | 1,059 ms | 43.2 ms | 4.0 | 0 | 2,702 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 19.9 | 20.7 | 799 ms | 48.4 ms | 1.0 | 0 | 597 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 40.7 | 20.7 | 1,181 ms | 48.3 ms | 2.0 | 0 | 1,216 | 29.9 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.