Speed test matrix
Kimi-K2.5 on H200 SXM 141GB ×8
vllm · 0.29.0 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- nvidia/Kimi-K2.5-Thinking-Eagle3
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve moonshotai/Kimi-K2.5 --served-model-name nvidia/Kimi-K2.5-Thinking-Eagle3 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.85 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --max-num-batched-tokens 16384 --kernel-config.enable_flashinfer_autotune=False --speculative-config '{"method":"eagle3","model":"nvidia/Kimi-K2.5-Thinking-Eagle3","num_speculative_tokens":3}'Prefill
prompt tok/s by context length8K context17.7K461 ms TTFT
16K context18.8K864 ms TTFT
32K context17.4K1,856 ms TTFT
64K context15.1K4,259 ms TTFT
128K context12.1K10,631 ms TTFT
251K context8,72528,923 ms TTFT
256K context8,68829,624 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 23.7143 ms TTFT | 45.2251 ms TTFT | 99.9379 ms TTFT | 187.7389 ms TTFT | 351.8507 ms TTFT | 711.4715 ms TTFT | Over capacity | Over capacity |
| 16K | 21.4157 ms TTFT | 46.1289 ms TTFT | 90.8399 ms TTFT | 181.6534 ms TTFT | 365.1641 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 23.2200 ms TTFT | 46.4332 ms TTFT | 92.1482 ms TTFT | 181.3672 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 22.0285 ms TTFT | 44.7455 ms TTFT | 94.1669 ms TTFT | 182.41,012 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 22.7471 ms TTFT | 47.9684 ms TTFT | 91.61,056 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 20.4802 ms TTFT | 38.21,146 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.7K | — | 461 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.8K | — | 864 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 17.4K | — | 1,856 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.1K | — | 4,259 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.1K | — | 10,631 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,725.0 | — | 28,923 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,688.0 | — | 29,624 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 23.7 | 24.3 | 143 ms | 41.1 ms | 1.0 | 0 | 709 | 29.9 s |
| Sustained decode | 8K | 2 | Valid | 45.2 | 23.3 | 251 ms | 43.0 ms | 2.0 | 0 | 1,354 | 29.9 s |
| Sustained decode | 8K | 4 | Valid | 99.9 | 24.5 | 379 ms | 40.9 ms | 4.0 | 0 | 2,989 | 29.9 s |
| Sustained decode | 8K | 8 | Valid | 187.7 | 23.9 | 389 ms | 41.8 ms | 8.0 | 0 | 5,617 | 29.9 s |
| Sustained decode | 8K | 16 | Valid | 351.8 | 22.8 | 507 ms | 43.8 ms | 16.0 | 0 | 10,555 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 711.4 | 22.9 | 715 ms | 43.7 ms | 32.0 | 0 | 21,329 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 21.4 | 22.0 | 157 ms | 45.5 ms | 1.0 | 0 | 642 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 46.1 | 23.5 | 289 ms | 42.6 ms | 2.0 | 0 | 1,380 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 90.8 | 23.8 | 399 ms | 42.1 ms | 4.0 | 0 | 2,723 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 181.6 | 23.5 | 534 ms | 42.5 ms | 8.0 | 0 | 5,444 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 365.1 | 23.5 | 641 ms | 42.6 ms | 16.0 | 0 | 10,953 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 23.2 | 23.8 | 200 ms | 42.0 ms | 1.0 | 0 | 694 | 29.9 s |
| Sustained decode | 32K | 2 | Valid | 46.4 | 23.8 | 332 ms | 42.0 ms | 2.0 | 0 | 1,391 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 92.1 | 23.1 | 482 ms | 43.3 ms | 4.0 | 0 | 2,757 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 181.3 | 23.2 | 672 ms | 43.0 ms | 8.0 | 0 | 5,431 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 22.0 | 22.8 | 285 ms | 43.8 ms | 1.0 | 0 | 659 | 29.9 s |
| Sustained decode | 64K | 2 | Valid | 44.7 | 23.3 | 455 ms | 42.8 ms | 2.0 | 0 | 1,337 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 94.1 | 24.4 | 669 ms | 41.0 ms | 4.0 | 0 | 2,823 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 182.4 | 23.1 | 1,012 ms | 43.3 ms | 8.0 | 0 | 5,462 | 29.9 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 22.7 | 23.3 | 471 ms | 43.0 ms | 1.0 | 0 | 679 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 47.9 | 24.5 | 684 ms | 41.0 ms | 2.0 | 0 | 1,437 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 91.6 | 23.3 | 1,056 ms | 42.9 ms | 4.0 | 0 | 2,744 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 20.4 | 20.8 | 802 ms | 48.2 ms | 1.0 | 0 | 612 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 38.2 | 19.7 | 1,146 ms | 50.8 ms | 2.0 | 0 | 1,145 | 29.9 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.