Speed test matrix
Kimi-K2.6 on H200 SXM 141GB ×8
vllm · 0.29.0 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- nvidia/Kimi-K2.6-DFlash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve moonshotai/Kimi-K2.6 --served-model-name nvidia/Kimi-K2.6-DFlash --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --max-num-batched-tokens 16384 --speculative-config '{"method":"dflash","model":"nvidia/Kimi-K2.6-DFlash","num_speculative_tokens":8}'Prefill
prompt tok/s by context length8K context17.6K465 ms TTFT
16K context18.7K868 ms TTFT
32K context17.3K1,865 ms TTFT
64K context15.2K4,253 ms TTFT
128K context12.2K10,531 ms TTFT
251K context8,82028,621 ms TTFT
256K context8,80829,233 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 26.2137 ms TTFT | 51.5259 ms TTFT | 103.7384 ms TTFT | 200.6391 ms TTFT | 384.7515 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 23.6161 ms TTFT | 51.2290 ms TTFT | 100.6405 ms TTFT | 196.9523 ms TTFT | 398.9700 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 23.6201 ms TTFT | 46.7321 ms TTFT | 98.8484 ms TTFT | 193.2673 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 23.7293 ms TTFT | 48.4460 ms TTFT | 96.3675 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 10.1465 ms TTFT | 19.8693 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 9.9796 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.6K | — | 465 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.7K | — | 868 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 17.3K | — | 1,865 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.2K | — | 4,253 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.2K | — | 10,531 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,820.0 | — | 28,621 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,808.0 | — | 29,233 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 26.2 | 27.6 | 137 ms | 36.2 ms | 1.0 | 0 | 785 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 51.5 | 26.6 | 259 ms | 37.6 ms | 2.0 | 0 | 1,542 | 29.9 s |
| Sustained decode | 8K | 4 | Valid | 103.7 | 27.2 | 384 ms | 36.8 ms | 4.0 | 0 | 3,100 | 29.9 s |
| Sustained decode | 8K | 8 | Valid | 200.6 | 26.2 | 391 ms | 38.1 ms | 8.0 | 0 | 6,017 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 384.7 | 25.2 | 515 ms | 39.8 ms | 16.0 | 0 | 11,510 | 29.9 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 23.6 | 24.9 | 161 ms | 40.1 ms | 1.0 | 0 | 707 | 29.9 s |
| Sustained decode | 16K | 2 | Valid | 51.2 | 26.2 | 290 ms | 38.1 ms | 2.0 | 0 | 1,535 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 100.6 | 26.4 | 405 ms | 37.9 ms | 4.0 | 0 | 3,013 | 29.9 s |
| Sustained decode | 16K | 8 | Valid | 196.9 | 26.3 | 523 ms | 38.0 ms | 8.0 | 0 | 5,896 | 29.9 s |
| Sustained decode | 16K | 16 | Valid | 398.9 | 26.0 | 700 ms | 38.5 ms | 16.0 | 0 | 11,962 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 23.6 | 25.0 | 201 ms | 40.1 ms | 1.0 | 0 | 708 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 46.7 | 24.8 | 321 ms | 40.3 ms | 2.0 | 0 | 1,401 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 98.8 | 25.9 | 484 ms | 38.5 ms | 4.0 | 0 | 2,960 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 193.2 | 25.4 | 673 ms | 39.3 ms | 8.0 | 0 | 5,787 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 23.7 | 25.0 | 293 ms | 40.1 ms | 1.0 | 0 | 709 | 29.9 s |
| Sustained decode | 64K | 2 | Valid | 48.4 | 25.4 | 460 ms | 39.4 ms | 2.0 | 0 | 1,451 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 96.3 | 25.0 | 675 ms | 40.0 ms | 4.0 | 0 | 2,879 | 29.9 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 10.1 | 10.1 | 465 ms | 99.4 ms | 1.0 | 0 | 302 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 19.8 | 9.9 | 693 ms | 101.1 ms | 2.0 | 0 | 593 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 9.9 | 9.9 | 796 ms | 100.6 ms | 1.0 | 0 | 297 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.