Speed test matrix
Kimi-K2.5-NVFP4 on H200 SXM 141GB ×8
vllm · 0.29.0 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- nvidia/Kimi-K2.5-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve nvidia/Kimi-K2.5-NVFP4 --served-model-name nvidia/Kimi-K2.5-NVFP4 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --max-num-batched-tokens 16384 --kernel-config.enable_flashinfer_autotune=False
Prefill
prompt tok/s by context length8K context16.7K490 ms TTFT
16K context17.8K910 ms TTFT
32K context16.9K1,914 ms TTFT
64K context14.9K4,316 ms TTFT
128K context12.1K10,661 ms TTFT
251K context8,79428,638 ms TTFT
256K context8,71729,463 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 9.9141 ms TTFT | 19.7274 ms TTFT | 39.7404 ms TTFT | 78.6408 ms TTFT | 158.2549 ms TTFT | 310.9727 ms TTFT | 603.01,171 ms TTFT | Over capacity |
| 16K | 9.9161 ms TTFT | 19.8299 ms TTFT | 39.4422 ms TTFT | 78.9562 ms TTFT | 157.3715 ms TTFT | 310.01,103 ms TTFT | Over capacity | Over capacity |
| 32K | 9.9202 ms TTFT | 19.7342 ms TTFT | 39.6536 ms TTFT | 78.8697 ms TTFT | 158.01,056 ms TTFT | 311.61,728 ms TTFT | Over capacity | Over capacity |
| 64K | 9.9278 ms TTFT | 19.7475 ms TTFT | 39.7714 ms TTFT | 78.91,046 ms TTFT | 157.61,677 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 9.9448 ms TTFT | 19.8704 ms TTFT | 39.61,083 ms TTFT | 79.11,732 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 9.9764 ms TTFT | 19.71,209 ms TTFT | 39.51,903 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.7K | — | 490 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 17.8K | — | 910 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16.9K | — | 1,914 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 14.9K | — | 4,316 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.1K | — | 10,661 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,794.0 | — | 28,638 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,717.0 | — | 29,463 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 9.9 | 9.9 | 141 ms | 100.8 ms | 1.0 | 0 | 298 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 19.7 | 9.8 | 274 ms | 101.6 ms | 2.0 | 0 | 592 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 39.7 | 9.9 | 404 ms | 101.2 ms | 4.0 | 0 | 1,188 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 78.6 | 9.8 | 408 ms | 101.9 ms | 8.0 | 0 | 2,351 | 29.9 s |
| Sustained decode | 8K | 16 | Valid | 158.2 | 9.9 | 549 ms | 101.2 ms | 16.0 | 0 | 4,735 | 29.9 s |
| Sustained decode | 8K | 32 | Valid | 310.9 | 9.7 | 727 ms | 103.6 ms | 32.0 | 0 | 9,310 | 29.9 s |
| Sustained decode | 8K | 64 | Valid | 603.0 | 9.4 | 1,171 ms | 106.7 ms | 64.0 | 0 | 18,046 | 29.9 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 9.9 | 9.9 | 161 ms | 100.8 ms | 1.0 | 0 | 297 | 29.9 s |
| Sustained decode | 16K | 2 | Valid | 19.8 | 9.9 | 299 ms | 101.5 ms | 2.0 | 0 | 592 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 39.4 | 9.8 | 422 ms | 101.9 ms | 4.0 | 0 | 1,177 | 29.9 s |
| Sustained decode | 16K | 8 | Valid | 78.9 | 9.8 | 562 ms | 101.7 ms | 8.0 | 0 | 2,366 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 157.3 | 9.8 | 715 ms | 102.0 ms | 16.0 | 0 | 4,720 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 310.0 | 9.6 | 1,103 ms | 103.8 ms | 32.0 | 0 | 9,285 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 9.9 | 9.9 | 202 ms | 100.9 ms | 1.0 | 0 | 298 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 19.7 | 9.8 | 342 ms | 101.7 ms | 2.0 | 0 | 590 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 39.6 | 9.9 | 536 ms | 101.5 ms | 4.0 | 0 | 1,184 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 78.8 | 9.8 | 697 ms | 102.1 ms | 8.0 | 0 | 2,360 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 158.0 | 9.8 | 1,056 ms | 101.8 ms | 16.0 | 0 | 4,735 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 311.6 | 9.6 | 1,728 ms | 103.7 ms | 32.0 | 0 | 9,344 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 9.9 | 9.9 | 278 ms | 100.7 ms | 1.0 | 0 | 298 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 19.7 | 9.8 | 475 ms | 101.5 ms | 2.0 | 0 | 592 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 39.7 | 9.9 | 714 ms | 101.3 ms | 4.0 | 0 | 1,186 | 29.9 s |
| Sustained decode | 64K | 8 | Valid | 78.9 | 9.8 | 1,046 ms | 102.0 ms | 8.0 | 0 | 2,367 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 157.6 | 9.8 | 1,677 ms | 102.4 ms | 16.0 | 0 | 4,718 | 29.9 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 9.9 | 9.9 | 448 ms | 100.8 ms | 1.0 | 0 | 298 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 19.8 | 9.9 | 704 ms | 101.5 ms | 2.0 | 0 | 592 | 29.9 s |
| Sustained decode | 128K | 4 | Valid | 39.6 | 9.9 | 1,083 ms | 101.5 ms | 4.0 | 0 | 1,187 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 79.1 | 9.8 | 1,732 ms | 102.0 ms | 8.0 | 0 | 2,367 | 29.9 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 9.9 | 9.9 | 764 ms | 101.2 ms | 1.0 | 0 | 296 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 19.7 | 9.8 | 1,209 ms | 101.6 ms | 2.0 | 0 | 592 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 39.5 | 9.8 | 1,903 ms | 101.8 ms | 4.0 | 0 | 1,184 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.