Speed test matrix
Kimi-K2.6-NVFP4 on H200 SXM 141GB ×8
vllm · 0.29.0 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- nvidia/Kimi-K2.6-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve nvidia/Kimi-K2.6-NVFP4 --served-model-name nvidia/Kimi-K2.6-NVFP4 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --max-num-batched-tokens 16384 --kernel-config.enable_flashinfer_autotune=False
Prefill
prompt tok/s by context length8K context17.6K466 ms TTFT
16K context18.7K869 ms TTFT
32K context17.5K1,845 ms TTFT
64K context15.4K4,193 ms TTFT
128K context12.4K10,402 ms TTFT
251K context8,92528,284 ms TTFT
256K context8,87829,001 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 8.6176 ms TTFT | 19.3442 ms TTFT | 41.3393 ms TTFT | 82.7399 ms TTFT | 165.8549 ms TTFT | 326.7710 ms TTFT | 630.81,164 ms TTFT | Over capacity |
| 16K | 8.6167 ms TTFT | 20.7292 ms TTFT | 41.4412 ms TTFT | 82.9562 ms TTFT | 165.1698 ms TTFT | 326.21,083 ms TTFT | Over capacity | Over capacity |
| 32K | 8.6220 ms TTFT | 20.6335 ms TTFT | 41.3517 ms TTFT | 82.5734 ms TTFT | 165.11,101 ms TTFT | 326.31,750 ms TTFT | Over capacity | Over capacity |
| 64K | 8.7277 ms TTFT | 20.6474 ms TTFT | 41.0703 ms TTFT | 82.61,075 ms TTFT | 165.01,704 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 8.8448 ms TTFT | 20.6694 ms TTFT | 41.21,167 ms TTFT | 82.51,749 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 8.4774 ms TTFT | 20.61,207 ms TTFT | 41.21,901 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.6K | — | 466 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.7K | — | 869 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 17.5K | — | 1,845 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.4K | — | 4,193 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.4K | — | 10,402 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 8,925.0 | — | 28,284 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,878.0 | — | 29,001 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 8.6 | 8.6 | 176 ms | 115.8 ms | 1.0 | 0 | 257 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 19.3 | 9.5 | 442 ms | 105.6 ms | 2.0 | 0 | 578 | 29.9 s |
| Sustained decode | 8K | 4 | Valid | 41.3 | 10.3 | 393 ms | 97.1 ms | 4.0 | 0 | 1,236 | 29.9 s |
| Sustained decode | 8K | 8 | Valid | 82.7 | 10.3 | 399 ms | 96.9 ms | 8.0 | 0 | 2,480 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 165.8 | 10.3 | 549 ms | 96.6 ms | 16.0 | 0 | 4,960 | 29.9 s |
| Sustained decode | 8K | 32 | Valid | 326.7 | 10.2 | 710 ms | 98.4 ms | 32.0 | 0 | 9,792 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 630.8 | 9.8 | 1,164 ms | 101.9 ms | 64.0 | 0 | 18,877 | 29.9 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 8.6 | 8.6 | 167 ms | 116.5 ms | 1.0 | 0 | 258 | 29.8 s |
| Sustained decode | 16K | 2 | Valid | 20.7 | 10.3 | 292 ms | 97.0 ms | 2.0 | 0 | 620 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 41.4 | 10.3 | 412 ms | 96.9 ms | 4.0 | 0 | 1,240 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 82.9 | 10.3 | 562 ms | 96.8 ms | 8.0 | 0 | 2,480 | 29.9 s |
| Sustained decode | 16K | 16 | Valid | 165.1 | 10.3 | 698 ms | 97.1 ms | 16.0 | 0 | 4,944 | 29.9 s |
| Sustained decode | 16K | 32 | Valid | 326.2 | 10.1 | 1,083 ms | 98.8 ms | 32.0 | 0 | 9,759 | 29.9 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 8.6 | 8.6 | 220 ms | 116.5 ms | 1.0 | 0 | 257 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 20.6 | 10.3 | 335 ms | 97.0 ms | 2.0 | 0 | 618 | 29.9 s |
| Sustained decode | 32K | 4 | Valid | 41.3 | 10.3 | 517 ms | 97.1 ms | 4.0 | 0 | 1,236 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 82.5 | 10.3 | 734 ms | 97.2 ms | 8.0 | 0 | 2,472 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 165.1 | 10.3 | 1,101 ms | 97.5 ms | 16.0 | 0 | 4,944 | 29.9 s |
| Sustained decode | 32K | 32 | Valid | 326.3 | 10.1 | 1,750 ms | 99.1 ms | 32.0 | 0 | 9,759 | 29.9 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 8.7 | 8.6 | 277 ms | 115.8 ms | 1.0 | 0 | 260 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 20.6 | 10.3 | 474 ms | 97.0 ms | 2.0 | 0 | 618 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 41.0 | 10.2 | 703 ms | 97.7 ms | 4.0 | 0 | 1,228 | 29.9 s |
| Sustained decode | 64K | 8 | Valid | 82.6 | 10.3 | 1,075 ms | 97.5 ms | 8.0 | 0 | 2,472 | 29.9 s |
| Sustained decode | 64K | 16 | Valid | 165.0 | 10.2 | 1,704 ms | 98.0 ms | 16.0 | 0 | 4,944 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 8.8 | 8.7 | 448 ms | 114.8 ms | 1.0 | 0 | 264 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 20.6 | 10.3 | 694 ms | 97.4 ms | 2.0 | 0 | 618 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 41.2 | 10.3 | 1,167 ms | 97.5 ms | 4.0 | 0 | 1,232 | 29.9 s |
| Sustained decode | 128K | 8 | Valid | 82.5 | 10.2 | 1,749 ms | 97.6 ms | 8.0 | 0 | 2,471 | 29.9 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 8.4 | 8.5 | 774 ms | 117.3 ms | 1.0 | 0 | 252 | 29.8 s |
| Sustained decode | 251K | 2 | Valid | 20.6 | 10.3 | 1,207 ms | 97.3 ms | 2.0 | 0 | 618 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 41.2 | 10.2 | 1,901 ms | 97.7 ms | 4.0 | 0 | 1,236 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.