Speed test matrix
Kimi-K2.6-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.1.dev1+g9fedd24fa · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g9fedd24fa
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
9fedd24fa864
Run
- Served model
- Kimi-K2.6-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-vllm-pr-9fedd24fa864/bin/vllm" serve nvidia/Kimi-K2.6-NVFP4 --served-model-name Kimi-K2.6-NVFP4 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.90 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend TRITON_MLA --moe-backend flashinfer_cutlass
Prefill
prompt tok/s by context length8K context6,2161,317 ms TTFT
16K context6,0642,676 ms TTFT
32K context5,7415,627 ms TTFT
64K context5,17312,462 ms TTFT
128K context4,36329,514 ms TTFT
251K context3,29976,496 ms TTFT
256K context3,25479,118 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 55.085 ms TTFT | 98.2413 ms TTFT | 159.8207 ms TTFT | 257.5232 ms TTFT | 362.0440 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 53.5128 ms TTFT | 95.0470 ms TTFT | 150.2294 ms TTFT | 238.5386 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 52.0219 ms TTFT | 90.5587 ms TTFT | 142.3435 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 48.7406 ms TTFT | 81.6856 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 43.1763 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 35.81,481 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,216.0 | — | 1,317 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,064.0 | — | 2,676 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,741.0 | — | 5,627 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,173.0 | — | 12,462 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,363.0 | — | 29,514 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,299.0 | — | 76,496 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,254.0 | — | 79,118 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 55.0 | 55.0 | 85 ms | 18.2 ms | 1.0 | 0 | 1,648 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 98.2 | 48.7 | 413 ms | 20.5 ms | 2.0 | 0 | 2,944 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 159.8 | 40.1 | 207 ms | 24.9 ms | 4.0 | 0 | 4,792 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 257.5 | 32.3 | 232 ms | 30.9 ms | 8.0 | 0 | 7,720 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 362.0 | 22.9 | 440 ms | 43.7 ms | 16.0 | 0 | 10,848 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 53.5 | 53.4 | 128 ms | 18.7 ms | 1.0 | 0 | 1,603 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 95.0 | 47.2 | 470 ms | 21.2 ms | 2.0 | 0 | 2,850 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 150.2 | 37.8 | 294 ms | 26.5 ms | 4.0 | 0 | 4,504 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 238.5 | 30.0 | 386 ms | 33.4 ms | 8.0 | 0 | 7,151 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 52.0 | 52.0 | 219 ms | 19.2 ms | 1.0 | 0 | 1,558 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 90.5 | 44.8 | 587 ms | 22.3 ms | 2.0 | 0 | 2,713 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 142.3 | 35.5 | 435 ms | 28.2 ms | 4.0 | 0 | 4,264 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 48.7 | 48.7 | 406 ms | 20.5 ms | 1.0 | 0 | 1,461 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 81.6 | 40.4 | 856 ms | 24.8 ms | 2.0 | 0 | 2,448 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 43.1 | 43.1 | 763 ms | 23.2 ms | 1.0 | 0 | 1,293 | 30.0 s |
| Sustained decode | 128K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 35.8 | 35.8 | 1,481 ms | 27.9 ms | 1.0 | 0 | 1,074 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.