Speed test matrix
Kimi-K2.7-Code-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.1.dev1+g9fedd24fa · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g9fedd24fa
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
9fedd24fa864
Run
- Served model
- Kimi-K2.7-Code-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 "$HOME/env_rtxpro6k-nvidia-vllm-pr-9fedd24fa864/bin/vllm" serve nvidia/Kimi-K2.7-Code-NVFP4 --served-model-name Kimi-K2.7-Code-NVFP4 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser kimi_k2 --enable-auto-tool-choice --tool-call-parser kimi_k2 --max-model-len 262144 --gpu-memory-utilization 0.90 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend TRITON_MLA --moe-backend flashinfer_cutlass
Prefill
prompt tok/s by context length8K context6,2061,320 ms TTFT
16K context6,0592,679 ms TTFT
32K context5,7405,629 ms TTFT
64K context5,17612,459 ms TTFT
128K context4,35529,575 ms TTFT
251K context3,30076,494 ms TTFT
256K context3,25379,160 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 54.987 ms TTFT | 98.0399 ms TTFT | 164.6208 ms TTFT | 259.7234 ms TTFT | 364.0418 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 53.3130 ms TTFT | 94.7465 ms TTFT | 155.5290 ms TTFT | 236.6392 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 52.0227 ms TTFT | 90.0589 ms TTFT | 145.6576 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 48.9400 ms TTFT | 81.4846 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 43.0782 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 35.71,508 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,206.0 | — | 1,320 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,059.0 | — | 2,679 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,740.0 | — | 5,629 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,176.0 | — | 12,459 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,355.0 | — | 29,575 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,300.0 | — | 76,494 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,253.0 | — | 79,160 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 54.9 | 54.9 | 87 ms | 18.2 ms | 1.0 | 0 | 1,647 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 98.0 | 48.6 | 399 ms | 20.6 ms | 2.0 | 0 | 2,940 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 164.6 | 41.2 | 208 ms | 24.3 ms | 4.0 | 0 | 4,936 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 259.7 | 32.5 | 234 ms | 30.8 ms | 8.0 | 0 | 7,784 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 364.0 | 22.8 | 418 ms | 43.8 ms | 16.0 | 0 | 10,912 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 53.3 | 53.3 | 130 ms | 18.8 ms | 1.0 | 0 | 1,600 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 94.7 | 46.9 | 465 ms | 21.3 ms | 2.0 | 0 | 2,840 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 155.5 | 38.9 | 290 ms | 25.7 ms | 4.0 | 0 | 4,664 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 236.6 | 29.6 | 392 ms | 33.8 ms | 8.0 | 0 | 7,096 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 52.0 | 52.0 | 227 ms | 19.2 ms | 1.0 | 0 | 1,561 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 90.0 | 44.6 | 589 ms | 22.4 ms | 2.0 | 0 | 2,698 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 145.6 | 36.4 | 576 ms | 27.5 ms | 4.0 | 0 | 4,368 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 48.9 | 48.8 | 400 ms | 20.5 ms | 1.0 | 0 | 1,466 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 81.4 | 40.2 | 846 ms | 24.9 ms | 2.0 | 0 | 2,440 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 43.0 | 43.0 | 782 ms | 23.2 ms | 1.0 | 0 | 1,290 | 30.0 s |
| Sustained decode | 128K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 35.7 | 35.8 | 1,508 ms | 28.0 ms | 1.0 | 0 | 1,071 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.