Speed test matrix
Kimi-K2.6-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
sglang · 0.5.6.post3.dev11412+g964c45cf3 · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11412+g964c45cf3
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
964c45cf31e8
Run
- Served model
- Kimi-K2.6-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-sglang-964c45cf3/bin/sglang" serve --model-path nvidia/Kimi-K2.6-NVFP4 --served-model-name Kimi-K2.6-NVFP4 --trust-remote-code --tp 8 --reasoning-parser kimi_k2 --tool-call-parser kimi_k2 --context-length 131072 --mem-fraction-static 0.86 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton --moe-runner-backend flashinfer_cutlass --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context2,9292,795 ms TTFT
16K context2,5496,366 ms TTFT
32K context1,98916,241 ms TTFT
64K context1,38546,542 ms TTFT
123K context879.0140,814 ms TTFT
128K context859.0149,939 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 63.3120 ms TTFT | 122.5213 ms TTFT | 212.9845 ms TTFT | 341.7241 ms TTFT | 487.4249 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 60.8216 ms TTFT | 118.1364 ms TTFT | 204.6440 ms TTFT | 323.9431 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 57.7403 ms TTFT | 110.1696 ms TTFT | 187.9815 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 53.7789 ms TTFT | 100.41,391 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 46.6Requests queued1,504 ms TTFT | 87.92,598 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,929.0 | — | 2,795 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 2,549.0 | — | 6,366 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,989.0 | — | 16,241 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,385.0 | — | 46,542 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 879.0 | — | 140,814 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 859.0 | — | 149,939 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 63.3 | 63.5 | 120 ms | 15.7 ms | 1.0 | 0 | 1,902 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 122.5 | 61.3 | 213 ms | 16.3 ms | 2.0 | 0 | 3,676 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 212.9 | 53.4 | 845 ms | 18.7 ms | 4.0 | 0 | 6,391 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 341.7 | 43.2 | 241 ms | 23.2 ms | 8.0 | 0 | 10,247 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 487.4 | 31.0 | 249 ms | 32.2 ms | 16.0 | 0 | 14,592 | 29.9 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 60.8 | 61.0 | 216 ms | 16.4 ms | 1.0 | 0 | 1,824 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 118.1 | 59.1 | 364 ms | 16.9 ms | 2.0 | 0 | 3,548 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 204.6 | 51.2 | 440 ms | 19.5 ms | 4.0 | 0 | 6,143 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 323.9 | 40.9 | 431 ms | 24.4 ms | 8.0 | 0 | 9,718 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 57.7 | 57.8 | 403 ms | 17.3 ms | 1.0 | 0 | 1,733 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 110.1 | 55.0 | 696 ms | 18.2 ms | 2.0 | 0 | 3,298 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 187.9 | 47.1 | 815 ms | 21.2 ms | 4.0 | 0 | 5,634 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 53.7 | 53.7 | 789 ms | 18.6 ms | 1.0 | 0 | 1,611 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 100.4 | 50.0 | 1,391 ms | 20.0 ms | 2.0 | 0 | 3,010 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Requests queued | 46.6 | 48.1 | 1,504 ms | 20.8 ms | 0.9 | 0 | 1,398 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 87.9 | 43.6 | 2,598 ms | 23.0 ms | 2.0 | 0 | 2,636 | 30.0 s |
| Sustained decode | 123K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.