Speed test matrix
MiMo-V2.6-Pro-RL on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.1.dev1+ged073f5eb · MXFP4+FP87 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+ged073f5eb
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
ed073f5eb93c
Run
- Served model
- mimo-v2.6-pro-rl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-xiaomimimo-flash-vllm-pr-58177/bin/vllm" serve "${HF_HOME:-$HOME/.cache/huggingface}/mimo-v2.6-pro-rl" --served-model-name mimo-v2.6-pro-rl --trust-remote-code --tensor-parallel-size 8 --reasoning-parser mimo --enable-auto-tool-choice --tool-call-parser mimo --max-model-len 524288 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config vllm --attention-backend TRITON_ATTN_DIFFKV --linear-backend deep_gemm --moe-backend deep_gemmPrefill
prompt tok/s by context length8K context5,3511,532 ms TTFT
16K context5,3363,041 ms TTFT
32K context5,8025,569 ms TTFT
64K context5,53611,647 ms TTFT
128K context4,98825,818 ms TTFT
256K context4,10162,775 ms TTFT
507K context3,013169,129 ms TTFT
512K context3,010170,997 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 66.562 ms TTFT | 123.087 ms TTFT | 189.2112 ms TTFT | 277.3149 ms TTFT | 415.9240 ms TTFT | 553.8350 ms TTFT | Over capacity | Over capacity |
| 16K | 63.765 ms TTFT | 120.087 ms TTFT | 185.4125 ms TTFT | 276.2179 ms TTFT | 410.5266 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 60.3106 ms TTFT | 114.6141 ms TTFT | 178.4199 ms TTFT | 271.2271 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 55.3184 ms TTFT | 105.0235 ms TTFT | 166.7322 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 47.0340 ms TTFT | 89.4453 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 36.1690 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 25.21,549 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,351.0 | — | 1,532 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,336.0 | — | 3,041 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,802.0 | — | 5,569 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,536.0 | — | 11,647 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,988.0 | — | 25,818 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,101.0 | — | 62,775 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 3,013.0 | — | 169,129 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 3,010.0 | — | 170,997 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 66.5 | 66.6 | 62 ms | 15.0 ms | 1.0 | 0 | 1,996 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 123.0 | 61.5 | 87 ms | 16.3 ms | 2.0 | 0 | 3,690 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 189.2 | 47.4 | 112 ms | 21.1 ms | 4.0 | 0 | 5,676 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 277.3 | 34.9 | 149 ms | 28.7 ms | 8.0 | 0 | 8,312 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 415.9 | 26.2 | 240 ms | 38.2 ms | 16.0 | 0 | 12,464 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 553.8 | 17.6 | 350 ms | 56.9 ms | 32.0 | 0 | 16,608 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 63.7 | 63.7 | 65 ms | 15.7 ms | 1.0 | 0 | 1,911 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 120.0 | 60.1 | 87 ms | 16.7 ms | 2.0 | 0 | 3,600 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 185.4 | 46.4 | 125 ms | 21.6 ms | 4.0 | 0 | 5,560 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 276.2 | 34.7 | 179 ms | 28.8 ms | 8.0 | 0 | 8,280 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 410.5 | 25.8 | 266 ms | 38.7 ms | 16.0 | 0 | 12,304 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 60.3 | 60.4 | 106 ms | 16.6 ms | 1.0 | 0 | 1,810 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 114.6 | 57.3 | 141 ms | 17.5 ms | 2.0 | 0 | 3,436 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 178.4 | 44.8 | 199 ms | 22.3 ms | 4.0 | 0 | 5,352 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 271.2 | 34.1 | 271 ms | 29.4 ms | 8.0 | 0 | 8,128 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 55.3 | 55.3 | 184 ms | 18.1 ms | 1.0 | 0 | 1,658 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 105.0 | 52.5 | 235 ms | 19.1 ms | 2.0 | 0 | 3,148 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 166.7 | 41.6 | 322 ms | 24.0 ms | 4.0 | 0 | 4,996 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 47.0 | 47.1 | 340 ms | 21.2 ms | 1.0 | 0 | 1,410 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 89.4 | 44.7 | 453 ms | 22.3 ms | 2.0 | 0 | 2,682 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 36.1 | 36.2 | 690 ms | 27.6 ms | 1.0 | 0 | 1,082 | 30.0 s |
| Sustained decode | 256K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 25.2 | 25.3 | 1,549 ms | 39.6 ms | 1.0 | 0 | 755 | 30.0 s |
| Sustained decode | 507K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.