Speed test matrix
MiMo-V2.6-Flash-RL on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+ged073f5eb · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+ged073f5eb
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
ed073f5eb93c
Run
- Served model
- mimo-v2.6-flash-rl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-xiaomimimo-flash-vllm-pr-58177/bin/vllm" serve "${MIMO_MODEL_PATH:?Set MIMO_MODEL_PATH to the complete local XiaomiMiMo/MiMo-V2.6-Flash-RL checkpoint}" --served-model-name mimo-v2.6-flash-rl --trust-remote-code --tensor-parallel-size 4 --reasoning-parser mimo --enable-auto-tool-choice --tool-call-parser mimo --max-model-len 1048576 --gpu-memory-utilization 0.80 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config vllm --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' --attention-backend TRITON_ATTN_DIFFKV --linear-backend deep_gemm --moe-backend deep_gemm --speculative-config "{\"method\":\"dflash\",\"model\":\"${MIMO_MODEL_PATH:?Set MIMO_MODEL_PATH to the complete local XiaomiMiMo/MiMo-V2.6-Flash-RL checkpoint}/dflash\",\"num_speculative_tokens\":7}"Prefill
prompt tok/s by context length8K context2,6783,062 ms TTFT
16K context10.7K1,521 ms TTFT
32K context12.7K2,535 ms TTFT
64K context11.6K5,576 ms TTFT
128K context9,77813,176 ms TTFT
256K context7,15835,973 ms TTFT
512K context4,580112,427 ms TTFT
1019K context2,674383,203 ms TTFT
1024K context2,668385,895 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 116.185 ms TTFT | 177.2121 ms TTFT | 265.7159 ms TTFT | 415.0187 ms TTFT | 695.9269 ms TTFT | Looped | 1,775.5655 ms TTFT | 2,567.81,062 ms TTFT |
| 16K | 110.494 ms TTFT | 165.2156 ms TTFT | 250.4180 ms TTFT | 412.2239 ms TTFT | 674.6344 ms TTFT | 1,108.0536 ms TTFT | 1,652.0916 ms TTFT | 2,447.01,608 ms TTFT |
| 32K | 79.7128 ms TTFT | 126.1220 ms TTFT | 213.3273 ms TTFT | 364.8400 ms TTFT | 600.6568 ms TTFT | 1,037.4898 ms TTFT | 1,432.41,538 ms TTFT | Over capacity |
| 64K | 58.3199 ms TTFT | Looped | 157.9421 ms TTFT | 309.3645 ms TTFT | Looped | Looped | Over capacity | Over capacity |
| 128K | 36.2369 ms TTFT | 68.9574 ms TTFT | 124.8787 ms TTFT | 222.11,179 ms TTFT | Looped | Over capacity | Over capacity | Over capacity |
| 256K | 26.4725 ms TTFT | 46.21,039 ms TTFT | 102.31,555 ms TTFT | 135.32,420 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 14.21,546 ms TTFT | 33.72,320 ms TTFT | 51.63,531 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 6.5Requests queued3,536 ms TTFT | 15.24,887 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,678.0 | — | 3,062 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.7K | — | 1,521 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.7K | — | 2,535 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.6K | — | 5,576 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,778.0 | — | 13,176 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,158.0 | — | 35,973 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 4,580.0 | — | 112,427 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 2,674.0 | — | 383,203 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 2,668.0 | — | 385,895 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 116.1 | 118.2 | 85 ms | 8.5 ms | 1.0 | 0 | 3,483 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 177.2 | 92.2 | 121 ms | 10.9 ms | 2.0 | 0 | 5,313 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 265.7 | 69.5 | 159 ms | 14.4 ms | 4.0 | 0 | 7,968 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 415.0 | 54.7 | 187 ms | 18.3 ms | 8.0 | 0 | 12,449 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 695.9 | 46.7 | 269 ms | 21.4 ms | 16.0 | 0 | 20,848 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 1,775.5 | 27.9 | 655 ms | 35.8 ms | 64.0 | 0 | 53,260 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,567.8 | 20.0 | 1,062 ms | 50.0 ms | 128.0 | 0 | 76,823 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 110.4 | 111.4 | 94 ms | 9.0 ms | 1.0 | 0 | 3,309 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 165.2 | 83.5 | 156 ms | 12.0 ms | 2.0 | 0 | 4,954 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 250.4 | 65.1 | 180 ms | 15.4 ms | 4.0 | 0 | 7,509 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 412.2 | 53.1 | 239 ms | 18.8 ms | 8.0 | 0 | 12,364 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 674.6 | 44.9 | 344 ms | 22.3 ms | 16.0 | 0 | 20,235 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,108.0 | 36.0 | 536 ms | 27.8 ms | 32.0 | 0 | 33,187 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,652.0 | 26.6 | 916 ms | 37.5 ms | 64.0 | 0 | 49,510 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,447.0 | 19.3 | 1,608 ms | 51.8 ms | 128.0 | 0 | 73,149 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 79.7 | 86.3 | 128 ms | 11.6 ms | 1.0 | 0 | 2,388 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 126.1 | 65.5 | 220 ms | 15.4 ms | 2.0 | 0 | 3,782 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 213.3 | 55.0 | 273 ms | 18.2 ms | 4.0 | 0 | 6,394 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 364.8 | 45.4 | 400 ms | 22.0 ms | 8.0 | 0 | 10,933 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 600.6 | 38.4 | 568 ms | 26.0 ms | 16.0 | 0 | 18,008 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,037.4 | 33.9 | 898 ms | 29.5 ms | 32.0 | 0 | 31,108 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,432.4 | 23.3 | 1,538 ms | 42.8 ms | 64.0 | 0 | 42,864 | 29.9 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 58.3 | 59.2 | 199 ms | 16.9 ms | 1.0 | 0 | 1,746 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Valid | 157.9 | 43.0 | 421 ms | 23.3 ms | 4.0 | 0 | 4,734 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 309.3 | 39.1 | 645 ms | 25.6 ms | 8.0 | 0 | 9,275 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 36.2 | 36.6 | 369 ms | 27.3 ms | 1.0 | 0 | 1,083 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 68.9 | 36.9 | 574 ms | 27.8 ms | 2.0 | 0 | 2,066 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 124.8 | 32.8 | 787 ms | 30.6 ms | 4.0 | 0 | 3,741 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 222.1 | 28.0 | 1,179 ms | 35.7 ms | 8.0 | 0 | 6,664 | 30.0 s |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 26.4 | 27.3 | 725 ms | 36.6 ms | 1.0 | 0 | 790 | 29.9 s |
| Sustained decode | 256K | 2 | Valid | 46.2 | 23.9 | 1,039 ms | 41.9 ms | 2.0 | 0 | 1,382 | 29.9 s |
| Sustained decode | 256K | 4 | Valid | 102.3 | 27.5 | 1,555 ms | 36.4 ms | 4.0 | 0 | 3,065 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 135.3 | 17.8 | 2,420 ms | 56.3 ms | 8.0 | 0 | 4,057 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 14.2 | 14.9 | 1,546 ms | 67.1 ms | 1.0 | 0 | 425 | 29.9 s |
| Sustained decode | 512K | 2 | Valid | 33.7 | 16.7 | 2,320 ms | 60.6 ms | 2.0 | 0 | 1,006 | 29.8 s |
| Sustained decode | 512K | 4 | Valid | 51.6 | 13.3 | 3,531 ms | 75.2 ms | 4.0 | 0 | 1,538 | 29.8 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 6.5 | 7.3 | 3,536 ms | 137.8 ms | 0.9 | 0 | 194 | 29.6 s |
| Sustained decode | 1019K | 2 | Valid | 15.2 | 8.2 | 4,887 ms | 123.8 ms | 2.0 | 0 | 450 | 29.7 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.