Speed test matrix
MiMo-V2.6-Flash-RL on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+ged073f5eb · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+ged073f5eb
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
ed073f5eb93c
Run
- Served model
- mimo-v2.6-flash-rl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-xiaomimimo-flash-vllm-pr-58177/bin/vllm" serve "${MIMO_MODEL_PATH:?Set MIMO_MODEL_PATH to the complete local XiaomiMiMo/MiMo-V2.6-Flash-RL checkpoint}" --served-model-name mimo-v2.6-flash-rl --trust-remote-code --tensor-parallel-size 4 --reasoning-parser mimo --enable-auto-tool-choice --tool-call-parser mimo --max-model-len 1048576 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config vllm --attention-backend TRITON_ATTN_DIFFKV --linear-backend deep_gemm --moe-backend deep_gemmPrefill
prompt tok/s by context length8K context7,9901,026 ms TTFT
16K context10.7K1,523 ms TTFT
32K context12.7K2,536 ms TTFT
64K context11.6K5,571 ms TTFT
128K context9,76713,189 ms TTFT
256K context7,16235,953 ms TTFT
512K context4,580112,405 ms TTFT
1019K context2,678382,645 ms TTFT
1024K context2,671385,497 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 135.050 ms TTFT | 232.362 ms TTFT | 336.183 ms TTFT | Looped | 682.2186 ms TTFT | 995.3283 ms TTFT | 1,541.9464 ms TTFT | 2,527.4833 ms TTFT |
| 16K | 129.459 ms TTFT | 223.0102 ms TTFT | 327.4121 ms TTFT | 473.2146 ms TTFT | 691.5256 ms TTFT | 977.3411 ms TTFT | 1,531.1736 ms TTFT | 2,435.81,366 ms TTFT |
| 32K | 118.1101 ms TTFT | 206.6164 ms TTFT | 309.1191 ms TTFT | 449.1263 ms TTFT | 671.8463 ms TTFT | 952.7767 ms TTFT | 1,464.01,285 ms TTFT | 2,334.22,367 ms TTFT |
| 64K | 101.4159 ms TTFT | 179.9256 ms TTFT | Looped | 415.8438 ms TTFT | 619.6754 ms TTFT | 901.71,388 ms TTFT | 1,369.32,616 ms TTFT | Over capacity |
| 128K | 78.8318 ms TTFT | 142.7465 ms TTFT | 230.1641 ms TTFT | 359.7963 ms TTFT | 562.91,690 ms TTFT | 809.63,193 ms TTFT | Over capacity | Over capacity |
| 256K | 54.1661 ms TTFT | 101.6911 ms TTFT | 173.41,313 ms TTFT | 289.22,132 ms TTFT | 453.93,917 ms TTFT | Over capacity | Over capacity | Over capacity |
| 512K | 33.71,486 ms TTFT | 64.82,128 ms TTFT | 117.53,087 ms TTFT | 202.05,101 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 17.7Requests queued3,285 ms TTFT | 36.64,667 ms TTFT | 71.16,937 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,990.0 | — | 1,026 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.7K | — | 1,523 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.7K | — | 2,536 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.6K | — | 5,571 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,767.0 | — | 13,189 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,162.0 | — | 35,953 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 4,580.0 | — | 112,405 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 2,678.0 | — | 382,645 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 2,671.0 | — | 385,497 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 135.0 | 136.4 | 50 ms | 7.3 ms | 1.0 | 1 | 4,050 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 232.3 | 116.3 | 62 ms | 8.6 ms | 2.0 | 0 | 6,968 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 336.1 | 84.2 | 83 ms | 11.9 ms | 4.0 | 0 | 10,080 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 682.2 | 44.3 | 186 ms | 22.6 ms | 16.0 | 0 | 20,460 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 995.3 | 31.5 | 283 ms | 31.8 ms | 32.0 | 0 | 29,856 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,541.9 | 24.6 | 464 ms | 40.7 ms | 64.0 | 0 | 46,239 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,527.4 | 20.1 | 833 ms | 49.8 ms | 128.0 | 0 | 75,778 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 129.4 | 130.5 | 59 ms | 7.7 ms | 1.0 | 1 | 3,881 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 223.0 | 111.7 | 102 ms | 9.0 ms | 2.0 | 0 | 6,688 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 327.4 | 82.0 | 121 ms | 12.2 ms | 4.0 | 0 | 9,820 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 473.2 | 59.3 | 146 ms | 16.8 ms | 8.0 | 0 | 14,193 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 691.5 | 43.7 | 256 ms | 22.9 ms | 16.0 | 0 | 20,736 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 977.3 | 30.8 | 411 ms | 32.4 ms | 32.0 | 0 | 29,311 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,531.1 | 24.2 | 736 ms | 41.3 ms | 64.0 | 0 | 45,889 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,435.8 | 19.3 | 1,366 ms | 51.8 ms | 128.0 | 0 | 72,961 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 118.1 | 118.2 | 101 ms | 8.5 ms | 1.0 | 0 | 3,542 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 206.6 | 103.4 | 164 ms | 9.7 ms | 2.0 | 0 | 6,198 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 309.1 | 77.3 | 191 ms | 12.9 ms | 4.0 | 0 | 9,272 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 449.1 | 56.4 | 263 ms | 17.7 ms | 8.0 | 0 | 13,470 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 671.8 | 42.2 | 463 ms | 23.7 ms | 16.0 | 0 | 20,144 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 952.7 | 29.9 | 767 ms | 33.4 ms | 32.0 | 0 | 28,576 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,464.0 | 23.0 | 1,285 ms | 43.4 ms | 64.0 | 0 | 43,903 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,334.2 | 18.4 | 2,367 ms | 54.5 ms | 128.0 | 0 | 70,017 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 101.4 | 101.5 | 159 ms | 9.9 ms | 1.0 | 0 | 3,041 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 179.9 | 90.0 | 256 ms | 11.1 ms | 2.0 | 0 | 5,396 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 415.8 | 52.0 | 438 ms | 19.2 ms | 8.0 | 0 | 12,472 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 619.6 | 38.8 | 754 ms | 25.8 ms | 16.0 | 0 | 18,587 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 901.7 | 28.1 | 1,388 ms | 35.5 ms | 32.0 | 0 | 27,040 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,369.3 | 21.3 | 2,616 ms | 46.9 ms | 64.0 | 0 | 41,025 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 78.8 | 78.8 | 318 ms | 12.7 ms | 1.0 | 0 | 2,362 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 142.7 | 71.4 | 465 ms | 14.0 ms | 2.0 | 0 | 4,282 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 230.1 | 57.4 | 641 ms | 17.4 ms | 4.0 | 0 | 6,904 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 359.7 | 44.9 | 963 ms | 22.2 ms | 8.0 | 0 | 10,784 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 562.9 | 35.0 | 1,690 ms | 28.6 ms | 16.0 | 0 | 16,880 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 809.6 | 25.2 | 3,193 ms | 39.8 ms | 32.0 | 0 | 24,289 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 54.1 | 54.2 | 661 ms | 18.4 ms | 1.0 | 0 | 1,624 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 101.6 | 50.8 | 911 ms | 19.7 ms | 2.0 | 0 | 3,046 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 173.4 | 43.2 | 1,313 ms | 23.1 ms | 4.0 | 0 | 5,200 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 289.2 | 36.0 | 2,132 ms | 27.8 ms | 8.0 | 0 | 8,672 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 453.9 | 28.2 | 3,917 ms | 35.5 ms | 16.0 | 0 | 13,616 | 30.0 s |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 33.7 | 33.8 | 1,486 ms | 29.6 ms | 1.0 | 0 | 1,009 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 64.8 | 32.4 | 2,128 ms | 30.9 ms | 2.0 | 0 | 1,942 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 117.5 | 29.2 | 3,087 ms | 34.3 ms | 4.0 | 0 | 3,523 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 202.0 | 25.0 | 5,101 ms | 40.0 ms | 8.0 | 0 | 6,056 | 30.0 s |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 17.7 | 19.4 | 3,285 ms | 51.4 ms | 0.9 | 0 | 530 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 36.6 | 18.3 | 4,667 ms | 54.5 ms | 2.0 | 0 | 1,098 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 71.1 | 17.5 | 6,937 ms | 57.1 ms | 4.0 | 0 | 2,132 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.