Speed test matrix
MiMo-V2.6-Flash-RL on H200 NVL 141GB ×2
vllm · 0.29.1rc1.dev533+g1ea7c63f4 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.1rc1.dev533+g1ea7c63f4
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
1ea7c63f4af7
Run
- Served model
- mimo-v2.6-flash-rl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-xiaomimimo-flash-vllm-1ea7c63/bin/vllm" serve "${MIMO_MODEL_PATH:?Set MIMO_MODEL_PATH to the complete local XiaomiMiMo/MiMo-V2.6-Flash-RL checkpoint}" --served-model-name mimo-v2.6-flash-rl --trust-remote-code --tensor-parallel-size 2 --reasoning-parser mimo --enable-auto-tool-choice --tool-call-parser mimo --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config vllm --attention-backend FLASH_ATTN_DIFFKV --linear-backend flashinfer_cutlass --moe-backend marlinPrefill
prompt tok/s by context length8K context7,5891,080 ms TTFT
16K context10.5K1,544 ms TTFT
32K context12.6K2,569 ms TTFT
64K context12.6K5,129 ms TTFT
128K context11.4K11,281 ms TTFT
256K context8,99728,610 ms TTFT
512K context6,12484,038 ms TTFT
1019K context3,626282,481 ms TTFT
1024K context3,622284,128 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 170.354 ms TTFT | 281.792 ms TTFT | Looped | 535.6160 ms TTFT | 841.1231 ms TTFT | 1,120.6365 ms TTFT | 1,447.6570 ms TTFT | Over capacity |
| 16K | 168.587 ms TTFT | 280.172 ms TTFT | 416.3162 ms TTFT | 601.9219 ms TTFT | 820.7288 ms TTFT | 1,065.8468 ms TTFT | 1,416.4757 ms TTFT | Over capacity |
| 32K | 165.9122 ms TTFT | 276.6147 ms TTFT | 415.4247 ms TTFT | 596.3314 ms TTFT | 788.6481 ms TTFT | 1,026.3755 ms TTFT | Over capacity | Over capacity |
| 64K | 161.0185 ms TTFT | 268.7205 ms TTFT | 399.1378 ms TTFT | Looped | 755.4830 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 152.0308 ms TTFT | 254.4376 ms TTFT | 372.2600 ms TTFT | 518.4915 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 136.8583 ms TTFT | 234.7769 ms TTFT | 329.21,175 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 117.21,175 ms TTFT | 200.51,565 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 83.9Requests queued2,548 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,589.0 | — | 1,080 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.5K | — | 1,544 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.6K | — | 2,569 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.6K | — | 5,129 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.4K | — | 11,281 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,997.0 | — | 28,610 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 6,124.0 | — | 84,038 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 3,626.0 | — | 282,481 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 3,622.0 | — | 284,128 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 170.3 | 170.8 | 54 ms | 5.9 ms | 1.0 | 1 | 5,108 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 281.7 | 141.3 | 92 ms | 7.1 ms | 2.0 | 2 | 8,449 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Valid | 535.6 | 75.5 | 160 ms | 13.3 ms | 8.0 | 0 | 16,065 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 841.1 | 52.9 | 231 ms | 18.9 ms | 16.0 | 0 | 25,232 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,120.6 | 35.5 | 365 ms | 28.2 ms | 32.0 | 0 | 33,600 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,447.6 | 23.0 | 570 ms | 43.5 ms | 64.0 | 0 | 43,388 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 168.5 | 169.1 | 87 ms | 5.9 ms | 1.0 | 1 | 5,053 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 280.1 | 140.5 | 72 ms | 7.1 ms | 2.0 | 2 | 8,404 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 416.3 | 104.4 | 162 ms | 9.6 ms | 4.0 | 0 | 12,488 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 601.9 | 75.6 | 219 ms | 13.2 ms | 8.0 | 0 | 18,056 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 820.7 | 51.8 | 288 ms | 19.3 ms | 16.0 | 0 | 24,608 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,065.8 | 34.7 | 468 ms | 28.8 ms | 32.0 | 0 | 31,973 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,416.4 | 22.4 | 757 ms | 44.6 ms | 64.0 | 0 | 42,494 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 165.9 | 166.9 | 122 ms | 6.0 ms | 1.0 | 1 | 4,976 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 276.6 | 139.3 | 147 ms | 7.2 ms | 2.0 | 2 | 8,299 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 415.4 | 104.0 | 247 ms | 9.6 ms | 4.0 | 0 | 12,460 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 596.3 | 74.7 | 314 ms | 13.4 ms | 8.0 | 0 | 17,888 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 788.6 | 49.7 | 481 ms | 20.1 ms | 16.0 | 0 | 23,659 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,026.3 | 33.3 | 755 ms | 30.0 ms | 32.0 | 0 | 30,780 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 161.0 | 162.5 | 185 ms | 6.2 ms | 1.0 | 1 | 4,829 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 268.7 | 136.2 | 205 ms | 7.3 ms | 2.0 | 2 | 8,060 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 399.1 | 100.0 | 378 ms | 10.0 ms | 4.0 | 0 | 11,972 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 755.4 | 47.4 | 830 ms | 21.1 ms | 16.0 | 0 | 22,658 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 152.0 | 154.2 | 308 ms | 6.5 ms | 1.0 | 1 | 4,559 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 254.4 | 130.9 | 376 ms | 7.6 ms | 2.0 | 2 | 7,632 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 372.2 | 93.2 | 600 ms | 10.7 ms | 4.0 | 0 | 11,164 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 518.4 | 65.1 | 915 ms | 15.4 ms | 8.0 | 0 | 15,552 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 136.8 | 141.6 | 583 ms | 7.1 ms | 1.0 | 1 | 4,104 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 234.7 | 117.7 | 769 ms | 8.5 ms | 2.0 | 0 | 7,038 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 329.2 | 82.7 | 1,175 ms | 12.1 ms | 4.0 | 0 | 9,876 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 117.2 | 117.8 | 1,175 ms | 8.5 ms | 1.0 | 0 | 3,516 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 200.5 | 100.8 | 1,565 ms | 9.9 ms | 2.0 | 0 | 6,016 | 30.0 s |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 83.9 | 90.0 | 2,548 ms | 11.1 ms | 0.9 | 0 | 2,517 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.