Speed test matrix
MiMo-V2.6-Flash-RL on H200 NVL 141GB ×2
vllm · 0.29.1rc1.dev533+g1ea7c63f4 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.1rc1.dev533+g1ea7c63f4
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
1ea7c63f4af7
Run
- Served model
- mimo-v2.6-flash-rl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-xiaomimimo-flash-vllm-1ea7c63/bin/vllm" serve "${MIMO_MODEL_PATH:?Set MIMO_MODEL_PATH to the complete local XiaomiMiMo/MiMo-V2.6-Flash-RL checkpoint}" --served-model-name mimo-v2.6-flash-rl --trust-remote-code --tensor-parallel-size 2 --reasoning-parser mimo --enable-auto-tool-choice --tool-call-parser mimo --max-model-len 1048576 --gpu-memory-utilization 0.87 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config vllm --attention-backend FLASH_ATTN_DIFFKV --linear-backend flashinfer_cutlass --moe-backend marlin --speculative-config "{\"method\":\"dflash\",\"model\":\"${MIMO_MODEL_PATH:?Set MIMO_MODEL_PATH to the complete local XiaomiMiMo/MiMo-V2.6-Flash-RL checkpoint}/dflash\",\"num_speculative_tokens\":7}"Prefill
prompt tok/s by context length8K context7,9591,030 ms TTFT
16K context10.5K1,543 ms TTFT
32K context12.6K2,571 ms TTFT
64K context12.5K5,139 ms TTFT
128K context11.4K11,298 ms TTFT
256K context8,95428,748 ms TTFT
512K context6,07684,714 ms TTFT
1019K context3,607283,959 ms TTFT
1024K context3,604285,550 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 206.769 ms TTFT | 302.887 ms TTFT | Looped | 697.2226 ms TTFT | Looped | Looped | 1,814.6750 ms TTFT | 2,986.61,145 ms TTFT |
| 16K | 212.8104 ms TTFT | 337.0126 ms TTFT | 483.1161 ms TTFT | 668.0291 ms TTFT | Looped | 1,383.9620 ms TTFT | 1,785.4944 ms TTFT | Over capacity |
| 32K | 222.6131 ms TTFT | 329.7151 ms TTFT | 464.0204 ms TTFT | 683.2376 ms TTFT | 920.5584 ms TTFT | 1,358.7911 ms TTFT | Over capacity | Over capacity |
| 64K | 224.2195 ms TTFT | 297.7230 ms TTFT | 432.4382 ms TTFT | 678.3498 ms TTFT | 879.5867 ms TTFT | 1,295.81,571 ms TTFT | Over capacity | Over capacity |
| 128K | 181.3314 ms TTFT | 287.1386 ms TTFT | 435.9674 ms TTFT | 589.31,073 ms TTFT | 781.01,832 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 176.3609 ms TTFT | 288.5780 ms TTFT | 374.41,507 ms TTFT | 552.82,275 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 163.41,195 ms TTFT | 250.61,414 ms TTFT | 346.93,076 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 145.6Requests queued2,426 ms TTFT | 212.3Requests queued3,932 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,959.0 | — | 1,030 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.5K | — | 1,543 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.6K | — | 2,571 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.5K | — | 5,139 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.4K | — | 11,298 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,954.0 | — | 28,748 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 6,076.0 | — | 84,714 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 3,607.0 | — | 283,959 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 3,604.0 | — | 285,550 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 206.7 | 209.4 | 69 ms | 4.8 ms | 1.0 | 1 | 6,199 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 302.8 | 155.9 | 87 ms | 6.4 ms | 2.0 | 2 | 9,088 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Valid | 697.2 | 83.9 | 226 ms | 11.9 ms | 8.0 | 1 | 20,916 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 1,814.6 | 28.9 | 750 ms | 34.6 ms | 64.0 | 0 | 54,443 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,986.6 | 22.8 | 1,145 ms | 43.8 ms | 128.0 | 0 | 89,498 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 212.8 | 214.0 | 104 ms | 4.7 ms | 1.0 | 1 | 6,384 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 337.0 | 168.9 | 126 ms | 5.9 ms | 2.0 | 2 | 10,116 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 483.1 | 129.7 | 161 ms | 7.7 ms | 4.0 | 3 | 14,498 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 668.0 | 85.3 | 291 ms | 11.7 ms | 8.0 | 0 | 20,038 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 1,383.9 | 44.2 | 620 ms | 22.6 ms | 32.0 | 0 | 41,496 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,785.4 | 28.7 | 944 ms | 34.8 ms | 64.0 | 0 | 53,417 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 222.6 | 229.4 | 131 ms | 4.4 ms | 1.0 | 1 | 6,677 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 329.7 | 174.3 | 151 ms | 5.8 ms | 2.0 | 2 | 9,893 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 464.0 | 119.4 | 204 ms | 8.4 ms | 4.0 | 1 | 13,910 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 683.2 | 86.5 | 376 ms | 11.6 ms | 8.0 | 0 | 20,496 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 920.5 | 58.8 | 584 ms | 17.0 ms | 16.0 | 0 | 27,612 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,358.7 | 43.7 | 911 ms | 22.9 ms | 32.0 | 0 | 40,714 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 224.2 | 232.3 | 195 ms | 4.3 ms | 1.0 | 1 | 6,725 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 297.7 | 155.2 | 230 ms | 6.4 ms | 2.0 | 2 | 8,934 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 432.4 | 113.2 | 382 ms | 8.8 ms | 4.0 | 0 | 12,969 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 678.3 | 80.8 | 498 ms | 12.4 ms | 8.0 | 1 | 20,343 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 879.5 | 56.0 | 867 ms | 17.9 ms | 16.0 | 0 | 26,371 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,295.8 | 41.7 | 1,571 ms | 24.0 ms | 32.0 | 0 | 38,862 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 181.3 | 194.5 | 314 ms | 5.2 ms | 1.0 | 1 | 5,439 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 287.1 | 155.4 | 386 ms | 6.5 ms | 2.0 | 2 | 8,610 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 435.9 | 114.9 | 674 ms | 8.7 ms | 4.0 | 1 | 13,073 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 589.3 | 74.9 | 1,073 ms | 13.4 ms | 8.0 | 0 | 17,683 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 781.0 | 53.0 | 1,832 ms | 18.9 ms | 16.0 | 0 | 23,414 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 176.3 | 188.4 | 609 ms | 5.3 ms | 1.0 | 1 | 5,287 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 288.5 | 153.4 | 780 ms | 6.5 ms | 1.9 | 2 | 8,653 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 374.4 | 98.2 | 1,507 ms | 10.2 ms | 4.0 | 0 | 11,231 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 552.8 | 72.5 | 2,275 ms | 13.8 ms | 8.0 | 0 | 16,581 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 163.4 | 174.4 | 1,195 ms | 5.7 ms | 1.0 | 1 | 4,900 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 250.6 | 137.7 | 1,414 ms | 7.3 ms | 1.9 | 2 | 7,518 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 346.9 | 91.6 | 3,076 ms | 10.9 ms | 4.0 | 0 | 10,404 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Requests queued | 145.6 | 156.3 | 2,426 ms | 6.4 ms | 0.9 | 1 | 4,096 | 28.1 s |
| Sustained decode | 1019K | 2 | Requests queued | 212.3 | 120.0 | 3,932 ms | 8.3 ms | 1.8 | 0 | 6,369 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.