Speed test matrix
MiMo-V2.6-Pro-RL on H200 SXM 141GB ×8
vllm · 0.29.1rc1.dev533+g1ea7c63f4 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.1rc1.dev533+g1ea7c63f4
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
1ea7c63f4af7
Run
- Served model
- mimo-v2.6-pro-rl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-xiaomimimo-flash-vllm-1ea7c63/bin/vllm" serve "${HF_HOME:-$HOME/.cache/huggingface}/mimo-v2.6-pro-rl" --served-model-name mimo-v2.6-pro-rl --trust-remote-code --tensor-parallel-size 8 --reasoning-parser mimo --enable-auto-tool-choice --tool-call-parser mimo --max-model-len 1048576 --gpu-memory-utilization 0.85 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config vllm --attention-backend FLASH_ATTN_DIFFKV --linear-backend flashinfer_cutlass --moe-backend marlin --speculative-config "{\"method\":\"dflash\",\"model\":\"${HF_HOME:-$HOME/.cache/huggingface}/mimo-v2.6-pro-rl/dflash\",\"num_speculative_tokens\":7}"Prefill
prompt tok/s by context length8K context16.1K510 ms TTFT
16K context15.9K1,022 ms TTFT
32K context15.8K2,042 ms TTFT
64K context15.8K4,083 ms TTFT
128K context14.8K8,682 ms TTFT
256K context12.9K19,963 ms TTFT
512K context9,50354,158 ms TTFT
1019K context6,180165,717 ms TTFT
1024K context6,178166,584 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 231.062 ms TTFT | 401.676 ms TTFT | 592.089 ms TTFT | 812.5179 ms TTFT | 1,196.8270 ms TTFT | 1,765.8371 ms TTFT | 2,530.9750 ms TTFT | 3,636.71,378 ms TTFT |
| 16K | 218.481 ms TTFT | 397.196 ms TTFT | 605.696 ms TTFT | 817.0223 ms TTFT | 1,207.4360 ms TTFT | 1,693.4585 ms TTFT | 2,509.71,064 ms TTFT | 3,559.12,115 ms TTFT |
| 32K | 222.8116 ms TTFT | 377.9125 ms TTFT | 579.5128 ms TTFT | 816.1296 ms TTFT | 1,199.2511 ms TTFT | 1,640.1990 ms TTFT | 2,454.21,827 ms TTFT | 3,432.03,553 ms TTFT |
| 64K | 245.3202 ms TTFT | 412.3240 ms TTFT | 572.8218 ms TTFT | 818.0619 ms TTFT | Looped | 1,614.91,935 ms TTFT | 2,383.13,520 ms TTFT | 3,186.16,987 ms TTFT |
| 128K | 209.7393 ms TTFT | 368.4444 ms TTFT | 593.4517 ms TTFT | 776.91,193 ms TTFT | 1,146.22,113 ms TTFT | 1,520.03,954 ms TTFT | 2,041.17,686 ms TTFT | Over capacity |
| 256K | 206.4821 ms TTFT | 371.2885 ms TTFT | 570.01,103 ms TTFT | Looped | 1,068.64,642 ms TTFT | 1,334.88,809 ms TTFT | Over capacity | Over capacity |
| 512K | 175.91,867 ms TTFT | 328.11,875 ms TTFT | 473.61,898 ms TTFT | 604.96,346 ms TTFT | 899.412,132 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | 121.24,087 ms TTFT | 243.1Requests queued5,775 ms TTFT | 411.4Requests queued7,384 ms TTFT | 520.113,350 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.1K | — | 510 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.9K | — | 1,022 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.8K | — | 2,042 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.8K | — | 4,083 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.8K | — | 8,682 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.9K | — | 19,963 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,503.0 | — | 54,158 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,180.0 | — | 165,717 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 6,178.0 | — | 166,584 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 231.0 | 233.2 | 62 ms | 4.3 ms | 1.0 | 1 | 6,928 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 401.6 | 202.1 | 76 ms | 4.9 ms | 2.0 | 2 | 12,046 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 592.0 | 155.8 | 89 ms | 6.4 ms | 4.0 | 4 | 17,754 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 812.5 | 104.7 | 179 ms | 9.5 ms | 8.0 | 0 | 24,367 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,196.8 | 78.9 | 270 ms | 12.7 ms | 16.0 | 0 | 35,871 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,765.8 | 56.2 | 371 ms | 17.8 ms | 32.0 | 1 | 52,918 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,530.9 | 41.0 | 750 ms | 24.4 ms | 64.0 | 0 | 75,884 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,636.7 | 27.1 | 1,378 ms | 36.9 ms | 128.0 | 0 | 109,068 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 218.4 | 224.1 | 81 ms | 4.5 ms | 1.0 | 1 | 6,553 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 397.1 | 200.1 | 96 ms | 5.0 ms | 2.0 | 2 | 11,912 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 605.6 | 160.1 | 96 ms | 6.2 ms | 4.0 | 4 | 18,169 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 817.0 | 106.9 | 223 ms | 9.4 ms | 8.0 | 0 | 24,496 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,207.4 | 79.8 | 360 ms | 12.5 ms | 16.0 | 0 | 36,204 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,693.4 | 55.1 | 585 ms | 18.1 ms | 32.0 | 0 | 50,762 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,509.7 | 40.5 | 1,064 ms | 24.7 ms | 64.0 | 0 | 75,261 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,559.1 | 28.6 | 2,115 ms | 35.0 ms | 128.0 | 0 | 106,640 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 222.8 | 232.2 | 116 ms | 4.3 ms | 1.0 | 1 | 6,682 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 377.9 | 188.0 | 125 ms | 5.3 ms | 2.0 | 2 | 11,336 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 579.5 | 159.5 | 128 ms | 6.3 ms | 4.0 | 4 | 17,383 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 816.1 | 104.8 | 296 ms | 9.5 ms | 8.0 | 0 | 24,467 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,199.2 | 79.4 | 511 ms | 12.6 ms | 16.0 | 0 | 35,954 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,640.1 | 53.2 | 990 ms | 18.8 ms | 32.0 | 0 | 49,163 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,454.2 | 40.3 | 1,827 ms | 24.8 ms | 64.0 | 0 | 73,520 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,432.0 | 28.9 | 3,553 ms | 34.6 ms | 128.0 | 0 | 102,731 | 29.9 s |
| Sustained decode | 64K | 1 | Valid | 245.3 | 253.7 | 202 ms | 3.9 ms | 1.0 | 2 | 7,360 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 412.3 | 193.6 | 240 ms | 5.2 ms | 2.0 | 2 | 12,369 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 572.8 | 148.1 | 218 ms | 6.8 ms | 4.0 | 4 | 17,174 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 818.0 | 105.7 | 619 ms | 9.5 ms | 8.0 | 0 | 24,537 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 1,614.9 | 54.3 | 1,935 ms | 18.4 ms | 32.0 | 0 | 48,402 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,383.1 | 39.3 | 3,520 ms | 25.5 ms | 64.0 | 0 | 71,347 | 29.9 s |
| Sustained decode | 64K | 128 | Valid | 3,186.1 | 27.3 | 6,987 ms | 36.6 ms | 128.0 | 0 | 95,503 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 209.7 | 220.3 | 393 ms | 4.5 ms | 1.0 | 1 | 6,289 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 368.4 | 192.1 | 444 ms | 5.2 ms | 2.0 | 2 | 11,050 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 593.4 | 155.1 | 517 ms | 6.4 ms | 4.0 | 4 | 17,796 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 776.9 | 102.8 | 1,193 ms | 9.7 ms | 8.0 | 0 | 23,303 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,146.2 | 75.7 | 2,113 ms | 13.2 ms | 16.0 | 0 | 34,386 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,520.0 | 50.2 | 3,954 ms | 19.9 ms | 32.0 | 0 | 45,531 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 2,041.1 | 35.8 | 7,686 ms | 28.0 ms | 64.0 | 0 | 61,222 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 206.4 | 216.2 | 821 ms | 4.6 ms | 1.0 | 1 | 6,190 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 371.2 | 196.8 | 885 ms | 5.1 ms | 1.9 | 2 | 11,132 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 570.0 | 148.3 | 1,103 ms | 6.7 ms | 3.9 | 4 | 17,096 | 30.0 s |
| Sustained decode | 256K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 16 | Valid | 1,068.6 | 69.1 | 4,642 ms | 14.5 ms | 16.0 | 0 | 32,033 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,334.8 | 46.3 | 8,809 ms | 21.6 ms | 32.0 | 0 | 39,980 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 175.9 | 191.3 | 1,867 ms | 5.2 ms | 0.9 | 1 | 5,275 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 328.1 | 176.7 | 1,875 ms | 5.7 ms | 1.9 | 2 | 9,839 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 473.6 | 130.6 | 1,898 ms | 7.7 ms | 3.8 | 4 | 14,202 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 604.9 | 90.0 | 6,346 ms | 11.1 ms | 8.0 | 1 | 18,136 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 899.4 | 58.1 | 12,132 ms | 17.2 ms | 16.0 | 0 | 26,952 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 121.2 | 151.6 | 4,087 ms | 6.6 ms | 0.9 | 1 | 3,635 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 243.1 | 127.1 | 5,775 ms | 7.9 ms | 1.8 | 2 | 6,785 | 27.9 s |
| Sustained decode | 1019K | 4 | Requests queued | 411.4 | 120.9 | 7,384 ms | 8.3 ms | 3.6 | 3 | 12,146 | 29.5 s |
| Sustained decode | 1019K | 8 | Valid | 520.1 | 72.8 | 13,350 ms | 13.7 ms | 7.9 | 1 | 15,601 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.