Speed test matrix
MiMo-V2.6-Pro-RL on H200 SXM 141GB ×8
vllm · 0.29.1rc1.dev533+g1ea7c63f4 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.1rc1.dev533+g1ea7c63f4
- Quantization
- MXFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
1ea7c63f4af7
Run
- Served model
- mimo-v2.6-pro-rl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-xiaomimimo-flash-vllm-1ea7c63/bin/vllm" serve "${HF_HOME:-$HOME/.cache/huggingface}/mimo-v2.6-pro-rl" --served-model-name mimo-v2.6-pro-rl --trust-remote-code --tensor-parallel-size 8 --reasoning-parser mimo --enable-auto-tool-choice --tool-call-parser mimo --max-model-len 1048576 --gpu-memory-utilization 0.87 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config vllm --attention-backend FLASH_ATTN_DIFFKV --linear-backend flashinfer_cutlass --moe-backend marlinPrefill
prompt tok/s by context length8K context16.1K509 ms TTFT
16K context15.9K1,022 ms TTFT
32K context15.8K2,043 ms TTFT
64K context15.8K4,082 ms TTFT
128K context14.8K8,674 ms TTFT
256K context12.9K19,919 ms TTFT
512K context9,60453,590 ms TTFT
1019K context6,245163,993 ms TTFT
1024K context6,242164,890 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 145.356 ms TTFT | 269.268 ms TTFT | 474.2107 ms TTFT | 760.6148 ms TTFT | 1,155.8227 ms TTFT | 1,649.2351 ms TTFT | 2,196.3592 ms TTFT | 2,889.41,087 ms TTFT |
| 16K | 144.171 ms TTFT | 267.4105 ms TTFT | 471.5120 ms TTFT | 760.0187 ms TTFT | 1,144.9311 ms TTFT | 1,619.5531 ms TTFT | 2,158.4953 ms TTFT | 2,817.31,789 ms TTFT |
| 32K | 142.4105 ms TTFT | 264.2143 ms TTFT | 465.2181 ms TTFT | 743.9279 ms TTFT | 1,122.9468 ms TTFT | 1,561.6853 ms TTFT | 2,061.21,610 ms TTFT | 2,651.73,230 ms TTFT |
| 64K | 138.9183 ms TTFT | 257.7240 ms TTFT | 456.4344 ms TTFT | 721.3556 ms TTFT | 1,066.5964 ms TTFT | 1,480.61,822 ms TTFT | 1,900.13,494 ms TTFT | Over capacity |
| 128K | 131.8365 ms TTFT | 244.9425 ms TTFT | 435.1708 ms TTFT | 676.41,172 ms TTFT | 981.22,080 ms TTFT | 1,308.63,954 ms TTFT | Over capacity | Over capacity |
| 256K | 122.5759 ms TTFT | 228.51,083 ms TTFT | 403.51,615 ms TTFT | 603.82,711 ms TTFT | 844.24,932 ms TTFT | Over capacity | Over capacity | Over capacity |
| 512K | 105.31,786 ms TTFT | 198.22,592 ms TTFT | 352.33,931 ms TTFT | 496.26,572 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 82.33,930 ms TTFT | 156.55,565 ms TTFT | 279.78,749 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.1K | — | 509 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.9K | — | 1,022 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.8K | — | 2,043 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 15.8K | — | 4,082 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.8K | — | 8,674 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.9K | — | 19,919 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,604.0 | — | 53,590 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 6,245.0 | — | 163,993 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 6,242.0 | — | 164,890 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 145.3 | 145.7 | 56 ms | 6.9 ms | 1.0 | 1 | 4,357 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 269.2 | 135.1 | 68 ms | 7.4 ms | 2.0 | 2 | 8,076 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 474.2 | 118.7 | 107 ms | 8.4 ms | 4.0 | 0 | 14,224 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 760.6 | 95.2 | 148 ms | 10.5 ms | 8.0 | 0 | 22,819 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,155.8 | 72.5 | 227 ms | 13.8 ms | 16.0 | 0 | 34,672 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,649.2 | 51.8 | 351 ms | 19.3 ms | 32.0 | 0 | 49,472 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,196.3 | 34.6 | 592 ms | 28.9 ms | 64.0 | 0 | 65,856 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,889.4 | 22.8 | 1,087 ms | 43.9 ms | 128.0 | 0 | 86,656 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 144.1 | 144.6 | 71 ms | 6.9 ms | 1.0 | 1 | 4,324 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 267.4 | 134.2 | 105 ms | 7.5 ms | 2.0 | 2 | 8,022 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 471.5 | 117.9 | 120 ms | 8.5 ms | 4.0 | 0 | 14,144 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 760.0 | 95.0 | 187 ms | 10.5 ms | 8.0 | 0 | 22,800 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,144.9 | 71.8 | 311 ms | 13.9 ms | 16.0 | 0 | 34,336 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,619.5 | 50.9 | 531 ms | 19.7 ms | 32.0 | 0 | 48,576 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,158.4 | 34.0 | 953 ms | 29.4 ms | 64.0 | 0 | 64,704 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,817.3 | 22.3 | 1,789 ms | 44.8 ms | 128.0 | 0 | 84,480 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 142.4 | 143.2 | 105 ms | 7.0 ms | 1.0 | 1 | 4,272 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 264.2 | 132.9 | 143 ms | 7.5 ms | 2.0 | 2 | 7,925 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 465.2 | 116.4 | 181 ms | 8.6 ms | 4.0 | 0 | 13,952 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 743.9 | 93.0 | 279 ms | 10.8 ms | 8.0 | 0 | 22,312 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,122.9 | 70.3 | 468 ms | 14.2 ms | 16.0 | 0 | 33,680 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,561.6 | 49.2 | 853 ms | 20.3 ms | 32.0 | 0 | 46,848 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,061.2 | 32.5 | 1,610 ms | 30.7 ms | 64.0 | 0 | 61,825 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,651.7 | 21.1 | 3,230 ms | 47.3 ms | 128.0 | 0 | 79,535 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 138.9 | 140.1 | 183 ms | 7.1 ms | 1.0 | 1 | 4,166 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 257.7 | 129.8 | 240 ms | 7.7 ms | 2.0 | 2 | 7,731 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 456.4 | 114.1 | 344 ms | 8.8 ms | 4.0 | 0 | 13,692 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 721.3 | 90.2 | 556 ms | 11.1 ms | 8.0 | 0 | 21,640 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,066.5 | 66.8 | 964 ms | 15.0 ms | 16.0 | 0 | 31,995 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,480.6 | 46.5 | 1,822 ms | 21.5 ms | 32.0 | 0 | 44,416 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,900.1 | 30.0 | 3,494 ms | 33.3 ms | 64.0 | 0 | 56,959 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 131.8 | 355.3 | 365 ms | 4.5 ms | 1.0 | 1 | 3,954 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 244.9 | 124.8 | 425 ms | 8.0 ms | 2.0 | 2 | 7,348 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 435.1 | 108.9 | 708 ms | 9.2 ms | 4.0 | 0 | 13,052 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 676.4 | 84.5 | 1,172 ms | 11.8 ms | 8.0 | 0 | 20,288 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 981.2 | 61.4 | 2,080 ms | 16.3 ms | 16.0 | 0 | 29,424 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,308.6 | 41.0 | 3,954 ms | 24.4 ms | 32.0 | 0 | 39,232 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 122.5 | 122.8 | 759 ms | 8.1 ms | 1.0 | 1 | 3,618 | 29.5 s |
| Sustained decode | 256K | 2 | Valid | 228.5 | 114.4 | 1,083 ms | 8.7 ms | 2.0 | 0 | 6,854 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 403.5 | 100.8 | 1,615 ms | 9.9 ms | 4.0 | 0 | 12,104 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 603.8 | 75.5 | 2,711 ms | 13.2 ms | 8.0 | 0 | 18,112 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 844.2 | 52.9 | 4,932 ms | 18.9 ms | 16.0 | 0 | 25,312 | 30.0 s |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 105.3 | 105.7 | 1,786 ms | 9.5 ms | 1.0 | 0 | 3,159 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 198.2 | 99.3 | 2,592 ms | 10.1 ms | 2.0 | 0 | 5,944 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 352.3 | 88.0 | 3,931 ms | 11.4 ms | 4.0 | 0 | 10,568 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 496.2 | 62.4 | 6,572 ms | 16.0 ms | 8.0 | 0 | 14,880 | 30.0 s |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 82.3 | 83.0 | 3,930 ms | 12.1 ms | 1.0 | 0 | 2,470 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 156.5 | 78.6 | 5,565 ms | 12.7 ms | 2.0 | 0 | 4,696 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 279.7 | 70.1 | 8,749 ms | 14.3 ms | 4.0 | 0 | 8,388 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.