Speed test matrix
MiniMax-M2 on H200 SXM 141GB ×4
vllm · 0.19.1rc1.dev203+g0f3ce4c74 · FP86 contexts (8K–187K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 187K, 192K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.19.1rc1.dev203+g0f3ce4c74
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
0f3ce4c74b18
Run
- Served model
- minimax-m2
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-minimax-m2-vllm-0f3ce4c74/bin/vllm" serve MiniMaxAI/MiniMax-M2 --served-model-name minimax-m2 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2 --enable-auto-tool-choice --tool-call-parser minimax_m2 --max-model-len 196608 --gpu-memory-utilization 0.858628 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --compilation-config '{"mode":3,"pass_config":{"fuse_minimax_qk_norm":true}}'Prefill
prompt tok/s by context length8K context27.6K297 ms TTFT
16K context31.4K516 ms TTFT
32K context27.9K1,154 ms TTFT
64K context22K2,923 ms TTFT
128K context15K8,571 ms TTFT
187K context11.5K16,338 ms TTFT
192K context11.3K17,067 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 83.772 ms TTFT | 154.5128 ms TTFT | 294.7157 ms TTFT | 547.6203 ms TTFT | 973.1466 ms TTFT | 1,688.1604 ms TTFT | 2,907.7828 ms TTFT | Over capacity |
| 16K | 62.3108 ms TTFT | 117.7151 ms TTFT | Looped | 426.6368 ms TTFT | Looped | 1,363.9846 ms TTFT | Over capacity | Over capacity |
| 32K | 41.0162 ms TTFT | 78.8242 ms TTFT | 152.1383 ms TTFT | 294.8507 ms TTFT | 552.01,230 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 24.2307 ms TTFT | 47.2474 ms TTFT | 92.6782 ms TTFT | 181.01,215 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 13.3524 ms TTFT | 26.2717 ms TTFT | 51.81,474 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 187K | 9.4840 ms TTFT | 18.51,454 ms TTFT | 36.82,086 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.6K | — | 297 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.4K | — | 516 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 27.9K | — | 1,154 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22K | — | 2,923 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15K | — | 8,571 ms | — | — | 1 | — | — |
| Prefill | 187K | 1 | Valid | 11.5K | — | 16,338 ms | — | — | 1 | — | — |
| Prefill | 192K | 1 | Valid | 11.3K | — | 17,067 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 83.7 | 84.3 | 72 ms | 11.9 ms | 1.0 | 0 | 2,509 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 154.5 | 77.9 | 128 ms | 12.8 ms | 2.0 | 0 | 4,634 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 294.7 | 74.1 | 157 ms | 13.5 ms | 4.0 | 0 | 8,840 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 547.6 | 59.6 | 203 ms | 16.8 ms | 8.0 | 0 | 16,424 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 973.1 | 61.3 | 466 ms | 16.3 ms | 16.0 | 0 | 29,184 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,688.1 | 53.3 | 604 ms | 18.8 ms | 32.0 | 0 | 50,624 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,907.7 | 45.9 | 828 ms | 21.8 ms | 64.0 | 0 | 87,296 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 62.3 | 62.6 | 108 ms | 16.0 ms | 1.0 | 0 | 1,869 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 117.7 | 59.1 | 151 ms | 16.9 ms | 2.0 | 0 | 3,532 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 426.6 | 53.6 | 368 ms | 18.7 ms | 8.0 | 0 | 12,800 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 1,363.9 | 42.9 | 846 ms | 23.3 ms | 32.0 | 0 | 40,896 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 41.0 | 41.1 | 162 ms | 24.3 ms | 1.0 | 0 | 1,230 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 78.8 | 39.5 | 242 ms | 25.3 ms | 2.0 | 0 | 2,364 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 152.1 | 38.1 | 383 ms | 26.2 ms | 4.0 | 0 | 4,560 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 294.8 | 36.9 | 507 ms | 27.1 ms | 8.0 | 0 | 8,840 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 552.0 | 34.6 | 1,230 ms | 28.9 ms | 16.0 | 0 | 16,560 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 24.2 | 24.2 | 307 ms | 41.3 ms | 1.0 | 0 | 725 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 47.2 | 23.7 | 474 ms | 42.3 ms | 2.0 | 0 | 1,416 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 92.6 | 23.2 | 782 ms | 43.1 ms | 4.0 | 0 | 2,776 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 181.0 | 22.6 | 1,215 ms | 44.2 ms | 8.0 | 0 | 5,424 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 13.3 | 13.3 | 524 ms | 75.3 ms | 1.0 | 0 | 397 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 26.2 | 13.1 | 717 ms | 76.3 ms | 2.0 | 0 | 784 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 51.8 | 13.0 | 1,474 ms | 77.1 ms | 4.0 | 0 | 1,552 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 1 | Valid | 9.4 | 9.4 | 840 ms | 106.5 ms | 1.0 | 0 | 281 | 29.9 s |
| Sustained decode | 187K | 2 | Valid | 18.5 | 9.3 | 1,454 ms | 107.8 ms | 2.0 | 0 | 556 | 30.0 s |
| Sustained decode | 187K | 4 | Valid | 36.8 | 9.2 | 2,086 ms | 108.6 ms | 4.0 | 0 | 1,100 | 29.9 s |
| Sustained decode | 187K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.