Speed test matrix
MiniMax-M2.1 on H200 SXM 141GB ×4
vllm · 0.19.1rc1.dev203+g0f3ce4c74 · FP86 contexts (8K–187K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 187K, 192K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.19.1rc1.dev203+g0f3ce4c74
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
0f3ce4c74b18
Run
- Served model
- minimax-m2.1
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-minimax-m2-vllm-0f3ce4c74/bin/vllm" serve MiniMaxAI/MiniMax-M2.1 --served-model-name minimax-m2.1 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2 --enable-auto-tool-choice --tool-call-parser minimax_m2 --max-model-len 196608 --gpu-memory-utilization 0.858628 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --compilation-config '{"mode":3,"pass_config":{"fuse_minimax_qk_norm":true}}'Prefill
prompt tok/s by context length8K context27.2K301 ms TTFT
16K context31.5K514 ms TTFT
32K context28.1K1,149 ms TTFT
64K context22K2,926 ms TTFT
128K context15K8,577 ms TTFT
187K context11.5K16,334 ms TTFT
192K context11.2K17,240 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 83.779 ms TTFT | 154.4149 ms TTFT | 289.9161 ms TTFT | 536.8245 ms TTFT | 942.6437 ms TTFT | 1,613.3486 ms TTFT | 2,829.6862 ms TTFT | Over capacity |
| 16K | 62.4110 ms TTFT | 118.0148 ms TTFT | 223.4228 ms TTFT | 421.2374 ms TTFT | 755.0489 ms TTFT | 1,349.3892 ms TTFT | Over capacity | Over capacity |
| 32K | 41.0174 ms TTFT | 78.8265 ms TTFT | 152.2433 ms TTFT | 292.8664 ms TTFT | 541.01,283 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 24.2256 ms TTFT | 47.3432 ms TTFT | 92.7749 ms TTFT | 180.7942 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 13.3537 ms TTFT | 26.2875 ms TTFT | 51.81,558 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 187K | 9.4785 ms TTFT | 18.51,371 ms TTFT | 36.82,376 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.2K | — | 301 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.5K | — | 514 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 28.1K | — | 1,149 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22K | — | 2,926 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15K | — | 8,577 ms | — | — | 1 | — | — |
| Prefill | 187K | 1 | Valid | 11.5K | — | 16,334 ms | — | — | 1 | — | — |
| Prefill | 192K | 1 | Valid | 11.2K | — | 17,240 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 83.7 | 84.3 | 79 ms | 11.9 ms | 1.0 | 0 | 2,510 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 154.4 | 77.9 | 149 ms | 12.8 ms | 2.0 | 0 | 4,630 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 289.9 | 73.1 | 161 ms | 13.7 ms | 4.0 | 0 | 8,700 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 536.8 | 67.6 | 245 ms | 14.8 ms | 8.0 | 0 | 16,104 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 942.6 | 59.6 | 437 ms | 16.8 ms | 16.0 | 0 | 28,272 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,613.3 | 50.9 | 486 ms | 19.6 ms | 32.0 | 0 | 48,416 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,829.6 | 44.7 | 862 ms | 22.4 ms | 64.0 | 0 | 84,864 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 62.4 | 62.6 | 110 ms | 16.0 ms | 1.0 | 0 | 1,871 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 118.0 | 59.2 | 148 ms | 16.9 ms | 2.0 | 0 | 3,540 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 223.4 | 56.1 | 228 ms | 17.8 ms | 4.0 | 0 | 6,704 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 421.2 | 53.0 | 374 ms | 18.9 ms | 8.0 | 0 | 12,640 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 755.0 | 47.6 | 489 ms | 21.0 ms | 16.0 | 0 | 22,656 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,349.3 | 42.4 | 892 ms | 23.6 ms | 32.0 | 0 | 40,480 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 41.0 | 41.1 | 174 ms | 24.3 ms | 1.0 | 0 | 1,229 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 78.8 | 39.5 | 265 ms | 25.3 ms | 2.0 | 0 | 2,362 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 152.2 | 38.2 | 433 ms | 26.2 ms | 4.0 | 0 | 4,568 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 292.8 | 36.7 | 664 ms | 27.2 ms | 8.0 | 0 | 8,776 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 541.0 | 34.1 | 1,283 ms | 29.4 ms | 16.0 | 0 | 16,224 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 24.2 | 24.2 | 256 ms | 41.3 ms | 1.0 | 0 | 726 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 47.3 | 23.7 | 432 ms | 42.3 ms | 2.0 | 0 | 1,416 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 92.7 | 23.2 | 749 ms | 43.1 ms | 4.0 | 0 | 2,780 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 180.7 | 22.7 | 942 ms | 44.1 ms | 8.0 | 0 | 5,424 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 13.3 | 13.3 | 537 ms | 75.3 ms | 1.0 | 0 | 398 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 26.2 | 13.1 | 875 ms | 76.3 ms | 2.0 | 0 | 786 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 51.8 | 13.0 | 1,558 ms | 77.0 ms | 4.0 | 0 | 1,552 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 1 | Valid | 9.4 | 9.4 | 785 ms | 106.8 ms | 1.0 | 0 | 280 | 29.9 s |
| Sustained decode | 187K | 2 | Valid | 18.5 | 9.3 | 1,371 ms | 107.8 ms | 2.0 | 0 | 554 | 29.9 s |
| Sustained decode | 187K | 4 | Valid | 36.8 | 9.2 | 2,376 ms | 108.6 ms | 4.0 | 0 | 1,100 | 29.9 s |
| Sustained decode | 187K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.