Speed test matrix
MiniMax-M2.7 on H200 SXM 141GB ×4
vllm · 0.19.1rc1.dev203+g0f3ce4c74 · FP86 contexts (8K–195K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 195K, 200K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.19.1rc1.dev203+g0f3ce4c74
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
0f3ce4c74b18
Run
- Served model
- minimax-m2.7
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-minimax-m2-vllm-0f3ce4c74/bin/vllm" serve MiniMaxAI/MiniMax-M2.7 --served-model-name minimax-m2.7 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2 --enable-auto-tool-choice --tool-call-parser minimax_m2 --max-model-len 204800 --gpu-memory-utilization 0.858642 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --compilation-config '{"mode":3,"pass_config":{"fuse_minimax_qk_norm":true}}'Prefill
prompt tok/s by context length8K context27.6K297 ms TTFT
16K context31.9K509 ms TTFT
32K context28.4K1,136 ms TTFT
64K context22.1K2,919 ms TTFT
128K context15K8,567 ms TTFT
195K context11.1K17,592 ms TTFT
200K context10.9K18,396 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 83.674 ms TTFT | 154.1111 ms TTFT | 288.9162 ms TTFT | 554.2234 ms TTFT | 931.1389 ms TTFT | 1,658.2471 ms TTFT | 2,914.9857 ms TTFT | Over capacity |
| 16K | 62.3104 ms TTFT | 117.4139 ms TTFT | 224.3205 ms TTFT | 421.4297 ms TTFT | 763.3396 ms TTFT | 1,352.11,164 ms TTFT | Over capacity | Over capacity |
| 32K | 40.9148 ms TTFT | 78.8224 ms TTFT | 153.4353 ms TTFT | 292.1497 ms TTFT | 546.2856 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 24.2283 ms TTFT | 47.3391 ms TTFT | 92.5702 ms TTFT | 181.31,410 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 13.2516 ms TTFT | 26.2915 ms TTFT | 51.81,468 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 195K | 9.0812 ms TTFT | 17.81,504 ms TTFT | 35.42,659 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.6K | — | 297 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.9K | — | 509 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 28.4K | — | 1,136 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.1K | — | 2,919 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15K | — | 8,567 ms | — | — | 1 | — | — |
| Prefill | 195K | 1 | Valid | 11.1K | — | 17,592 ms | — | — | 1 | — | — |
| Prefill | 200K | 1 | Valid | 10.9K | — | 18,396 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 83.6 | 84.2 | 74 ms | 11.9 ms | 1.0 | 0 | 2,508 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 154.1 | 77.7 | 111 ms | 12.9 ms | 2.0 | 0 | 4,624 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 288.9 | 72.9 | 162 ms | 13.7 ms | 4.0 | 0 | 8,668 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 554.2 | 69.6 | 234 ms | 14.4 ms | 8.0 | 0 | 16,624 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 931.1 | 58.9 | 389 ms | 17.0 ms | 16.0 | 0 | 27,936 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,658.2 | 52.2 | 471 ms | 19.1 ms | 32.0 | 0 | 49,728 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,914.9 | 45.9 | 857 ms | 21.8 ms | 64.0 | 0 | 87,424 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 62.3 | 62.6 | 104 ms | 16.0 ms | 1.0 | 0 | 1,869 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 117.4 | 59.0 | 139 ms | 16.9 ms | 2.0 | 0 | 3,522 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 224.3 | 56.3 | 205 ms | 17.8 ms | 4.0 | 0 | 6,728 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 421.4 | 53.0 | 297 ms | 18.9 ms | 8.0 | 0 | 12,656 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 763.3 | 48.0 | 396 ms | 20.8 ms | 16.0 | 0 | 22,896 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,352.1 | 42.6 | 1,164 ms | 23.5 ms | 32.0 | 0 | 40,544 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 40.9 | 41.0 | 148 ms | 24.4 ms | 1.0 | 0 | 1,228 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 78.8 | 39.5 | 224 ms | 25.3 ms | 2.0 | 0 | 2,364 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 153.4 | 38.4 | 353 ms | 26.0 ms | 4.0 | 0 | 4,600 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 292.1 | 36.7 | 497 ms | 27.3 ms | 8.0 | 0 | 8,760 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 546.2 | 34.3 | 856 ms | 29.2 ms | 16.0 | 0 | 16,384 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 24.2 | 24.2 | 283 ms | 41.3 ms | 1.0 | 0 | 725 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 47.3 | 23.6 | 391 ms | 42.3 ms | 2.0 | 0 | 1,418 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 92.5 | 23.2 | 702 ms | 43.2 ms | 4.0 | 0 | 2,772 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 181.3 | 22.7 | 1,410 ms | 44.0 ms | 8.0 | 0 | 5,432 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 13.2 | 13.3 | 516 ms | 75.4 ms | 1.0 | 0 | 397 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 26.2 | 13.1 | 915 ms | 76.4 ms | 2.0 | 0 | 784 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 51.8 | 12.9 | 1,468 ms | 77.3 ms | 4.0 | 0 | 1,552 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 1 | Valid | 9.0 | 9.0 | 812 ms | 111.1 ms | 1.0 | 0 | 269 | 29.9 s |
| Sustained decode | 195K | 2 | Valid | 17.8 | 8.9 | 1,504 ms | 112.1 ms | 2.0 | 0 | 533 | 29.9 s |
| Sustained decode | 195K | 4 | Valid | 35.4 | 8.9 | 2,659 ms | 112.5 ms | 4.0 | 0 | 1,060 | 30.0 s |
| Sustained decode | 195K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.