Speed test matrix
MiniMax-M2.5 on H200 SXM 141GB ×4
vllm · 0.28.0 · FP86 contexts (8K–187K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 187K, 192K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.0
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- minimax-m2.5
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-minimax-m25-vllm-v0280/bin/vllm" serve MiniMaxAI/MiniMax-M2.5 --served-model-name minimax-m2.5 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2 --enable-auto-tool-choice --tool-call-parser minimax_m2 --max-model-len 196608 --gpu-memory-utilization 0.866584 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --compilation-config '{"cudagraph_mode":"PIECEWISE"}' --max-num-seqs 512 --max-num-batched-tokens 32768 --kv-cache-dtype fp8 --linear-backend cutlass --moe-backend tritonPrefill
prompt tok/s by context length8K context26.4K310 ms TTFT
16K context28.1K577 ms TTFT
32K context26.5K1,215 ms TTFT
64K context22.4K2,879 ms TTFT
128K context15.9K8,091 ms TTFT
187K context12.2K15,418 ms TTFT
192K context11.6K16,590 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 96.083 ms TTFT | 190.893 ms TTFT | 381.2125 ms TTFT | 752.9163 ms TTFT | 1,167.6245 ms TTFT | 1,794.0418 ms TTFT | 2,687.5651 ms TTFT | 3,640.41,157 ms TTFT |
| 16K | Looped | 189.8135 ms TTFT | 378.8165 ms TTFT | 716.1226 ms TTFT | 1,067.0357 ms TTFT | 1,538.4615 ms TTFT | 2,244.21,053 ms TTFT | Over capacity |
| 32K | 96.0159 ms TTFT | 190.1170 ms TTFT | 376.4269 ms TTFT | 641.9370 ms TTFT | 928.0592 ms TTFT | 1,269.01,020 ms TTFT | Over capacity | Over capacity |
| 64K | 96.0230 ms TTFT | 190.7314 ms TTFT | 353.5435 ms TTFT | Looped | 711.91,183 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 95.8445 ms TTFT | 190.2557 ms TTFT | 284.7757 ms TTFT | 365.01,249 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 187K | 96.0711 ms TTFT | 176.3870 ms TTFT | 241.41,272 ms TTFT | 322.12,115 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 26.4K | — | 310 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.1K | — | 577 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 26.5K | — | 1,215 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.4K | — | 2,879 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.9K | — | 8,091 ms | — | — | 1 | — | — |
| Prefill | 187K | 1 | Valid | 12.2K | — | 15,418 ms | — | — | 1 | — | — |
| Prefill | 192K | 1 | Valid | 11.6K | — | 16,590 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 96.0 | 96.0 | 83 ms | 10.4 ms | 1.0 | 0 | 2,882 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 190.8 | 95.5 | 93 ms | 10.5 ms | 2.0 | 0 | 5,724 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 381.2 | 95.3 | 125 ms | 10.5 ms | 4.0 | 0 | 11,432 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 752.9 | 94.2 | 163 ms | 10.6 ms | 8.0 | 0 | 22,584 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,167.6 | 73.5 | 245 ms | 13.6 ms | 16.0 | 0 | 35,072 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,794.0 | 56.6 | 418 ms | 17.7 ms | 32.0 | 0 | 53,822 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,687.5 | 42.9 | 651 ms | 23.3 ms | 64.0 | 0 | 80,747 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,640.4 | 28.9 | 1,157 ms | 34.6 ms | 128.0 | 0 | 109,209 | 30.0 s |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 189.8 | 95.0 | 135 ms | 10.5 ms | 2.0 | 0 | 5,694 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 378.8 | 94.7 | 165 ms | 10.6 ms | 4.0 | 0 | 11,364 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 716.1 | 89.8 | 226 ms | 11.1 ms | 8.0 | 0 | 21,481 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,067.0 | 67.2 | 357 ms | 14.9 ms | 16.0 | 0 | 32,001 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,538.4 | 50.2 | 615 ms | 19.9 ms | 32.0 | 0 | 46,154 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,244.2 | 35.4 | 1,053 ms | 28.3 ms | 64.0 | 0 | 67,328 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 96.0 | 96.1 | 159 ms | 10.4 ms | 1.0 | 0 | 2,881 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 190.1 | 95.1 | 170 ms | 10.5 ms | 2.0 | 0 | 5,702 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 376.4 | 94.2 | 269 ms | 10.6 ms | 4.0 | 0 | 11,288 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 641.9 | 80.4 | 370 ms | 12.4 ms | 8.0 | 0 | 19,257 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 928.0 | 58.2 | 592 ms | 17.2 ms | 16.0 | 0 | 27,841 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,269.0 | 39.9 | 1,020 ms | 25.1 ms | 32.0 | 0 | 38,048 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 96.0 | 96.1 | 230 ms | 10.4 ms | 1.0 | 0 | 2,878 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 190.7 | 95.5 | 314 ms | 10.5 ms | 2.0 | 0 | 5,722 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 353.5 | 88.6 | 435 ms | 11.3 ms | 4.0 | 0 | 10,604 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 711.9 | 44.8 | 1,183 ms | 22.3 ms | 16.0 | 0 | 21,376 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 95.8 | 96.0 | 445 ms | 10.4 ms | 1.0 | 0 | 2,874 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 190.2 | 95.3 | 557 ms | 10.5 ms | 2.0 | 0 | 5,708 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 284.7 | 71.4 | 757 ms | 14.0 ms | 4.0 | 0 | 8,548 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 365.0 | 49.8 | 1,249 ms | 20.1 ms | 8.0 | 0 | 10,949 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 1 | Valid | 96.0 | 96.2 | 711 ms | 10.4 ms | 1.0 | 0 | 2,880 | 30.0 s |
| Sustained decode | 187K | 2 | Valid | 176.3 | 88.5 | 870 ms | 11.3 ms | 2.0 | 0 | 5,288 | 30.0 s |
| Sustained decode | 187K | 4 | Valid | 241.4 | 60.7 | 1,272 ms | 16.5 ms | 4.0 | 0 | 7,241 | 30.0 s |
| Sustained decode | 187K | 8 | Valid | 322.1 | 40.6 | 2,115 ms | 24.6 ms | 8.0 | 0 | 9,657 | 30.0 s |
| Sustained decode | 187K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.