Speed test matrix
MiniMax-M2.5 on H200 SXM 141GB ×4
sglang · 0.5.10.post1 · FP86 contexts (8K–187K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 187K, 192K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.10.post1
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
Run
- Served model
- minimax-m2.5
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-minimax-m2-sglang-v0510-post1/bin/sglang" serve --model-path MiniMaxAI/MiniMax-M2.5 --served-model-name minimax-m2.5 --trust-remote-code --tp 4 --reasoning-parser minimax --tool-call-parser minimax-m2 --context-length 196608 --mem-fraction-static 0.839155 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16
Prefill
prompt tok/s by context length8K context33.5K245 ms TTFT
16K context33.7K481 ms TTFT
32K context28.8K1,120 ms TTFT
64K context22.8K2,823 ms TTFT
128K context15.3K8,382 ms TTFT
187K context11.7K16,050 ms TTFT
192K context11.5K16,825 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 142.549 ms TTFT | 237.465 ms TTFT | 431.196 ms TTFT | 683.3141 ms TTFT | 1,111.9250 ms TTFT | 1,720.4442 ms TTFT | 2,668.4868 ms TTFT | Over capacity |
| 16K | 138.773 ms TTFT | 231.992 ms TTFT | 414.4135 ms TTFT | Looped | Looped | 1,542.6800 ms TTFT | Over capacity | Over capacity |
| 32K | 133.1102 ms TTFT | 220.6146 ms TTFT | 388.2243 ms TTFT | 594.5406 ms TTFT | 902.9788 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 124.3176 ms TTFT | 201.5250 ms TTFT | 343.6429 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 111.5320 ms TTFT | 172.5502 ms TTFT | 280.7954 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 187K | 101.3474 ms TTFT | 152.8833 ms TTFT | 241.61,444 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.5K | — | 245 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33.7K | — | 481 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 28.8K | — | 1,120 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.8K | — | 2,823 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 15.3K | — | 8,382 ms | — | — | 1 | — | — |
| Prefill | 187K | 1 | Valid | 11.7K | — | 16,050 ms | — | — | 1 | — | — |
| Prefill | 192K | 1 | Valid | 11.5K | — | 16,825 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 142.5 | 143.1 | 49 ms | 7.0 ms | 1.0 | 1 | 4,275 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 237.4 | 120.5 | 65 ms | 8.3 ms | 2.0 | 2 | 7,122 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 431.1 | 107.9 | 96 ms | 9.3 ms | 4.0 | 0 | 12,932 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 683.3 | 86.0 | 141 ms | 11.6 ms | 8.0 | 0 | 20,496 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,111.9 | 70.2 | 250 ms | 14.2 ms | 16.0 | 0 | 33,360 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,720.4 | 54.5 | 442 ms | 18.3 ms | 32.0 | 0 | 51,616 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,668.4 | 42.5 | 868 ms | 23.5 ms | 64.0 | 0 | 80,064 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 138.7 | 139.5 | 73 ms | 7.2 ms | 1.0 | 1 | 4,161 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 231.9 | 116.1 | 92 ms | 8.6 ms | 2.0 | 0 | 6,956 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 414.4 | 103.8 | 135 ms | 9.6 ms | 4.0 | 0 | 12,432 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 1,542.6 | 48.8 | 800 ms | 20.5 ms | 32.0 | 0 | 46,271 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 133.1 | 134.0 | 102 ms | 7.5 ms | 1.0 | 1 | 3,992 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 220.6 | 110.4 | 146 ms | 9.1 ms | 2.0 | 0 | 6,616 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 388.2 | 97.2 | 243 ms | 10.3 ms | 4.0 | 0 | 11,644 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 594.5 | 74.9 | 406 ms | 13.4 ms | 8.0 | 0 | 17,831 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 902.9 | 57.2 | 788 ms | 17.5 ms | 16.0 | 0 | 27,087 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 124.3 | 126.3 | 176 ms | 7.9 ms | 1.0 | 1 | 3,727 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 201.5 | 100.9 | 250 ms | 9.9 ms | 2.0 | 0 | 6,044 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 343.6 | 86.3 | 429 ms | 11.6 ms | 4.0 | 0 | 10,304 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 111.5 | 111.5 | 320 ms | 9.0 ms | 1.0 | 0 | 3,344 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 172.5 | 86.6 | 502 ms | 11.5 ms | 2.0 | 0 | 5,176 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 280.7 | 70.9 | 954 ms | 14.1 ms | 4.0 | 0 | 8,420 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 1 | Valid | 101.3 | 101.4 | 474 ms | 9.9 ms | 1.0 | 0 | 3,039 | 30.0 s |
| Sustained decode | 187K | 2 | Valid | 152.8 | 76.9 | 833 ms | 13.0 ms | 2.0 | 0 | 4,582 | 30.0 s |
| Sustained decode | 187K | 4 | Valid | 241.6 | 61.3 | 1,444 ms | 16.3 ms | 4.0 | 0 | 7,244 | 30.0 s |
| Sustained decode | 187K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.