Speed test matrix
MiniMax-M2.5 on H200 SXM 141GB ×4
vllm · 0.29.0 · FP86 contexts (8K–187K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 187K, 192K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- z-lab/MiniMax-M2.5-DFlash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve MiniMaxAI/MiniMax-M2.5 --served-model-name z-lab/MiniMax-M2.5-DFlash --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2 --enable-auto-tool-choice --tool-call-parser minimax_m2 --max-model-len 196608 --gpu-memory-utilization 0.81 --host 0.0.0.0 --port 8000 --api-key <redacted> --kernel-config.enable_flashinfer_autotune=False --speculative-config '{"method":"dflash","model":"z-lab/MiniMax-M2.5-DFlash","num_speculative_tokens":8}'Prefill
prompt tok/s by context length8K context32.9K249 ms TTFT
16K context32.7K495 ms TTFT
32K context28.1K1,148 ms TTFT
64K context21.7K2,961 ms TTFT
128K context14.7K8,742 ms TTFT
187K context11.3K16,549 ms TTFT
192K context11K17,456 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 261.764 ms TTFT | Looped | 759.069 ms TTFT | 1,057.4102 ms TTFT | Looped | Looped | Looped | 3,962.41,498 ms TTFT |
| 16K | 244.796 ms TTFT | 474.7105 ms TTFT | 648.2125 ms TTFT | 924.3221 ms TTFT | 1,559.9453 ms TTFT | Looped | 1,963.51,214 ms TTFT | Looped |
| 32K | Looped | 432.4702 ms TTFT | Looped | 909.7437 ms TTFT | 1,448.1719 ms TTFT | Looped | Looped | 3,535.34,604 ms TTFT |
| 64K | 218.61,690 ms TTFT | 435.5339 ms TTFT | 584.7401 ms TTFT | 805.5858 ms TTFT | Looped | 1,706.92,795 ms TTFT | Looped | 2,455.29,546 ms TTFT |
| 128K | Looped | 320.54,863 ms TTFT | 357.51,354 ms TTFT | 700.31,829 ms TTFT | 925.63,136 ms TTFT | 1,137.55,513 ms TTFT | 1,398.010,333 ms TTFT | Over capacity |
| 187K | 164.08,866 ms TTFT | Looped | 392.61,899 ms TTFT | 557.82,957 ms TTFT | 749.54,995 ms TTFT | 890.29,377 ms TTFT | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 32.9K | — | 249 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.7K | — | 495 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 28.1K | — | 1,148 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.7K | — | 2,961 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.7K | — | 8,742 ms | — | — | 1 | — | — |
| Prefill | 187K | 1 | Valid | 11.3K | — | 16,549 ms | — | — | 1 | — | — |
| Prefill | 192K | 1 | Valid | 11K | — | 17,456 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 261.7 | 268.9 | 64 ms | 3.7 ms | 1.0 | 2 | 7,852 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Valid | 759.0 | 171.8 | 69 ms | 5.8 ms | 4.0 | 4 | 22,768 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,057.4 | 126.2 | 102 ms | 7.9 ms | 8.0 | 6 | 31,701 | 30.0 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Valid | 3,962.4 | 31.3 | 1,498 ms | 31.9 ms | 128.0 | 0 | 118,712 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 244.7 | 271.3 | 96 ms | 3.7 ms | 1.0 | 2 | 7,339 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 474.7 | 236.3 | 105 ms | 4.2 ms | 2.0 | 3 | 14,240 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 648.2 | 160.4 | 125 ms | 6.2 ms | 3.9 | 4 | 19,438 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 924.3 | 116.7 | 221 ms | 8.6 ms | 8.0 | 2 | 27,716 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,559.9 | 97.0 | 453 ms | 10.3 ms | 16.0 | 1 | 46,798 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Valid | 1,963.5 | 31.2 | 1,214 ms | 32.1 ms | 64.0 | 0 | 58,855 | 30.0 s |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Valid | 432.4 | 216.6 | 702 ms | 4.6 ms | 2.0 | 2 | 12,971 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 909.7 | 112.4 | 437 ms | 8.9 ms | 8.0 | 1 | 27,290 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,448.1 | 87.0 | 719 ms | 11.5 ms | 16.0 | 1 | 43,410 | 30.0 s |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Valid | 3,535.3 | 27.2 | 4,604 ms | 36.8 ms | 128.0 | 0 | 106,015 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 218.6 | 224.9 | 1,690 ms | 4.4 ms | 1.0 | 1 | 6,556 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 435.5 | 197.1 | 339 ms | 5.1 ms | 2.0 | 3 | 13,060 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 584.7 | 152.2 | 401 ms | 6.6 ms | 3.9 | 4 | 17,540 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 805.5 | 97.2 | 858 ms | 10.3 ms | 8.0 | 1 | 24,154 | 30.0 s |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 1,706.9 | 50.5 | 2,795 ms | 19.8 ms | 32.0 | 0 | 51,199 | 30.0 s |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Valid | 2,455.2 | 17.0 | 9,546 ms | 58.9 ms | 128.0 | 0 | 73,424 | 29.9 s |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Valid | 320.5 | 159.9 | 4,863 ms | 6.3 ms | 2.0 | 2 | 9,615 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 357.5 | 117.1 | 1,354 ms | 8.5 ms | 4.0 | 0 | 10,722 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 700.3 | 80.1 | 1,829 ms | 12.5 ms | 8.0 | 1 | 20,992 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 925.6 | 52.9 | 3,136 ms | 18.9 ms | 16.0 | 0 | 27,769 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,137.5 | 33.2 | 5,513 ms | 30.1 ms | 32.0 | 0 | 34,103 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 1,398.0 | 19.0 | 10,333 ms | 52.8 ms | 64.0 | 0 | 41,823 | 29.9 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 1 | Valid | 164.0 | 174.6 | 8,866 ms | 5.7 ms | 1.0 | 1 | 4,920 | 30.0 s |
| Sustained decode | 187K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 187K | 4 | Valid | 392.6 | 93.3 | 1,899 ms | 10.7 ms | 4.0 | 0 | 11,768 | 30.0 s |
| Sustained decode | 187K | 8 | Valid | 557.8 | 67.4 | 2,957 ms | 14.9 ms | 8.0 | 0 | 16,717 | 30.0 s |
| Sustained decode | 187K | 16 | Valid | 749.5 | 42.5 | 4,995 ms | 23.5 ms | 16.0 | 0 | 22,471 | 30.0 s |
| Sustained decode | 187K | 32 | Valid | 890.2 | 25.1 | 9,377 ms | 39.9 ms | 32.0 | 0 | 26,701 | 30.0 s |
| Sustained decode | 187K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.