Speed test matrix
MiniMax-M2 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+ge0317b28c · FP86 contexts (8K–187K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 187K, 192K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+ge0317b28c
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e0317b28ce3b
Run
- Served model
- minimax-m2
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-minimaxai-vllm-e0317b28ce3b/bin/vllm" serve MiniMaxAI/MiniMax-M2 --served-model-name minimax-m2 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --max-model-len 196608 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend triton --moe-backend triton
Prefill
prompt tok/s by context length8K context15.2K538 ms TTFT
16K context11.4K1,426 ms TTFT
32K context12.5K2,583 ms TTFT
64K context9,8706,520 ms TTFT
128K context6,84718,771 ms TTFT
187K context5,34535,119 ms TTFT
192K context5,21536,952 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 85.949 ms TTFT | 155.172 ms TTFT | Looped | 387.4117 ms TTFT | 630.5196 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | Looped | 148.192 ms TTFT | 249.4121 ms TTFT | 370.5189 ms TTFT | 562.7279 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 77.4117 ms TTFT | 138.5163 ms TTFT | Looped | 331.4290 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 68.6220 ms TTFT | Looped | 202.7407 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 55.0417 ms TTFT | 97.1534 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 187K | Looped | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.2K | — | 538 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11.4K | — | 1,426 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.5K | — | 2,583 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,870.0 | — | 6,520 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,847.0 | — | 18,771 ms | — | — | 1 | — | — |
| Prefill | 187K | 1 | Valid | 5,345.0 | — | 35,119 ms | — | — | 1 | — | — |
| Prefill | 192K | 1 | Valid | 5,215.0 | — | 36,952 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 85.9 | 86.0 | 49 ms | 11.6 ms | 1.0 | 0 | 2,576 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 155.1 | 77.6 | 72 ms | 12.9 ms | 2.0 | 0 | 4,652 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Valid | 387.4 | 48.6 | 117 ms | 20.6 ms | 8.0 | 0 | 11,616 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 630.5 | 39.6 | 196 ms | 25.2 ms | 16.0 | 0 | 18,912 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 148.1 | 74.1 | 92 ms | 13.5 ms | 2.0 | 0 | 4,442 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 249.4 | 62.5 | 121 ms | 16.0 ms | 4.0 | 0 | 7,480 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 370.5 | 46.5 | 189 ms | 21.5 ms | 8.0 | 0 | 11,112 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 562.7 | 35.4 | 279 ms | 28.3 ms | 16.0 | 0 | 16,880 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 77.4 | 77.5 | 117 ms | 12.9 ms | 1.0 | 0 | 2,323 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 138.5 | 69.2 | 163 ms | 14.4 ms | 2.0 | 0 | 4,154 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 331.4 | 41.5 | 290 ms | 24.1 ms | 8.0 | 0 | 9,936 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 68.6 | 68.7 | 220 ms | 14.6 ms | 1.0 | 0 | 2,059 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Valid | 202.7 | 50.6 | 407 ms | 19.8 ms | 4.0 | 0 | 6,080 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 55.0 | 55.1 | 417 ms | 18.2 ms | 1.0 | 0 | 1,651 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 97.1 | 48.5 | 534 ms | 20.6 ms | 2.0 | 0 | 2,914 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 187K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.