Speed test matrix
MiniMax-M2.7 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+ge0317b28c · FP86 contexts (8K–195K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 195K, 200K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+ge0317b28c
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e0317b28ce3b
Run
- Served model
- minimax-m2.7
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-minimaxai-vllm-e0317b28ce3b/bin/vllm" serve MiniMaxAI/MiniMax-M2.7 --served-model-name minimax-m2.7 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --max-model-len 204800 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASH_ATTN --linear-backend triton --moe-backend triton
Prefill
prompt tok/s by context length8K context15.3K536 ms TTFT
16K context14.2K1,138 ms TTFT
32K context12.5K2,585 ms TTFT
64K context9,8616,525 ms TTFT
128K context6,84318,783 ms TTFT
195K context5,17637,806 ms TTFT
200K context5,05939,675 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 86.049 ms TTFT | 155.271 ms TTFT | 261.1113 ms TTFT | 412.4113 ms TTFT | 629.4192 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | Looped | 149.2109 ms TTFT | 248.8153 ms TTFT | 383.1228 ms TTFT | 572.1318 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | Looped | 138.2153 ms TTFT | 231.8218 ms TTFT | 338.2333 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 68.7220 ms TTFT | 121.8288 ms TTFT | 205.3428 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 55.2430 ms TTFT | 97.1544 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 195K | 46.4710 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.3K | — | 536 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.2K | — | 1,138 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.5K | — | 2,585 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,861.0 | — | 6,525 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,843.0 | — | 18,783 ms | — | — | 1 | — | — |
| Prefill | 195K | 1 | Valid | 5,176.0 | — | 37,806 ms | — | — | 1 | — | — |
| Prefill | 200K | 1 | Valid | 5,059.0 | — | 39,675 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 86.0 | 86.1 | 49 ms | 11.6 ms | 1.0 | 0 | 2,580 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 155.2 | 77.7 | 71 ms | 12.9 ms | 2.0 | 0 | 4,654 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 261.1 | 65.5 | 113 ms | 15.3 ms | 4.0 | 0 | 7,832 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 412.4 | 51.4 | 113 ms | 19.4 ms | 8.0 | 0 | 12,368 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 629.4 | 39.5 | 192 ms | 25.3 ms | 16.0 | 0 | 18,880 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 149.2 | 74.6 | 109 ms | 13.4 ms | 2.0 | 0 | 4,476 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 248.8 | 62.4 | 153 ms | 16.0 ms | 4.0 | 0 | 7,464 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 383.1 | 47.8 | 228 ms | 20.9 ms | 8.0 | 0 | 11,488 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 572.1 | 35.9 | 318 ms | 27.8 ms | 16.0 | 0 | 17,152 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Valid | 138.2 | 69.1 | 153 ms | 14.5 ms | 2.0 | 0 | 4,146 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 231.8 | 58.1 | 218 ms | 17.2 ms | 4.0 | 0 | 6,952 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 338.2 | 42.3 | 333 ms | 23.7 ms | 8.0 | 0 | 10,144 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 68.7 | 68.8 | 220 ms | 14.5 ms | 1.0 | 0 | 2,061 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 121.8 | 60.9 | 288 ms | 16.4 ms | 2.0 | 0 | 3,654 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 205.3 | 51.3 | 428 ms | 19.5 ms | 4.0 | 0 | 6,156 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 55.2 | 55.2 | 430 ms | 18.1 ms | 1.0 | 0 | 1,654 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 97.1 | 48.5 | 544 ms | 20.6 ms | 2.0 | 0 | 2,912 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 1 | Valid | 46.4 | 46.4 | 710 ms | 21.5 ms | 1.0 | 0 | 1,391 | 30.0 s |
| Sustained decode | 195K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 195K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.