Speed test matrix
MiniMax-M2.5 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.1.dev1+ge0317b28c · FP86 contexts (8K–187K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 187K, 192K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+ge0317b28c
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
e0317b28ce3b
Run
- Served model
- minimax-m2.5
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.9
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-minimaxai-vllm-e0317b28ce3b/bin/vllm" serve MiniMaxAI/MiniMax-M2.5 --served-model-name minimax-m2.5 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser minimax_m2_append_think --enable-auto-tool-choice --tool-call-parser minimax_m2 --max-model-len 196608 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --max-num-seqs 512 --max-num-batched-tokens 32768 --compilation-config '{"cudagraph_mode":"PIECEWISE"}' --attention-backend FLASHINFER --linear-backend triton --moe-backend tritonPrefill
prompt tok/s by context length8K context15.1K543 ms TTFT
16K context14.6K1,107 ms TTFT
32K context13K2,476 ms TTFT
64K context10.7K6,001 ms TTFT
128K context7,81616,446 ms TTFT
187K context6,28129,885 ms TTFT
192K context6,06231,791 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 90.040 ms TTFT | 164.258 ms TTFT | 277.381 ms TTFT | 401.4101 ms TTFT | 641.7163 ms TTFT | 922.3267 ms TTFT | Over capacity | Over capacity |
| 16K | 86.648 ms TTFT | 161.466 ms TTFT | 268.998 ms TTFT | 393.3142 ms TTFT | 614.8223 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 84.376 ms TTFT | 158.2105 ms TTFT | 261.8146 ms TTFT | 373.3232 ms TTFT | 568.0379 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 81.5131 ms TTFT | 149.3180 ms TTFT | 245.5254 ms TTFT | 342.9416 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 73.3262 ms TTFT | 133.0350 ms TTFT | 219.2525 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 187K | 66.2414 ms TTFT | 122.2573 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.1K | — | 543 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 14.6K | — | 1,107 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13K | — | 2,476 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.7K | — | 6,001 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,816.0 | — | 16,446 ms | — | — | 1 | — | — |
| Prefill | 187K | 1 | Valid | 6,281.0 | — | 29,885 ms | — | — | 1 | — | — |
| Prefill | 192K | 1 | Valid | 6,062.0 | — | 31,791 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 90.0 | 90.0 | 40 ms | 11.1 ms | 1.0 | 0 | 2,699 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 164.2 | 82.3 | 58 ms | 12.2 ms | 2.0 | 0 | 4,926 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 277.3 | 69.4 | 81 ms | 14.4 ms | 4.0 | 0 | 8,320 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 401.4 | 50.5 | 101 ms | 19.8 ms | 8.0 | 0 | 12,040 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 641.7 | 40.5 | 163 ms | 24.7 ms | 16.0 | 0 | 19,248 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 922.3 | 29.2 | 267 ms | 34.2 ms | 32.0 | 0 | 27,648 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 86.6 | 86.6 | 48 ms | 11.5 ms | 1.0 | 0 | 2,597 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 161.4 | 80.7 | 66 ms | 12.4 ms | 2.0 | 0 | 4,840 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 268.9 | 67.4 | 98 ms | 14.8 ms | 4.0 | 0 | 8,064 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 393.3 | 49.4 | 142 ms | 20.3 ms | 8.0 | 0 | 11,800 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 614.8 | 38.7 | 223 ms | 25.8 ms | 16.0 | 0 | 18,432 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 84.3 | 84.3 | 76 ms | 11.9 ms | 1.0 | 0 | 2,530 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 158.2 | 79.1 | 105 ms | 12.6 ms | 2.0 | 0 | 4,746 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 261.8 | 65.5 | 146 ms | 15.3 ms | 4.0 | 0 | 7,852 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 373.3 | 46.9 | 232 ms | 21.3 ms | 8.0 | 0 | 11,200 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 568.0 | 35.7 | 379 ms | 28.0 ms | 16.0 | 0 | 17,040 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 81.5 | 81.6 | 131 ms | 12.3 ms | 1.0 | 0 | 2,444 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 149.3 | 74.7 | 180 ms | 13.4 ms | 2.0 | 0 | 4,478 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 245.5 | 61.5 | 254 ms | 16.3 ms | 4.0 | 0 | 7,364 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 342.9 | 43.0 | 416 ms | 23.2 ms | 8.0 | 0 | 10,280 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 73.3 | 73.4 | 262 ms | 13.6 ms | 1.0 | 0 | 2,200 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 133.0 | 66.6 | 350 ms | 15.0 ms | 2.0 | 0 | 3,990 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 219.2 | 54.9 | 525 ms | 18.2 ms | 4.0 | 0 | 6,572 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 1 | Valid | 66.2 | 66.3 | 414 ms | 15.1 ms | 1.0 | 0 | 1,984 | 30.0 s |
| Sustained decode | 187K | 2 | Valid | 122.2 | 61.2 | 573 ms | 16.3 ms | 2.0 | 0 | 3,666 | 30.0 s |
| Sustained decode | 187K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 187K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.