Speed test matrix

MiniMax-M3-MXFP8 on H200 SXM 141GB ×8

vllm · 0.29.0 · MXFP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 16, 2026, 20:47 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 8
VRAM: 1128 GB (8 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: MXFP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: MiniMaxAI/MiniMax-M3-MXFP8
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve MiniMaxAI/MiniMax-M3-MXFP8 --served-model-name MiniMaxAI/MiniMax-M3-MXFP8 --trust-remote-code --tensor-parallel-size 8 --reasoning-parser minimax_m3 --enable-auto-tool-choice --tool-call-parser minimax_m3 --max-model-len 1048576 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --enforce-eager --block-size 128 --limit-mm-per-prompt '{"image":1,"video":0}'

Prefill

prompt tok/s by context length

8K context18.4K444 ms TTFT

16K context18.8K853 ms TTFT

32K context18.6K1,713 ms TTFT

64K context18K3,516 ms TTFT

128K context16.9K7,495 ms TTFT

256K context15.1K16,725 ms TTFT

512K context12.4K40,814 ms TTFT

1019K context9,215108,915 ms TTFT

1024K context9,121110,580 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	11.1129 ms TTFT	22.3258 ms TTFT	44.2345 ms TTFT	88.7374 ms TTFT	175.8393 ms TTFT	347.5591 ms TTFT	624.41,249 ms TTFT	Over capacity
16K	11.1149 ms TTFT	22.0257 ms TTFT	43.7387 ms TTFT	87.5402 ms TTFT	173.3592 ms TTFT	347.3810 ms TTFT	629.01,288 ms TTFT	Over capacity
32K	11.1193 ms TTFT	21.9300 ms TTFT	43.6435 ms TTFT	87.2625 ms TTFT	173.5829 ms TTFT	348.01,287 ms TTFT	Over capacity	Over capacity
64K	11.1287 ms TTFT	22.0443 ms TTFT	43.8622 ms TTFT	87.4920 ms TTFT	172.91,427 ms TTFT	Over capacity	Over capacity	Over capacity
128K	11.1487 ms TTFT	22.0686 ms TTFT	43.61,000 ms TTFT	87.41,617 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
256K	11.11,031 ms TTFT	22.01,417 ms TTFT	43.72,036 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
512K	11.12,350 ms TTFT	21.93,150 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	11.15,193 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	18.4K	—	444 ms	—	—	1	—	—
Prefill	16K	1	Valid	18.8K	—	853 ms	—	—	1	—	—
Prefill	32K	1	Valid	18.6K	—	1,713 ms	—	—	1	—	—
Prefill	64K	1	Valid	18K	—	3,516 ms	—	—	1	—	—
Prefill	128K	1	Valid	16.9K	—	7,495 ms	—	—	1	—	—
Prefill	256K	1	Valid	15.1K	—	16,725 ms	—	—	1	—	—
Prefill	512K	1	Valid	12.4K	—	40,814 ms	—	—	1	—	—
Prefill	1019K	1	Valid	9,215.0	—	108,915 ms	—	—	1	—	—
Prefill	1024K	1	Valid	9,121.0	—	110,580 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	11.1	11.1	129 ms	90.2 ms	1.0	0	332	29.9 s
Sustained decode	8K	2	Valid	22.3	11.1	258 ms	90.0 ms	2.0	0	668	30.0 s
Sustained decode	8K	4	Valid	44.2	11.0	345 ms	90.5 ms	4.0	0	1,324	29.9 s
Sustained decode	8K	8	Valid	88.7	11.1	374 ms	90.2 ms	8.0	0	2,656	29.9 s
Sustained decode	8K	16	Valid	175.8	11.0	393 ms	91.3 ms	16.0	0	5,264	30.0 s
Sustained decode	8K	32	Valid	347.5	10.4	591 ms	96.2 ms	32.0	0	10,400	29.9 s
Sustained decode	8K	64	Valid	624.4	9.4	1,249 ms	106.9 ms	64.0	0	18,688	29.9 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	11.1	11.1	149 ms	90.2 ms	1.0	0	333	30.0 s
Sustained decode	16K	2	Valid	22.0	11.0	257 ms	91.1 ms	2.0	0	658	29.9 s
Sustained decode	16K	4	Valid	43.7	10.9	387 ms	91.6 ms	4.0	0	1,312	30.0 s
Sustained decode	16K	8	Valid	87.5	10.9	402 ms	91.5 ms	8.0	0	2,624	30.0 s
Sustained decode	16K	16	Valid	173.3	10.8	592 ms	92.5 ms	16.0	0	5,184	29.9 s
Sustained decode	16K	32	Valid	347.3	10.8	810 ms	92.3 ms	32.0	0	10,400	29.9 s
Sustained decode	16K	64	Valid	629.0	9.8	1,288 ms	102.1 ms	64.0	0	18,816	29.9 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	11.1	11.1	193 ms	90.1 ms	1.0	0	332	29.9 s
Sustained decode	32K	2	Valid	21.9	11.0	300 ms	91.3 ms	2.0	0	658	30.0 s
Sustained decode	32K	4	Valid	43.6	10.9	435 ms	91.7 ms	4.0	0	1,308	30.0 s
Sustained decode	32K	8	Valid	87.2	10.9	625 ms	91.8 ms	8.0	0	2,616	30.0 s
Sustained decode	32K	16	Valid	173.5	10.8	829 ms	92.5 ms	16.0	0	5,200	30.0 s
Sustained decode	32K	32	Valid	348.0	10.8	1,287 ms	92.4 ms	32.0	0	10,432	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	11.1	11.1	287 ms	90.2 ms	1.0	0	332	29.9 s
Sustained decode	64K	2	Valid	22.0	11.0	443 ms	91.0 ms	2.0	0	660	30.0 s
Sustained decode	64K	4	Valid	43.8	10.9	622 ms	91.4 ms	4.0	0	1,312	30.0 s
Sustained decode	64K	8	Valid	87.4	10.9	920 ms	91.6 ms	8.0	0	2,616	29.9 s
Sustained decode	64K	16	Valid	172.9	10.7	1,427 ms	93.1 ms	16.0	0	5,184	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	11.1	11.1	487 ms	90.1 ms	1.0	0	332	29.9 s
Sustained decode	128K	2	Valid	22.0	11.0	686 ms	91.0 ms	2.0	0	658	30.0 s
Sustained decode	128K	4	Valid	43.6	10.9	1,000 ms	91.6 ms	4.0	0	1,308	30.0 s
Sustained decode	128K	8	Valid	87.4	10.9	1,617 ms	91.6 ms	8.0	0	2,616	29.9 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	11.1	11.2	1,031 ms	89.5 ms	1.0	0	333	30.0 s
Sustained decode	256K	2	Valid	22.0	11.0	1,417 ms	90.6 ms	2.0	0	658	30.0 s
Sustained decode	256K	4	Valid	43.7	11.0	2,036 ms	91.2 ms	4.0	0	1,308	30.0 s
Sustained decode	256K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	11.1	11.2	2,350 ms	89.2 ms	1.0	0	333	30.0 s
Sustained decode	512K	2	Valid	21.9	11.1	3,150 ms	90.5 ms	2.0	0	656	29.9 s
Sustained decode	512K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	11.1	11.3	5,193 ms	88.3 ms	1.0	0	333	30.0 s
Sustained decode	1019K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.