Speed test matrix

MiMo-V2.5 on H200 SXM 141GB ×4

vllm · 0.29.0 · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 17, 2026, 19:19 UTC

Hardware

Type: Discrete GPU
GPU: H200 SXM
GPU count: 4
VRAM: 564 GB (4 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.29.0
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: XiaomiMiMo/MiMo-V2.5
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_V2_MODEL_RUNNER=1 FLASH_ATTENTION_CUTE_DSL_CACHE_ENABLED=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve XiaomiMiMo/MiMo-V2.5 --served-model-name XiaomiMiMo/MiMo-V2.5 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser qwen3 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --kernel-config.enable_flashinfer_autotune=False

Prefill

prompt tok/s by context length

8K context35.5K231 ms TTFT

16K context38K427 ms TTFT

32K context37.2K868 ms TTFT

64K context33.9K1,899 ms TTFT

128K context28.7K4,486 ms TTFT

256K context22K11,662 ms TTFT

512K context15K34,184 ms TTFT

1019K context9,161111,641 ms TTFT

1024K context8,767117,226 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	221.749 ms TTFT	387.965 ms TTFT	689.977 ms TTFT	1,054.591 ms TTFT	2,043.2108 ms TTFT	3,193.8326 ms TTFT	4,273.7549 ms TTFT	4,684.31,054 ms TTFT
16K	219.566 ms TTFT	385.376 ms TTFT	684.998 ms TTFT	1,042.7111 ms TTFT	2,003.5141 ms TTFT	3,100.7487 ms TTFT	4,113.2882 ms TTFT	4,494.41,810 ms TTFT
32K	215.6103 ms TTFT	378.8120 ms TTFT	674.2135 ms TTFT	1,019.2120 ms TTFT	1,930.8321 ms TTFT	2,930.91,005 ms TTFT	3,823.41,922 ms TTFT	Over capacity
64K	208.8184 ms TTFT	367.5206 ms TTFT	654.2215 ms TTFT	975.7218 ms TTFT	1,806.31,087 ms TTFT	2,645.32,035 ms TTFT	Over capacity	Over capacity
128K	195.3369 ms TTFT	346.5387 ms TTFT	615.5403 ms TTFT	912.91,290 ms TTFT	1,593.72,345 ms TTFT	Over capacity	Over capacity	Over capacity
256K	172.2765 ms TTFT	308.1881 ms TTFT	547.5838 ms TTFT	797.12,814 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
512K	138.01,705 ms TTFT	250.8Requests queued1,883 ms TTFT	459.02,499 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	108.14,262 ms TTFT	201.76,078 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	35.5K	—	231 ms	—	—	1	—	—
Prefill	16K	1	Valid	38K	—	427 ms	—	—	1	—	—
Prefill	32K	1	Valid	37.2K	—	868 ms	—	—	1	—	—
Prefill	64K	1	Valid	33.9K	—	1,899 ms	—	—	1	—	—
Prefill	128K	1	Valid	28.7K	—	4,486 ms	—	—	1	—	—
Prefill	256K	1	Valid	22K	—	11,662 ms	—	—	1	—	—
Prefill	512K	1	Valid	15K	—	34,184 ms	—	—	1	—	—
Prefill	1019K	1	Valid	9,161.0	—	111,641 ms	—	—	1	—	—
Prefill	1024K	1	Valid	8,767.0	—	117,226 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	221.7	222.1	49 ms	4.5 ms	1.0	1	6,650	30.0 s
Sustained decode	8K	2	Valid	387.9	194.3	65 ms	5.1 ms	2.0	2	11,636	30.0 s
Sustained decode	8K	4	Valid	689.9	172.9	77 ms	5.8 ms	4.0	4	20,695	30.0 s
Sustained decode	8K	8	Valid	1,054.5	132.2	91 ms	7.6 ms	8.0	8	31,631	30.0 s
Sustained decode	8K	16	Valid	2,043.2	128.1	108 ms	7.8 ms	16.0	16	61,296	30.0 s
Sustained decode	8K	32	Valid	3,193.8	99.6	326 ms	10.0 ms	32.0	0	95,808	30.0 s
Sustained decode	8K	64	Valid	4,273.7	66.6	549 ms	15.0 ms	64.0	0	128,192	30.0 s
Sustained decode	8K	128	Valid	4,684.3	36.8	1,054 ms	27.2 ms	128.0	0	140,544	30.0 s
Sustained decode	16K	1	Valid	219.5	220.0	66 ms	4.5 ms	1.0	1	6,585	30.0 s
Sustained decode	16K	2	Valid	385.3	193.1	76 ms	5.2 ms	2.0	2	11,558	30.0 s
Sustained decode	16K	4	Valid	684.9	171.7	98 ms	5.8 ms	4.0	4	20,545	30.0 s
Sustained decode	16K	8	Valid	1,042.7	130.7	111 ms	7.6 ms	8.0	8	31,278	30.0 s
Sustained decode	16K	16	Valid	2,003.5	125.7	141 ms	8.0 ms	15.9	16	60,106	30.0 s
Sustained decode	16K	32	Valid	3,100.7	96.9	487 ms	10.3 ms	32.0	0	93,025	30.0 s
Sustained decode	16K	64	Valid	4,113.2	64.4	882 ms	15.5 ms	64.0	0	123,392	30.0 s
Sustained decode	16K	128	Valid	4,494.4	35.4	1,810 ms	28.2 ms	128.0	0	134,784	30.0 s
Sustained decode	32K	1	Valid	215.6	216.5	103 ms	4.6 ms	1.0	1	6,469	30.0 s
Sustained decode	32K	2	Valid	378.8	190.2	120 ms	5.3 ms	2.0	2	11,364	30.0 s
Sustained decode	32K	4	Valid	674.2	169.2	135 ms	5.9 ms	4.0	4	20,223	30.0 s
Sustained decode	32K	8	Valid	1,019.2	127.9	120 ms	7.8 ms	8.0	8	30,576	30.0 s
Sustained decode	32K	16	Valid	1,930.8	120.8	321 ms	8.3 ms	16.0	10	57,906	30.0 s
Sustained decode	32K	32	Valid	2,930.9	91.5	1,005 ms	10.9 ms	32.0	0	87,904	30.0 s
Sustained decode	32K	64	Valid	3,823.4	59.9	1,922 ms	16.7 ms	64.0	0	114,688	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	208.8	210.2	184 ms	4.8 ms	1.0	1	6,264	30.0 s
Sustained decode	64K	2	Valid	367.5	185.1	206 ms	5.4 ms	2.0	2	11,024	30.0 s
Sustained decode	64K	4	Valid	654.2	164.6	215 ms	6.1 ms	4.0	4	19,625	30.0 s
Sustained decode	64K	8	Valid	975.7	122.8	218 ms	8.1 ms	8.0	8	29,258	30.0 s
Sustained decode	64K	16	Valid	1,806.3	112.7	1,087 ms	8.9 ms	16.0	0	54,176	30.0 s
Sustained decode	64K	32	Valid	2,645.3	82.6	2,035 ms	12.1 ms	32.0	0	79,360	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	195.3	198.1	369 ms	5.0 ms	1.0	1	5,860	30.0 s
Sustained decode	128K	2	Valid	346.5	175.7	387 ms	5.7 ms	2.0	2	10,396	30.0 s
Sustained decode	128K	4	Valid	615.5	156.0	403 ms	6.4 ms	4.0	4	18,466	30.0 s
Sustained decode	128K	8	Valid	912.9	114.2	1,290 ms	8.8 ms	8.0	0	27,385	30.0 s
Sustained decode	128K	16	Valid	1,593.7	99.5	2,345 ms	10.1 ms	16.0	0	47,808	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	172.2	177.3	765 ms	5.6 ms	1.0	1	5,166	30.0 s
Sustained decode	256K	2	Valid	308.1	159.0	881 ms	6.3 ms	1.9	2	9,244	30.0 s
Sustained decode	256K	4	Valid	547.5	140.9	838 ms	7.1 ms	3.9	4	16,424	30.0 s
Sustained decode	256K	8	Valid	797.1	99.9	2,814 ms	10.0 ms	8.0	0	23,912	30.0 s
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	138.0	148.5	1,705 ms	6.7 ms	0.9	1	4,140	30.0 s
Sustained decode	512K	2	Requests queued	250.8	135.2	1,883 ms	7.4 ms	1.9	2	7,524	30.0 s
Sustained decode	512K	4	Valid	459.0	119.0	2,499 ms	8.4 ms	3.9	3	13,770	30.0 s
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	108.1	108.9	4,262 ms	9.2 ms	1.0	0	3,242	30.0 s
Sustained decode	1019K	2	Valid	201.7	101.5	6,078 ms	9.9 ms	2.0	0	6,050	30.0 s
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.