Speed test matrix

Devstral-2-123B-Instruct-2512 on H200 NVL 141GB ×2

vllm · 0.26.1rc1.dev638+g52be12cfa · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 4, 2026, 07:13 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: devstral
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-mistralai-vllm/bin/vllm" serve mistralai/Devstral-2-123B-Instruct-2512 --served-model-name devstral --trust-remote-code --tensor-parallel-size 2 --enable-auto-tool-choice --tool-call-parser mistral --max-model-len 262144 --gpu-memory-utilization 0.853163 --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context4,8821,679 ms TTFT

16K context4,9843,264 ms TTFT

32K context4,5207,164 ms TTFT

64K context3,53018,306 ms TTFT

128K context2,39853,837 ms TTFT

251K context1,491169,758 ms TTFT

256K context1,471175,448 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	48.2208 ms TTFT	95.2333 ms TTFT	186.0597 ms TTFT	357.71,156 ms TTFT	648.22,041 ms TTFT	Over capacity	Over capacity	Over capacity
16K	47.1222 ms TTFT	92.6365 ms TTFT	178.3563 ms TTFT	331.21,190 ms TTFT	569.72,258 ms TTFT	Over capacity	Over capacity	Over capacity
32K	45.5239 ms TTFT	88.0401 ms TTFT	165.6677 ms TTFT	288.11,323 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
64K	42.6274 ms TTFT	80.2443 ms TTFT	143.9733 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
128K	38.0	36.9494 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	31.5	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	4,882.0	—	1,679 ms	—	—	1	—	—
Prefill	16K	1	Valid	4,984.0	—	3,264 ms	—	—	1	—	—
Prefill	32K	1	Valid	4,520.0	—	7,164 ms	—	—	1	—	—
Prefill	64K	1	Valid	3,530.0	—	18,306 ms	—	—	1	—	—
Prefill	128K	1	Valid	2,398.0	—	53,837 ms	—	—	1	—	—
Prefill	251K	1	Valid	1,491.0	—	169,758 ms	—	—	1	—	—
Prefill	256K	1	Valid	1,471.0	—	175,448 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	48.2	48.2	208 ms	20.8 ms	1.0	0	1,444	30.0 s
Sustained decode	8K	2	Valid	95.2	47.6	333 ms	21.0 ms	2.0	0	2,854	30.0 s
Sustained decode	8K	4	Valid	186.0	46.6	597 ms	21.5 ms	4.0	0	5,580	30.0 s
Sustained decode	8K	8	Valid	357.7	44.8	1,156 ms	22.3 ms	8.0	0	10,728	30.0 s
Sustained decode	8K	16	Valid	648.2	40.7	2,041 ms	24.6 ms	16.0	0	19,440	30.0 s
Sustained decode	8K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	47.1	47.2	222 ms	21.2 ms	1.0	0	1,415	30.0 s
Sustained decode	16K	2	Valid	92.6	46.3	365 ms	21.6 ms	2.0	0	2,776	30.0 s
Sustained decode	16K	4	Valid	178.3	44.6	563 ms	22.4 ms	4.0	0	5,351	30.0 s
Sustained decode	16K	8	Valid	331.2	41.5	1,190 ms	24.1 ms	8.0	0	9,936	30.0 s
Sustained decode	16K	16	Valid	569.7	35.8	2,258 ms	27.9 ms	16.0	0	17,088	30.0 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	45.5	45.5	239 ms	22.0 ms	1.0	0	1,365	30.0 s
Sustained decode	32K	2	Valid	88.0	44.0	401 ms	22.7 ms	2.0	0	2,638	30.0 s
Sustained decode	32K	4	Valid	165.6	41.5	677 ms	24.1 ms	4.0	0	4,964	30.0 s
Sustained decode	32K	8	Valid	288.1	36.1	1,323 ms	27.7 ms	8.0	0	8,640	30.0 s
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	42.6	42.6	274 ms	23.5 ms	1.0	0	1,278	30.0 s
Sustained decode	64K	2	Valid	80.2	40.1	443 ms	24.9 ms	2.0	0	2,406	30.0 s
Sustained decode	64K	4	Valid	143.9	36.1	733 ms	27.7 ms	4.0	0	4,316	30.0 s
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	38.0	—	—	—	1.0	0	177	4.7 s
Sustained decode	128K	2	Valid	36.9	34.4	494 ms	29.1 ms	2.0	0	1,108	30.0 s
Sustained decode	128K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	31.5	—	—	—	1.0	0	0	30.0 s
Sustained decode	251K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.