Speed test matrix

DeepSeek-V4-Flash on H200 NVL 141GB ×2

vllm · 0.25.0 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Aug 29, 2026, 05:04 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.25.0
Quantization: MXFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: deepseek
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-deepseek-vllm/bin/vllm" serve deepseek-ai/DeepSeek-V4-Flash --served-model-name deepseek --trust-remote-code --tensor-parallel-size 2 --reasoning-parser deepseek_v4 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --max-model-len 1048576 --gpu-memory-utilization 0.846011 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --block-size 256 --tokenizer-mode deepseek_v4

Prefill

prompt tok/s by context length

8K context14K584 ms TTFT

16K context14.9K1,090 ms TTFT

32K context15K2,153 ms TTFT

64K context15K4,314 ms TTFT

128K context14.3K9,052 ms TTFT

256K context12.6K20,482 ms TTFT

512K context10.1K50,933 ms TTFT

1019K context7,159143,365 ms TTFT

1024K context7,106145,136 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	120.159 ms TTFT	220.949 ms TTFT	378.2105 ms TTFT	602.2145 ms TTFT	909.2210 ms TTFT	1,345.3367 ms TTFT	1,968.3499 ms TTFT	2,811.9840 ms TTFT
16K	119.595 ms TTFT	220.2176 ms TTFT	375.1201 ms TTFT	604.9315 ms TTFT	899.5405 ms TTFT	1,332.2640 ms TTFT	1,898.91,099 ms TTFT	2,777.21,519 ms TTFT
32K	119.0149 ms TTFT	218.4260 ms TTFT	377.4277 ms TTFT	596.1358 ms TTFT	891.7533 ms TTFT	1,294.3853 ms TTFT	1,927.11,488 ms TTFT	2,752.52,559 ms TTFT
64K	118.4209 ms TTFT	216.4312 ms TTFT	370.1360 ms TTFT	594.5549 ms TTFT	888.3876 ms TTFT	1,299.61,418 ms TTFT	1,854.62,699 ms TTFT	2,707.45,085 ms TTFT
128K	117.5306 ms TTFT	214.4460 ms TTFT	366.7606 ms TTFT	578.9935 ms TTFT	862.71,783 ms TTFT	1,231.93,278 ms TTFT	1,753.26,023 ms TTFT	2,595.911,604 ms TTFT
256K	115.5602 ms TTFT	210.2982 ms TTFT	357.51,249 ms TTFT	562.42,011 ms TTFT	850.43,383 ms TTFT	1,211.16,101 ms TTFT	1,713.711,808 ms TTFT	2,457.423,291 ms TTFT
512K	112.11,270 ms TTFT	203.71,842 ms TTFT	344.52,686 ms TTFT	536.74,439 ms TTFT	795.97,864 ms TTFT	1,111.915,128 ms TTFT	1,557.229,747 ms TTFT	2,171.862,718 ms TTFT
1019K	108.02,580 ms TTFT	196.73,817 ms TTFT	328.35,460 ms TTFT	507.99,278 ms TTFT	734.616,539 ms TTFT	1,007.532,264 ms TTFT	1,305.959,263 ms TTFT	1,465.1Requests queued104,729 ms TTFT

Requests queued: the server could not run all requests at once, so some waited in a queue.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	14K	—	584 ms	—	—	1	—	—
Prefill	16K	1	Valid	14.9K	—	1,090 ms	—	—	1	—	—
Prefill	32K	1	Valid	15K	—	2,153 ms	—	—	1	—	—
Prefill	64K	1	Valid	15K	—	4,314 ms	—	—	1	—	—
Prefill	128K	1	Valid	14.3K	—	9,052 ms	—	—	1	—	—
Prefill	256K	1	Valid	12.6K	—	20,482 ms	—	—	1	—	—
Prefill	512K	1	Valid	10.1K	—	50,933 ms	—	—	1	—	—
Prefill	1019K	1	Valid	7,159.0	—	143,365 ms	—	—	1	—	—
Prefill	1024K	1	Valid	7,106.0	—	145,136 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	120.1	120.1	59 ms	8.3 ms	1.0	0	3,603	30.0 s
Sustained decode	8K	2	Valid	220.9	110.5	49 ms	9.0 ms	2.0	0	6,628	30.0 s
Sustained decode	8K	4	Valid	378.2	94.8	105 ms	10.5 ms	4.0	0	11,345	30.0 s
Sustained decode	8K	8	Valid	602.2	75.8	145 ms	13.2 ms	8.0	0	18,064	30.0 s
Sustained decode	8K	16	Valid	909.2	57.4	210 ms	17.4 ms	16.0	0	27,264	30.0 s
Sustained decode	8K	32	Valid	1,345.3	42.7	367 ms	23.4 ms	32.0	0	40,353	30.0 s
Sustained decode	8K	64	Valid	1,968.3	31.3	499 ms	31.9 ms	64.0	0	59,013	30.0 s
Sustained decode	8K	128	Valid	2,811.9	22.4	840 ms	44.7 ms	128.0	0	84,359	30.0 s
Sustained decode	16K	1	Valid	119.5	119.5	95 ms	8.4 ms	1.0	0	3,584	30.0 s
Sustained decode	16K	2	Valid	220.2	110.1	176 ms	9.1 ms	2.0	0	6,607	30.0 s
Sustained decode	16K	4	Valid	375.1	94.1	201 ms	10.6 ms	4.0	0	11,252	30.0 s
Sustained decode	16K	8	Valid	604.9	76.0	315 ms	13.2 ms	8.0	0	18,144	30.0 s
Sustained decode	16K	16	Valid	899.5	56.5	405 ms	17.7 ms	16.0	0	26,977	30.0 s
Sustained decode	16K	32	Valid	1,332.2	42.1	640 ms	23.8 ms	32.0	0	39,968	30.0 s
Sustained decode	16K	64	Valid	1,898.9	30.1	1,099 ms	33.2 ms	64.0	0	56,959	30.0 s
Sustained decode	16K	128	Valid	2,777.2	21.7	1,519 ms	46.1 ms	128.0	0	83,336	30.0 s
Sustained decode	32K	1	Valid	119.0	119.1	149 ms	8.4 ms	1.0	0	3,571	30.0 s
Sustained decode	32K	2	Valid	218.4	109.3	260 ms	9.2 ms	2.0	0	6,554	30.0 s
Sustained decode	32K	4	Valid	377.4	94.7	277 ms	10.6 ms	4.0	0	11,328	30.0 s
Sustained decode	32K	8	Valid	596.1	74.8	358 ms	13.4 ms	8.0	0	17,880	30.0 s
Sustained decode	32K	16	Valid	891.7	56.1	533 ms	17.8 ms	16.0	0	26,739	30.0 s
Sustained decode	32K	32	Valid	1,294.3	40.7	853 ms	24.6 ms	32.0	0	38,819	30.0 s
Sustained decode	32K	64	Valid	1,927.1	30.2	1,488 ms	33.1 ms	64.0	0	57,792	30.0 s
Sustained decode	32K	128	Valid	2,752.5	21.5	2,559 ms	46.6 ms	128.0	0	82,569	30.0 s
Sustained decode	64K	1	Valid	118.4	118.6	209 ms	8.4 ms	1.0	0	3,555	30.0 s
Sustained decode	64K	2	Valid	216.4	108.3	312 ms	9.2 ms	2.0	0	6,490	30.0 s
Sustained decode	64K	4	Valid	370.1	92.6	360 ms	10.8 ms	4.0	0	11,100	30.0 s
Sustained decode	64K	8	Valid	594.5	74.7	549 ms	13.4 ms	8.0	0	17,832	30.0 s
Sustained decode	64K	16	Valid	888.3	55.9	876 ms	17.9 ms	16.0	0	26,639	30.0 s
Sustained decode	64K	32	Valid	1,299.6	40.9	1,418 ms	24.5 ms	32.0	0	38,977	30.0 s
Sustained decode	64K	64	Valid	1,854.6	29.2	2,699 ms	34.3 ms	64.0	0	55,618	30.0 s
Sustained decode	64K	128	Valid	2,707.4	21.2	5,085 ms	47.1 ms	128.0	0	81,155	30.0 s
Sustained decode	128K	1	Valid	117.5	117.7	306 ms	8.5 ms	1.0	0	3,526	30.0 s
Sustained decode	128K	2	Valid	214.4	107.5	460 ms	9.3 ms	2.0	0	6,432	30.0 s
Sustained decode	128K	4	Valid	366.7	91.9	606 ms	10.9 ms	4.0	0	11,000	30.0 s
Sustained decode	128K	8	Valid	578.9	72.7	935 ms	13.8 ms	8.0	0	17,368	30.0 s
Sustained decode	128K	16	Valid	862.7	54.3	1,783 ms	18.4 ms	16.0	0	25,872	30.0 s
Sustained decode	128K	32	Valid	1,231.9	39.0	3,278 ms	25.7 ms	32.0	0	36,960	30.0 s
Sustained decode	128K	64	Valid	1,753.2	28.0	6,023 ms	35.7 ms	64.0	0	52,606	30.0 s
Sustained decode	128K	128	Valid	2,595.9	20.8	11,604 ms	48.2 ms	128.0	0	77,827	30.0 s
Sustained decode	256K	1	Valid	115.5	115.8	602 ms	8.6 ms	1.0	0	3,465	30.0 s
Sustained decode	256K	2	Valid	210.2	105.8	982 ms	9.5 ms	2.0	0	6,311	30.0 s
Sustained decode	256K	4	Valid	357.5	89.6	1,249 ms	11.2 ms	4.0	0	10,724	30.0 s
Sustained decode	256K	8	Valid	562.4	70.6	2,011 ms	14.2 ms	8.0	0	16,873	30.0 s
Sustained decode	256K	16	Valid	850.4	53.3	3,383 ms	18.8 ms	16.0	0	25,504	30.0 s
Sustained decode	256K	32	Valid	1,211.1	38.0	6,101 ms	26.3 ms	32.0	0	36,321	30.0 s
Sustained decode	256K	64	Valid	1,713.7	27.1	11,808 ms	36.9 ms	64.0	0	51,393	30.0 s
Sustained decode	256K	128	Valid	2,457.4	19.7	23,291 ms	50.9 ms	128.0	0	73,724	30.0 s
Sustained decode	512K	1	Valid	112.1	112.6	1,270 ms	8.9 ms	1.0	0	3,364	30.0 s
Sustained decode	512K	2	Valid	203.7	102.2	1,842 ms	9.8 ms	2.0	0	6,114	30.0 s
Sustained decode	512K	4	Valid	344.5	86.6	2,686 ms	11.5 ms	4.0	0	10,332	30.0 s
Sustained decode	512K	8	Valid	536.7	67.5	4,439 ms	14.8 ms	8.0	0	16,096	30.0 s
Sustained decode	512K	16	Valid	795.9	50.0	7,864 ms	20.0 ms	16.0	0	23,872	30.0 s
Sustained decode	512K	32	Valid	1,111.9	35.0	15,128 ms	28.6 ms	32.0	0	33,340	30.0 s
Sustained decode	512K	64	Valid	1,557.2	25.1	29,747 ms	39.8 ms	64.0	0	46,720	30.0 s
Sustained decode	512K	128	Valid	2,171.8	17.9	62,718 ms	55.8 ms	128.0	0	65,156	30.0 s
Sustained decode	1019K	1	Valid	108.0	108.7	2,580 ms	9.2 ms	1.0	0	3,241	30.0 s
Sustained decode	1019K	2	Valid	196.7	98.9	3,817 ms	10.1 ms	2.0	0	5,900	30.0 s
Sustained decode	1019K	4	Valid	328.3	82.8	5,460 ms	12.1 ms	4.0	0	9,852	30.0 s
Sustained decode	1019K	8	Valid	507.9	63.9	9,278 ms	15.6 ms	8.0	0	15,235	30.0 s
Sustained decode	1019K	16	Valid	734.6	46.3	16,539 ms	21.6 ms	16.0	0	22,033	30.0 s
Sustained decode	1019K	32	Valid	1,007.5	32.2	32,264 ms	31.0 ms	32.0	0	30,211	30.0 s
Sustained decode	1019K	64	Valid	1,305.9	21.8	59,263 ms	45.8 ms	63.7	4	39,159	30.0 s
Sustained decode	1019K	128	Requests queued	1,465.1	17.6	104,729 ms	56.9 ms	91.4	12	43,887	30.0 s

Requests queued: the server could not run all requests at once, so some waited in a queue.