Speed test matrix

DeepSeek-V4-Flash-0731 on H200 NVL 141GB ×2

vllm · 0.25.0 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Aug 29, 2026, 00:06 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.25.0
Quantization: MXFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: deepseek
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-deepseek-vllm/bin/vllm" serve deepseek-ai/DeepSeek-V4-Flash-0731 --served-model-name deepseek --trust-remote-code --tensor-parallel-size 2 --reasoning-parser deepseek_v4 --enable-auto-tool-choice --tool-call-parser deepseek_v4 --max-model-len 1048576 --gpu-memory-utilization 0.846011 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --block-size 256 --tokenizer-mode deepseek_v4

Prefill

prompt tok/s by context length

8K context14.7K556 ms TTFT

16K context15K1,081 ms TTFT

32K context15.2K2,130 ms TTFT

64K context15.1K4,291 ms TTFT

128K context11.6K11,101 ms TTFT

256K context12.2K21,223 ms TTFT

512K context10.1K50,939 ms TTFT

1019K context7,123144,087 ms TTFT

1024K context7,129144,675 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	120.349 ms TTFT	220.869 ms TTFT	380.5120 ms TTFT	606.3177 ms TTFT	910.8241 ms TTFT	1,351.2331 ms TTFT	1,985.1484 ms TTFT	2,877.6876 ms TTFT
16K	119.5117 ms TTFT	219.8141 ms TTFT	377.8198 ms TTFT	616.4298 ms TTFT	902.8641 ms TTFT	1,342.5717 ms TTFT	1,966.11,076 ms TTFT	2,835.61,593 ms TTFT
32K	119.0138 ms TTFT	218.1242 ms TTFT	375.8250 ms TTFT	597.5367 ms TTFT	918.1513 ms TTFT	1,361.3892 ms TTFT	1,969.71,395 ms TTFT	2,827.62,687 ms TTFT
64K	118.3201 ms TTFT	216.6328 ms TTFT	369.5318 ms TTFT	596.5468 ms TTFT	921.1793 ms TTFT	1,338.81,398 ms TTFT	1,946.82,622 ms TTFT	2,783.15,116 ms TTFT
128K	117.6305 ms TTFT	214.4444 ms TTFT	370.5650 ms TTFT	587.7955 ms TTFT	895.21,605 ms TTFT	1,302.43,275 ms TTFT	1,879.46,184 ms TTFT	2,708.911,959 ms TTFT
256K	115.8598 ms TTFT	211.0919 ms TTFT	363.71,336 ms TTFT	581.62,001 ms TTFT	869.63,396 ms TTFT	1,250.56,263 ms TTFT	1,851.011,836 ms TTFT	2,565.723,093 ms TTFT
512K	112.11,380 ms TTFT	203.91,984 ms TTFT	349.83,165 ms TTFT	549.04,302 ms TTFT	826.27,927 ms TTFT	1,168.315,974 ms TTFT	1,652.633,009 ms TTFT	2,253.964,664 ms TTFT
1019K	108.02,801 ms TTFT	198.24,168 ms TTFT	335.46,136 ms TTFT	524.810,433 ms TTFT	785.120,453 ms TTFT	1,053.937,007 ms TTFT	1,378.4Requests queued50,884 ms TTFT	1,476.0Requests queued105,620 ms TTFT

Requests queued: the server could not run all requests at once, so some waited in a queue.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	14.7K	—	556 ms	—	—	1	—	—
Prefill	16K	1	Valid	15K	—	1,081 ms	—	—	1	—	—
Prefill	32K	1	Valid	15.2K	—	2,130 ms	—	—	1	—	—
Prefill	64K	1	Valid	15.1K	—	4,291 ms	—	—	1	—	—
Prefill	128K	1	Valid	11.6K	—	11,101 ms	—	—	1	—	—
Prefill	256K	1	Valid	12.2K	—	21,223 ms	—	—	1	—	—
Prefill	512K	1	Valid	10.1K	—	50,939 ms	—	—	1	—	—
Prefill	1019K	1	Valid	7,123.0	—	144,087 ms	—	—	1	—	—
Prefill	1024K	1	Valid	7,129.0	—	144,675 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	120.3	120.2	49 ms	8.3 ms	1.0	0	3,607	30.0 s
Sustained decode	8K	2	Valid	220.8	110.4	69 ms	9.1 ms	2.0	0	6,622	30.0 s
Sustained decode	8K	4	Valid	380.5	95.2	120 ms	10.5 ms	4.0	0	11,412	30.0 s
Sustained decode	8K	8	Valid	606.3	76.2	177 ms	13.1 ms	8.0	0	18,184	30.0 s
Sustained decode	8K	16	Valid	910.8	57.4	241 ms	17.4 ms	16.0	0	27,313	30.0 s
Sustained decode	8K	32	Valid	1,351.2	42.8	331 ms	23.4 ms	32.0	0	40,512	30.0 s
Sustained decode	8K	64	Valid	1,985.1	31.5	484 ms	31.7 ms	64.0	0	59,523	30.0 s
Sustained decode	8K	128	Valid	2,877.6	22.8	876 ms	43.9 ms	128.0	0	86,276	30.0 s
Sustained decode	16K	1	Valid	119.5	119.6	117 ms	8.4 ms	1.0	0	3,586	30.0 s
Sustained decode	16K	2	Valid	219.8	110.0	141 ms	9.1 ms	2.0	0	6,598	30.0 s
Sustained decode	16K	4	Valid	377.8	94.7	198 ms	10.6 ms	4.0	0	11,331	30.0 s
Sustained decode	16K	8	Valid	616.4	77.3	298 ms	12.9 ms	8.0	0	18,488	30.0 s
Sustained decode	16K	16	Valid	902.8	57.1	641 ms	17.5 ms	16.0	0	27,072	30.0 s
Sustained decode	16K	32	Valid	1,342.5	42.5	717 ms	23.5 ms	32.0	0	40,257	30.0 s
Sustained decode	16K	64	Valid	1,966.1	31.1	1,076 ms	32.1 ms	64.0	0	58,943	30.0 s
Sustained decode	16K	128	Valid	2,835.6	22.2	1,593 ms	45.1 ms	128.0	0	84,994	30.0 s
Sustained decode	32K	1	Valid	119.0	119.0	138 ms	8.4 ms	1.0	0	3,570	30.0 s
Sustained decode	32K	2	Valid	218.1	109.2	242 ms	9.2 ms	2.0	0	6,542	30.0 s
Sustained decode	32K	4	Valid	375.8	94.2	250 ms	10.6 ms	4.0	0	11,272	30.0 s
Sustained decode	32K	8	Valid	597.5	75.1	367 ms	13.3 ms	8.0	0	17,920	30.0 s
Sustained decode	32K	16	Valid	918.1	57.5	513 ms	17.4 ms	16.0	0	27,536	30.0 s
Sustained decode	32K	32	Valid	1,361.3	42.7	892 ms	23.4 ms	32.0	0	40,832	30.0 s
Sustained decode	32K	64	Valid	1,969.7	30.8	1,395 ms	32.5 ms	64.0	0	59,071	30.0 s
Sustained decode	32K	128	Valid	2,827.6	22.1	2,687 ms	45.3 ms	128.0	0	84,868	30.0 s
Sustained decode	64K	1	Valid	118.3	118.4	201 ms	8.4 ms	1.0	0	3,550	30.0 s
Sustained decode	64K	2	Valid	216.6	108.4	328 ms	9.2 ms	2.0	0	6,497	30.0 s
Sustained decode	64K	4	Valid	369.5	92.6	318 ms	10.8 ms	4.0	0	11,084	30.0 s
Sustained decode	64K	8	Valid	596.5	74.8	468 ms	13.4 ms	8.0	0	17,897	30.0 s
Sustained decode	64K	16	Valid	921.1	57.8	793 ms	17.3 ms	16.0	0	27,632	30.0 s
Sustained decode	64K	32	Valid	1,338.8	42.1	1,398 ms	23.8 ms	32.0	0	40,160	30.0 s
Sustained decode	64K	64	Valid	1,946.8	30.6	2,622 ms	32.7 ms	64.0	0	58,367	30.0 s
Sustained decode	64K	128	Valid	2,783.1	21.8	5,116 ms	45.9 ms	128.0	0	83,457	30.0 s
Sustained decode	128K	1	Valid	117.6	117.8	305 ms	8.5 ms	1.0	0	3,526	30.0 s
Sustained decode	128K	2	Valid	214.4	107.3	444 ms	9.3 ms	2.0	0	6,432	30.0 s
Sustained decode	128K	4	Valid	370.5	92.8	650 ms	10.8 ms	4.0	0	11,112	30.0 s
Sustained decode	128K	8	Valid	587.7	73.8	955 ms	13.5 ms	8.0	0	17,624	30.0 s
Sustained decode	128K	16	Valid	895.2	56.1	1,605 ms	17.8 ms	16.0	0	26,848	30.0 s
Sustained decode	128K	32	Valid	1,302.4	41.0	3,275 ms	24.4 ms	32.0	0	39,071	30.0 s
Sustained decode	128K	64	Valid	1,879.4	29.9	6,184 ms	33.5 ms	64.0	0	56,383	30.0 s
Sustained decode	128K	128	Valid	2,708.9	21.7	11,959 ms	46.1 ms	128.0	0	81,279	30.0 s
Sustained decode	256K	1	Valid	115.8	116.0	598 ms	8.6 ms	1.0	0	3,472	30.0 s
Sustained decode	256K	2	Valid	211.0	105.9	919 ms	9.4 ms	2.0	0	6,328	30.0 s
Sustained decode	256K	4	Valid	363.7	91.0	1,336 ms	11.0 ms	4.0	0	10,908	30.0 s
Sustained decode	256K	8	Valid	581.6	72.8	2,001 ms	13.7 ms	8.0	0	17,448	30.0 s
Sustained decode	256K	16	Valid	869.6	54.4	3,396 ms	18.4 ms	16.0	0	26,081	30.0 s
Sustained decode	256K	32	Valid	1,250.5	39.2	6,263 ms	25.5 ms	32.0	0	37,505	30.0 s
Sustained decode	256K	64	Valid	1,851.0	29.1	11,836 ms	34.4 ms	64.0	0	55,490	30.0 s
Sustained decode	256K	128	Valid	2,565.7	20.3	23,093 ms	49.1 ms	128.0	0	76,930	30.0 s
Sustained decode	512K	1	Valid	112.1	112.5	1,380 ms	8.9 ms	1.0	0	3,362	30.0 s
Sustained decode	512K	2	Valid	203.9	102.4	1,984 ms	9.8 ms	2.0	0	6,116	30.0 s
Sustained decode	512K	4	Valid	349.8	87.9	3,165 ms	11.4 ms	4.0	0	10,492	30.0 s
Sustained decode	512K	8	Valid	549.0	69.0	4,302 ms	14.5 ms	8.0	0	16,464	30.0 s
Sustained decode	512K	16	Valid	826.2	51.8	7,927 ms	19.3 ms	16.0	0	24,783	30.0 s
Sustained decode	512K	32	Valid	1,168.3	36.7	15,974 ms	27.3 ms	32.0	0	35,040	30.0 s
Sustained decode	512K	64	Valid	1,652.6	26.5	33,009 ms	37.7 ms	64.0	0	49,536	30.0 s
Sustained decode	512K	128	Valid	2,253.9	18.9	64,664 ms	52.9 ms	127.9	2	67,536	30.0 s
Sustained decode	1019K	1	Valid	108.0	108.7	2,801 ms	9.2 ms	1.0	0	3,239	30.0 s
Sustained decode	1019K	2	Valid	198.2	99.8	4,168 ms	10.0 ms	2.0	0	5,944	30.0 s
Sustained decode	1019K	4	Valid	335.4	84.5	6,136 ms	11.8 ms	4.0	0	10,060	30.0 s
Sustained decode	1019K	8	Valid	524.8	66.0	10,433 ms	15.1 ms	8.0	0	15,736	30.0 s
Sustained decode	1019K	16	Valid	785.1	49.3	20,453 ms	20.3 ms	16.0	0	23,552	30.0 s
Sustained decode	1019K	32	Valid	1,053.9	33.8	37,007 ms	29.6 ms	31.9	1	31,596	30.0 s
Sustained decode	1019K	64	Requests queued	1,378.4	22.5	50,884 ms	44.4 ms	63.4	19	41,309	30.0 s
Sustained decode	1019K	128	Requests queued	1,476.0	19.5	105,620 ms	51.2 ms	83.1	15	44,282	30.0 s

Requests queued: the server could not run all requests at once, so some waited in a queue.