Speed test matrix

DeepSeek-V4.1-Flash-1017 on H200 NVL 141GB ×2

vllm · 0.32.0 · MXFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Oct 19, 2026, 16:26 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: vllm
Version: 0.32.0
Quantization: MXFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: deepseek-ai/DeepSeek-V4.1-Flash-1017
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_RUST_FRONTEND=0 "$HOME/env_h200-deepseek-v41-1017-vllm-5df63a720/bin/vllm" serve deepseek-ai/DeepSeek-V4.1-Flash-1017 --served-model-name deepseek-ai/DeepSeek-V4.1-Flash-1017 --trust-remote-code --tensor-parallel-size 2 --reasoning-parser deepseek_v41 --enable-auto-tool-choice --tool-call-parser deepseek_v41 --max-model-len 262144 --gpu-memory-utilization 0.93 --host 0.0.0.0 --port 8000 --api-key <redacted> --tokenizer-mode deepseek_v41 --mm-encoder-tp-mode data --engram-config '{"cpu_offload":true,"use_thp":false}' --offload-backend uva --cpu-offload-gb 23 --cpu-offload-params w13_weight w13_weight_scale w2_weight_scale

Prefill

prompt tok/s by context length

8K context2,7093,023 ms TTFT

16K context2,7595,875 ms TTFT

32K context2,82211,430 ms TTFT

64K context2,84422,626 ms TTFT

128K context2,83545,329 ms TTFT

251K context2,78890,353 ms TTFT

256K context2,77792,488 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	43.6741 ms TTFT	56.61,122 ms TTFT	71.21,677 ms TTFT	89.71,853 ms TTFT	118.82,023 ms TTFT	163.42,943 ms TTFT	224.53,746 ms TTFT	303.76,541 ms TTFT
16K	43.5692 ms TTFT	56.21,023 ms TTFT	71.91,894 ms TTFT	88.41,623 ms TTFT	118.01,939 ms TTFT	161.92,522 ms TTFT	224.63,197 ms TTFT	Over capacity
32K	43.6793 ms TTFT	56.61,165 ms TTFT	69.11,721 ms TTFT	88.91,955 ms TTFT	118.32,200 ms TTFT	161.22,852 ms TTFT	Over capacity	Over capacity
64K	43.5799 ms TTFT	55.91,214 ms TTFT	69.91,710 ms TTFT	89.71,917 ms TTFT	118.22,235 ms TTFT	Over capacity	Over capacity	Over capacity
128K	43.5843 ms TTFT	55.71,182 ms TTFT	69.52,053 ms TTFT	90.22,525 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
251K	43.31,258 ms TTFT	57.11,624 ms TTFT	71.32,334 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	2,709.0	—	3,023 ms	—	—	1	—	—
Prefill	16K	1	Valid	2,759.0	—	5,875 ms	—	—	1	—	—
Prefill	32K	1	Valid	2,822.0	—	11,430 ms	—	—	1	—	—
Prefill	64K	1	Valid	2,844.0	—	22,626 ms	—	—	1	—	—
Prefill	128K	1	Valid	2,835.0	—	45,329 ms	—	—	1	—	—
Prefill	251K	1	Valid	2,788.0	—	90,353 ms	—	—	1	—	—
Prefill	256K	1	Valid	2,777.0	—	92,488 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	43.6	43.6	741 ms	22.9 ms	1.0	0	1,308	30.0 s
Sustained decode	8K	2	Valid	56.6	28.0	1,122 ms	35.7 ms	2.0	0	1,696	30.0 s
Sustained decode	8K	4	Valid	71.2	17.9	1,677 ms	55.8 ms	4.0	0	2,136	30.0 s
Sustained decode	8K	8	Valid	89.7	11.5	1,853 ms	87.1 ms	8.0	0	2,688	30.0 s
Sustained decode	8K	16	Valid	118.8	7.7	2,023 ms	129.2 ms	16.0	0	3,551	29.9 s
Sustained decode	8K	32	Valid	163.4	5.4	2,943 ms	185.6 ms	32.0	0	4,894	30.0 s
Sustained decode	8K	64	Valid	224.5	3.5	3,746 ms	285.8 ms	64.0	0	6,668	29.7 s
Sustained decode	8K	128	Valid	303.7	2.3	6,541 ms	439.6 ms	128.0	0	9,086	29.9 s
Sustained decode	16K	1	Valid	43.5	43.6	692 ms	22.9 ms	1.0	0	1,306	30.0 s
Sustained decode	16K	2	Valid	56.2	28.0	1,023 ms	35.7 ms	2.0	0	1,683	30.0 s
Sustained decode	16K	4	Valid	71.9	18.2	1,894 ms	55.1 ms	4.0	0	2,155	30.0 s
Sustained decode	16K	8	Valid	88.4	11.4	1,623 ms	88.0 ms	8.0	0	2,648	30.0 s
Sustained decode	16K	16	Valid	118.0	7.7	1,939 ms	129.8 ms	16.0	0	3,536	30.0 s
Sustained decode	16K	32	Valid	161.9	5.3	2,522 ms	188.1 ms	32.0	0	4,833	29.9 s
Sustained decode	16K	64	Valid	224.6	3.5	3,197 ms	287.5 ms	64.0	0	6,727	29.9 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	43.6	43.6	793 ms	23.0 ms	1.0	0	1,306	30.0 s
Sustained decode	32K	2	Valid	56.6	28.0	1,165 ms	35.7 ms	2.0	0	1,696	30.0 s
Sustained decode	32K	4	Valid	69.1	17.7	1,721 ms	56.7 ms	4.0	0	2,069	30.0 s
Sustained decode	32K	8	Valid	88.9	11.4	1,955 ms	87.7 ms	8.0	0	2,664	30.0 s
Sustained decode	32K	16	Valid	118.3	7.7	2,200 ms	130.0 ms	16.0	0	3,536	29.9 s
Sustained decode	32K	32	Valid	161.2	5.0	2,852 ms	198.5 ms	32.0	0	4,832	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	43.5	43.5	799 ms	23.0 ms	1.0	0	1,305	30.0 s
Sustained decode	64K	2	Valid	55.9	27.8	1,214 ms	36.0 ms	2.0	0	1,676	30.0 s
Sustained decode	64K	4	Valid	69.9	17.6	1,710 ms	56.8 ms	4.0	0	2,097	30.0 s
Sustained decode	64K	8	Valid	89.7	11.6	1,917 ms	86.5 ms	8.0	0	2,687	30.0 s
Sustained decode	64K	16	Valid	118.2	7.5	2,235 ms	134.0 ms	16.0	0	3,536	29.9 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	43.5	43.5	843 ms	23.0 ms	1.0	0	1,303	30.0 s
Sustained decode	128K	2	Valid	55.7	27.7	1,182 ms	36.0 ms	2.0	0	1,672	30.0 s
Sustained decode	128K	4	Valid	69.5	17.7	2,053 ms	56.6 ms	4.0	0	2,084	30.0 s
Sustained decode	128K	8	Valid	90.2	11.3	2,525 ms	88.2 ms	8.0	0	2,704	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	43.3	43.4	1,258 ms	23.0 ms	1.0	0	1,299	30.0 s
Sustained decode	251K	2	Valid	57.1	28.4	1,624 ms	35.2 ms	2.0	0	1,713	30.0 s
Sustained decode	251K	4	Valid	71.3	17.7	2,334 ms	56.6 ms	4.0	0	2,140	30.0 s
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.