Speed test matrix

DeepSeek-V4-Flash-0731 on H200 NVL 141GB ×2

sglang · 0.5.16 · MXFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Aug 29, 2026, 07:24 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.16
Quantization: MXFP4+FP8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2

Run

Served model: deepseek
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-deepseek-sglang/bin/sglang" serve --model-path deepseek-ai/DeepSeek-V4-Flash-0731 --served-model-name deepseek --trust-remote-code --tp 2 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --context-length 1048576 --mem-fraction-static 0.808234 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --moe-runner-backend marlin --speculative-algorithm DSPARK

Prefill

prompt tok/s by context length

8K context14.2K575 ms TTFT

16K context14.8K1,098 ms TTFT

32K context15K2,158 ms TTFT

64K context14.7K4,399 ms TTFT

128K context13.7K9,402 ms TTFT

256K context12K21,494 ms TTFT

512K context9,56653,895 ms TTFT

1019K context6,776151,413 ms TTFT

1024K context6,687154,169 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	221.2124 ms TTFT	411.0167 ms TTFT	580.3165 ms TTFT	1,048.8319 ms TTFT	1,402.0516 ms TTFT	1,827.1753 ms TTFT	3,520.51,343 ms TTFT	3,837.52,228 ms TTFT
16K	387.9131 ms TTFT	446.3177 ms TTFT	599.0191 ms TTFT	986.9278 ms TTFT	1,313.8436 ms TTFT	2,034.7679 ms TTFT	2,997.81,289 ms TTFT	4,020.33,887 ms TTFT
32K	246.0167 ms TTFT	394.5201 ms TTFT	613.1233 ms TTFT	945.9432 ms TTFT	1,436.5770 ms TTFT	2,012.91,471 ms TTFT	2,979.72,870 ms TTFT	Over capacity
64K	311.3223 ms TTFT	363.8236 ms TTFT	588.7330 ms TTFT	946.4844 ms TTFT	1,475.71,534 ms TTFT	2,050.43,063 ms TTFT	Over capacity	Over capacity
128K	385.8320 ms TTFT	335.6379 ms TTFT	646.0844 ms TTFT	998.91,639 ms TTFT	1,540.03,188 ms TTFT	Over capacity	Over capacity	Over capacity
256K	267.0535 ms TTFT	390.1677 ms TTFT	645.7839 ms TTFT	889.63,070 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
512K	200.9963 ms TTFT	415.9Requests queued1,015 ms TTFT	487.23,231 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	170.02,050 ms TTFT	287.7Requests queued2,777 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	14.2K	—	575 ms	—	—	1	—	—
Prefill	16K	1	Valid	14.8K	—	1,098 ms	—	—	1	—	—
Prefill	32K	1	Valid	15K	—	2,158 ms	—	—	1	—	—
Prefill	64K	1	Valid	14.7K	—	4,399 ms	—	—	1	—	—
Prefill	128K	1	Valid	13.7K	—	9,402 ms	—	—	1	—	—
Prefill	256K	1	Valid	12K	—	21,494 ms	—	—	1	—	—
Prefill	512K	1	Valid	9,566.0	—	53,895 ms	—	—	1	—	—
Prefill	1019K	1	Valid	6,776.0	—	151,413 ms	—	—	1	—	—
Prefill	1024K	1	Valid	6,687.0	—	154,169 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	221.2	225.9	124 ms	4.4 ms	1.0	1	6,634	30.0 s
Sustained decode	8K	2	Valid	411.0	190.9	167 ms	5.3 ms	2.0	2	12,324	30.0 s
Sustained decode	8K	4	Valid	580.3	140.3	165 ms	7.1 ms	3.9	4	17,399	30.0 s
Sustained decode	8K	8	Valid	1,048.8	106.6	319 ms	9.4 ms	8.0	4	31,448	30.0 s
Sustained decode	8K	16	Valid	1,402.0	75.7	516 ms	13.2 ms	16.0	2	42,032	30.0 s
Sustained decode	8K	32	Valid	1,827.1	55.4	753 ms	18.1 ms	32.0	0	54,702	29.9 s
Sustained decode	8K	64	Valid	3,520.5	45.7	1,343 ms	21.9 ms	64.0	0	105,634	30.0 s
Sustained decode	8K	128	Valid	3,837.5	27.9	2,228 ms	35.9 ms	128.0	0	115,148	30.0 s
Sustained decode	16K	1	Valid	387.9	411.5	131 ms	2.4 ms	1.0	2	11,637	30.0 s
Sustained decode	16K	2	Valid	446.3	218.3	177 ms	4.6 ms	2.0	2	13,385	30.0 s
Sustained decode	16K	4	Valid	599.0	147.6	191 ms	6.8 ms	3.9	4	17,965	30.0 s
Sustained decode	16K	8	Valid	986.9	107.0	278 ms	9.3 ms	7.9	3	29,608	30.0 s
Sustained decode	16K	16	Valid	1,313.8	77.6	436 ms	12.9 ms	16.0	1	39,399	30.0 s
Sustained decode	16K	32	Valid	2,034.7	56.0	679 ms	17.9 ms	32.0	0	61,042	30.0 s
Sustained decode	16K	64	Valid	2,997.8	42.9	1,289 ms	23.3 ms	64.0	0	89,914	30.0 s
Sustained decode	16K	128	Valid	4,020.3	28.0	3,887 ms	35.7 ms	128.0	0	120,459	30.0 s
Sustained decode	32K	1	Valid	246.0	248.2	167 ms	4.0 ms	1.0	2	7,379	30.0 s
Sustained decode	32K	2	Valid	394.5	176.6	201 ms	5.7 ms	2.0	2	11,832	30.0 s
Sustained decode	32K	4	Valid	613.1	143.3	233 ms	7.0 ms	4.0	3	18,385	30.0 s
Sustained decode	32K	8	Valid	945.9	114.4	432 ms	8.7 ms	8.0	4	28,355	30.0 s
Sustained decode	32K	16	Valid	1,436.5	80.1	770 ms	12.5 ms	16.0	2	43,085	30.0 s
Sustained decode	32K	32	Valid	2,012.9	54.8	1,471 ms	18.2 ms	32.0	0	60,308	30.0 s
Sustained decode	32K	64	Valid	2,979.7	43.0	2,870 ms	23.2 ms	64.0	0	89,283	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	311.3	308.7	223 ms	3.2 ms	1.0	2	9,338	30.0 s
Sustained decode	64K	2	Valid	363.8	180.4	236 ms	5.5 ms	2.0	2	10,913	30.0 s
Sustained decode	64K	4	Valid	588.7	145.9	330 ms	6.9 ms	3.9	4	17,654	30.0 s
Sustained decode	64K	8	Valid	946.4	107.3	844 ms	9.3 ms	8.0	3	28,393	30.0 s
Sustained decode	64K	16	Valid	1,475.7	80.0	1,534 ms	12.5 ms	15.8	3	44,257	30.0 s
Sustained decode	64K	32	Valid	2,050.4	55.6	3,063 ms	18.0 ms	32.0	0	61,460	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	385.8	401.9	320 ms	2.5 ms	0.9	3	11,572	30.0 s
Sustained decode	128K	2	Valid	335.6	170.9	379 ms	5.9 ms	2.0	2	10,067	30.0 s
Sustained decode	128K	4	Valid	646.0	158.4	844 ms	6.3 ms	3.9	4	19,375	30.0 s
Sustained decode	128K	8	Valid	998.9	109.9	1,639 ms	9.1 ms	7.9	3	29,945	30.0 s
Sustained decode	128K	16	Valid	1,540.0	80.5	3,188 ms	12.4 ms	15.8	6	46,175	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	267.0	206.7	535 ms	4.8 ms	1.0	2	8,008	30.0 s
Sustained decode	256K	2	Valid	390.1	169.3	677 ms	5.9 ms	1.9	2	11,702	30.0 s
Sustained decode	256K	4	Valid	645.7	134.6	839 ms	7.4 ms	3.9	5	19,371	30.0 s
Sustained decode	256K	8	Valid	889.6	109.0	3,070 ms	9.2 ms	7.9	3	26,671	30.0 s
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	200.9	208.9	963 ms	4.8 ms	1.0	1	6,027	30.0 s
Sustained decode	512K	2	Requests queued	415.9	165.0	1,015 ms	6.1 ms	1.9	3	12,474	30.0 s
Sustained decode	512K	4	Valid	487.2	135.4	3,231 ms	7.4 ms	3.9	4	14,591	30.0 s
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	170.0	186.0	2,050 ms	5.4 ms	1.0	1	5,102	30.0 s
Sustained decode	1019K	2	Requests queued	287.7	157.0	2,777 ms	6.4 ms	1.9	2	8,627	30.0 s
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.