Speed test matrix

DeepSeek-V4-Flash-NVFP4 on H200 NVL 141GB ×2

sglang · 0.5.16 · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 19, 2026, 17:55 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.16
Quantization: NVFP4+FP8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2

Run

Served model: deepseek
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nvidia-deepseek-sglang/bin/sglang" serve --model-path nvidia/DeepSeek-V4-Flash-NVFP4 --served-model-name deepseek --trust-remote-code --tp 2 --reasoning-parser deepseek-v4 --tool-call-parser deepseekv4 --context-length 1048576 --mem-fraction-static 0.874248 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --disable-cuda-graph --max-running-requests 16 --moe-runner-backend marlin

Prefill

prompt tok/s by context length

8K context13.6K601 ms TTFT

16K context14.7K1,105 ms TTFT

32K context14.9K2,173 ms TTFT

64K context14.8K4,363 ms TTFT

128K context13.9K9,308 ms TTFT

256K context12.2K21,194 ms TTFT

512K context9,68353,240 ms TTFT

1019K context6,902148,645 ms TTFT

1024K context6,801151,585 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	17.0173 ms TTFT	33.4201 ms TTFT	66.8438 ms TTFT	133.2307 ms TTFT	266.0758 ms TTFT	Not run	Not run	Not run
16K	16.8183 ms TTFT	33.0228 ms TTFT	65.1281 ms TTFT	129.9340 ms TTFT	262.6421 ms TTFT	Not run	Not run	Not run
32K	16.8242 ms TTFT	33.1254 ms TTFT	66.0330 ms TTFT	132.4443 ms TTFT	262.7777 ms TTFT	Not run	Not run	Over capacity
64K	16.9271 ms TTFT	33.1335 ms TTFT	66.2428 ms TTFT	131.8840 ms TTFT	263.81,558 ms TTFT	Not run	Not run	Over capacity
128K	16.8377 ms TTFT	33.0501 ms TTFT	66.4904 ms TTFT	132.31,655 ms TTFT	264.33,167 ms TTFT	Not run	Over capacity	Over capacity
256K	17.1595 ms TTFT	33.5925 ms TTFT	66.41,582 ms TTFT	132.83,211 ms TTFT	265.26,256 ms TTFT	Over capacity	Over capacity	Over capacity
512K	16.91,016 ms TTFT	33.31,749 ms TTFT	67.03,308 ms TTFT	133.86,327 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
1019K	16.92,075 ms TTFT	33.73,725 ms TTFT	67.17,035 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	13.6K	—	601 ms	—	—	1	—	—
Prefill	16K	1	Valid	14.7K	—	1,105 ms	—	—	1	—	—
Prefill	32K	1	Valid	14.9K	—	2,173 ms	—	—	1	—	—
Prefill	64K	1	Valid	14.8K	—	4,363 ms	—	—	1	—	—
Prefill	128K	1	Valid	13.9K	—	9,308 ms	—	—	1	—	—
Prefill	256K	1	Valid	12.2K	—	21,194 ms	—	—	1	—	—
Prefill	512K	1	Valid	9,683.0	—	53,240 ms	—	—	1	—	—
Prefill	1019K	1	Valid	6,902.0	—	148,645 ms	—	—	1	—	—
Prefill	1024K	1	Valid	6,801.0	—	151,585 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	17.0	17.0	173 ms	59.0 ms	1.0	0	510	30.0 s
Sustained decode	8K	2	Valid	33.4	16.7	201 ms	60.0 ms	2.0	0	1,000	30.0 s
Sustained decode	8K	4	Valid	66.8	16.7	438 ms	59.8 ms	4.0	0	2,003	30.0 s
Sustained decode	8K	8	Valid	133.2	16.6	307 ms	60.2 ms	8.0	0	3,992	30.0 s
Sustained decode	8K	16	Valid	266.0	16.6	758 ms	60.2 ms	16.0	0	7,968	30.0 s
Sustained decode	8K	32	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	16.8	16.8	183 ms	59.6 ms	1.0	0	504	30.0 s
Sustained decode	16K	2	Valid	33.0	16.5	228 ms	60.7 ms	2.0	0	990	30.0 s
Sustained decode	16K	4	Valid	65.1	16.2	281 ms	61.6 ms	4.0	0	1,948	29.9 s
Sustained decode	16K	8	Valid	129.9	16.2	340 ms	61.7 ms	8.0	0	3,896	30.0 s
Sustained decode	16K	16	Valid	262.6	16.3	421 ms	61.2 ms	16.0	0	7,872	30.0 s
Sustained decode	16K	32	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	16.8	16.7	242 ms	59.8 ms	1.0	0	503	30.0 s
Sustained decode	32K	2	Valid	33.1	16.5	254 ms	60.5 ms	2.0	0	994	30.0 s
Sustained decode	32K	4	Valid	66.0	16.4	330 ms	60.8 ms	4.0	0	1,978	30.0 s
Sustained decode	32K	8	Valid	132.4	16.5	443 ms	60.7 ms	8.0	0	3,970	30.0 s
Sustained decode	32K	16	Valid	262.7	16.4	777 ms	61.2 ms	16.0	0	7,872	30.0 s
Sustained decode	32K	32	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	16.9	16.9	271 ms	59.3 ms	1.0	0	505	29.9 s
Sustained decode	64K	2	Valid	33.1	16.5	335 ms	60.7 ms	2.0	0	990	29.9 s
Sustained decode	64K	4	Valid	66.2	16.5	428 ms	60.6 ms	4.0	0	1,984	30.0 s
Sustained decode	64K	8	Valid	131.8	16.5	840 ms	60.7 ms	8.0	0	3,952	30.0 s
Sustained decode	64K	16	Valid	263.8	16.5	1,558 ms	60.6 ms	16.0	0	7,902	30.0 s
Sustained decode	64K	32	Not run	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	16.8	16.8	377 ms	59.5 ms	1.0	0	505	30.0 s
Sustained decode	128K	2	Valid	33.0	16.5	501 ms	60.6 ms	2.0	0	990	30.0 s
Sustained decode	128K	4	Valid	66.4	16.6	904 ms	60.2 ms	4.0	0	1,990	30.0 s
Sustained decode	128K	8	Valid	132.3	16.6	1,655 ms	60.1 ms	8.0	0	3,968	30.0 s
Sustained decode	128K	16	Valid	264.3	16.8	3,167 ms	59.4 ms	16.0	0	7,920	30.0 s
Sustained decode	128K	32	Not run	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	17.1	17.1	595 ms	58.6 ms	1.0	0	513	30.0 s
Sustained decode	256K	2	Valid	33.5	16.8	925 ms	59.6 ms	2.0	0	1,006	30.0 s
Sustained decode	256K	4	Valid	66.4	16.7	1,582 ms	59.8 ms	4.0	0	1,988	30.0 s
Sustained decode	256K	8	Valid	132.8	17.0	3,211 ms	58.8 ms	8.0	0	3,985	30.0 s
Sustained decode	256K	16	Valid	265.2	17.6	6,256 ms	56.7 ms	16.0	0	7,951	30.0 s
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	16.9	16.9	1,016 ms	59.3 ms	1.0	0	506	30.0 s
Sustained decode	512K	2	Valid	33.3	16.7	1,749 ms	59.7 ms	2.0	0	996	29.9 s
Sustained decode	512K	4	Valid	67.0	17.2	3,308 ms	58.3 ms	4.0	0	2,008	30.0 s
Sustained decode	512K	8	Valid	133.8	17.8	6,327 ms	56.3 ms	8.0	0	4,009	30.0 s
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	16.9	16.9	2,075 ms	59.2 ms	1.0	0	506	29.9 s
Sustained decode	1019K	2	Valid	33.7	17.2	3,725 ms	58.3 ms	2.0	0	1,010	30.0 s
Sustained decode	1019K	4	Valid	67.1	17.7	7,035 ms	56.4 ms	4.0	0	2,012	30.0 s
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.