Speed test matrix

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on H200 NVL 141GB

sglang · 0.0.0.dev17157+gccfcdd93a · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 19, 2026, 22:39 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.0.0.dev17157+gccfcdd93a
Quantization: NVFP4+FP8
Backend: cuda
Repository: github.com/rystewart-nvidia/sglang
Build: cuda-13.2
Commit: ccfcdd93a8c2

Run

Served model: nemotron
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang-pr-33554/bin/sglang" serve --model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nemotron --trust-remote-code --tp 1 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 1048576 --mem-fraction-static 0.859957 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --mamba-ssm-dtype float16 --cuda-graph-max-bs-decode 16 --speculative-algorithm DFLASH --speculative-draft-model-path nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash --speculative-dflash-block-size 4

Prefill

prompt tok/s by context length

8K context26.1K314 ms TTFT

16K context29.1K559 ms TTFT

32K context28.7K1,124 ms TTFT

64K context27.5K2,346 ms TTFT

128K context24.5K5,265 ms TTFT

256K context19.5K13,183 ms TTFT

512K context13.9K37,060 ms TTFT

1019K context8,777116,644 ms TTFT

1024K context8,742117,689 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	379.363 ms TTFT	647.069 ms TTFT	1,055.968 ms TTFT	1,464.193 ms TTFT	948.5191 ms TTFT	1,867.3335 ms TTFT	Not run	Not run
16K	383.667 ms TTFT	349.6108 ms TTFT	976.589 ms TTFT	1,400.7147 ms TTFT	888.8345 ms TTFT	1,811.2655 ms TTFT	Not run	Not run
32K	376.4110 ms TTFT	330.8139 ms TTFT	954.5116 ms TTFT	1,418.7271 ms TTFT	979.3766 ms TTFT	1,888.11,425 ms TTFT	Not run	Not run
64K	393.7157 ms TTFT	212.7259 ms TTFT	925.5288 ms TTFT	1,372.8511 ms TTFT	945.61,475 ms TTFT	2,024.82,953 ms TTFT	Not run	Over capacity
128K	369.6241 ms TTFT	243.6440 ms TTFT	1,018.6394 ms TTFT	1,092.6941 ms TTFT	902.82,820 ms TTFT	1,755.45,749 ms TTFT	Over capacity	Over capacity
256K	324.7454 ms TTFT	196.9828 ms TTFT	880.4844 ms TTFT	1,057.03,133 ms TTFT	901.96,234 ms TTFT	Over capacity	Over capacity	Over capacity
512K	316.7841 ms TTFT	143.51,713 ms TTFT	618.92,736 ms TTFT	671.06,096 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
1019K	215.51,713 ms TTFT	288.22,474 ms TTFT	422.76,266 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	26.1K	—	314 ms	—	—	1	—	—
Prefill	16K	1	Valid	29.1K	—	559 ms	—	—	1	—	—
Prefill	32K	1	Valid	28.7K	—	1,124 ms	—	—	1	—	—
Prefill	64K	1	Valid	27.5K	—	2,346 ms	—	—	1	—	—
Prefill	128K	1	Valid	24.5K	—	5,265 ms	—	—	1	—	—
Prefill	256K	1	Valid	19.5K	—	13,183 ms	—	—	1	—	—
Prefill	512K	1	Valid	13.9K	—	37,060 ms	—	—	1	—	—
Prefill	1019K	1	Valid	8,777.0	—	116,644 ms	—	—	1	—	—
Prefill	1024K	1	Valid	8,742.0	—	117,689 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	379.3	396.3	63 ms	2.5 ms	1.0	3	11,380	30.0 s
Sustained decode	8K	2	Valid	647.0	322.1	69 ms	3.1 ms	2.0	4	19,408	30.0 s
Sustained decode	8K	4	Valid	1,055.9	252.9	68 ms	4.0 ms	3.9	8	31,674	30.0 s
Sustained decode	8K	8	Valid	1,464.1	190.3	93 ms	5.3 ms	9.5	8	43,915	30.0 s
Sustained decode	8K	16	Valid	948.5	59.4	191 ms	16.8 ms	22.0	0	28,447	30.0 s
Sustained decode	8K	32	Valid	1,867.3	56.1	335 ms	17.8 ms	35.6	3	56,022	30.0 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	383.6	393.5	67 ms	2.5 ms	1.0	3	11,508	30.0 s
Sustained decode	16K	2	Valid	349.6	220.4	108 ms	5.2 ms	16.8	2	10,488	30.0 s
Sustained decode	16K	4	Valid	976.5	248.5	89 ms	4.0 ms	4.1	7	29,289	30.0 s
Sustained decode	16K	8	Valid	1,400.7	181.9	147 ms	5.5 ms	9.3	8	42,012	30.0 s
Sustained decode	16K	16	Valid	888.8	56.7	345 ms	17.6 ms	22.9	0	26,650	30.0 s
Sustained decode	16K	32	Valid	1,811.2	55.9	655 ms	17.9 ms	38.1	0	54,337	30.0 s
Sustained decode	16K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	376.4	385.5	110 ms	2.6 ms	1.0	3	11,290	30.0 s
Sustained decode	32K	2	Valid	330.8	230.1	139 ms	5.3 ms	17.3	2	9,923	30.0 s
Sustained decode	32K	4	Valid	954.5	235.8	116 ms	4.2 ms	4.4	5	28,636	30.0 s
Sustained decode	32K	8	Valid	1,418.7	179.5	271 ms	5.6 ms	8.4	8	42,553	30.0 s
Sustained decode	32K	16	Valid	979.3	62.7	766 ms	15.9 ms	22.3	0	29,374	30.0 s
Sustained decode	32K	32	Valid	1,888.1	57.0	1,425 ms	17.6 ms	37.6	1	56,611	30.0 s
Sustained decode	32K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Not run	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	393.7	391.8	157 ms	2.6 ms	1.0	3	11,810	30.0 s
Sustained decode	64K	2	Valid	212.7	101.5	259 ms	9.9 ms	22.2	0	6,379	30.0 s
Sustained decode	64K	4	Valid	925.5	237.7	288 ms	4.2 ms	4.0	5	27,658	29.9 s
Sustained decode	64K	8	Valid	1,372.8	173.3	511 ms	5.8 ms	8.3	8	41,178	30.0 s
Sustained decode	64K	16	Valid	945.6	58.2	1,475 ms	17.2 ms	21.8	0	28,359	30.0 s
Sustained decode	64K	32	Valid	2,024.8	58.7	2,953 ms	17.0 ms	34.6	8	60,717	30.0 s
Sustained decode	64K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	369.6	369.2	241 ms	2.7 ms	1.0	3	11,019	29.8 s
Sustained decode	128K	2	Valid	243.6	115.1	440 ms	8.7 ms	14.0	0	7,309	30.0 s
Sustained decode	128K	4	Valid	1,018.6	255.0	394 ms	3.9 ms	3.9	8	30,384	29.8 s
Sustained decode	128K	8	Valid	1,092.6	156.3	941 ms	6.4 ms	10.2	8	32,752	30.0 s
Sustained decode	128K	16	Valid	902.8	60.1	2,820 ms	16.7 ms	24.0	0	27,085	30.0 s
Sustained decode	128K	32	Valid	1,755.4	52.0	5,749 ms	19.2 ms	35.8	2	52,634	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	324.7	335.1	454 ms	3.0 ms	1.0	2	9,741	30.0 s
Sustained decode	256K	2	Valid	196.9	94.5	828 ms	10.6 ms	17.5	0	5,907	30.0 s
Sustained decode	256K	4	Valid	880.4	223.2	844 ms	4.5 ms	3.9	5	26,411	30.0 s
Sustained decode	256K	8	Valid	1,057.0	135.2	3,133 ms	7.4 ms	8.6	6	31,185	29.5 s
Sustained decode	256K	16	Valid	901.9	63.9	6,234 ms	15.6 ms	21.8	0	27,043	30.0 s
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	316.7	293.8	841 ms	3.4 ms	0.9	2	9,499	30.0 s
Sustained decode	512K	2	Valid	143.5	71.4	1,713 ms	14.0 ms	15.6	0	4,304	30.0 s
Sustained decode	512K	4	Valid	618.9	168.3	2,736 ms	6.0 ms	3.9	4	18,563	30.0 s
Sustained decode	512K	8	Valid	671.0	86.9	6,096 ms	11.5 ms	10.4	0	20,127	30.0 s
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	215.5	231.0	1,713 ms	4.3 ms	1.0	1	6,465	30.0 s
Sustained decode	1019K	2	Valid	288.2	170.4	2,474 ms	6.0 ms	2.9	2	8,644	30.0 s
Sustained decode	1019K	4	Valid	422.7	107.2	6,266 ms	9.3 ms	4.8	1	12,679	30.0 s
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.