Speed test matrix

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 on H200 NVL 141GB

sglang · 0.5.6.post3.dev9335+gdd15fb57b · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 20, 2026, 00:40 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9335+gdd15fb57b
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: dd15fb57b5ef

Run

Served model: nemotron
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --served-model-name nemotron --trust-remote-code --tp 1 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.876541 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>

Prefill

prompt tok/s by context length

8K context26.9K304 ms TTFT

16K context31.7K512 ms TTFT

32K context31.1K1,040 ms TTFT

64K context28.6K2,253 ms TTFT

128K context24.5K5,250 ms TTFT

251K context19.3K13,044 ms TTFT

256K context19.3K13,350 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	291.783 ms TTFT	427.0108 ms TTFT	676.0134 ms TTFT	1,256.3184 ms TTFT	1,975.3340 ms TTFT	597.5608 ms TTFT	1,602.41,130 ms TTFT	2,384.22,200 ms TTFT
16K	286.5295 ms TTFT	287.1738 ms TTFT	708.7675 ms TTFT	1,205.0338 ms TTFT	2,086.3595 ms TTFT	1,301.71,105 ms TTFT	1,659.12,213 ms TTFT	2,923.14,526 ms TTFT
32K	281.9359 ms TTFT	141.4539 ms TTFT	631.5721 ms TTFT	1,070.3726 ms TTFT	675.81,437 ms TTFT	1,805.52,887 ms TTFT	2,539.95,660 ms TTFT	3,619.511,073 ms TTFT
64K	275.6497 ms TTFT	69.8677 ms TTFT	546.3862 ms TTFT	563.21,662 ms TTFT	1,115.13,198 ms TTFT	1,575.86,222 ms TTFT	2,179.612,539 ms TTFT	Over capacity
128K	264.4714 ms TTFT	312.3832 ms TTFT	517.91,594 ms TTFT	785.53,177 ms TTFT	1,367.46,242 ms TTFT	1,595.812,139 ms TTFT	Over capacity	Over capacity
251K	251.81,247 ms TTFT	385.21,569 ms TTFT	517.62,849 ms TTFT	677.95,944 ms TTFT	1,035.412,136 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	26.9K	—	304 ms	—	—	1	—	—
Prefill	16K	1	Valid	31.7K	—	512 ms	—	—	1	—	—
Prefill	32K	1	Valid	31.1K	—	1,040 ms	—	—	1	—	—
Prefill	64K	1	Valid	28.6K	—	2,253 ms	—	—	1	—	—
Prefill	128K	1	Valid	24.5K	—	5,250 ms	—	—	1	—	—
Prefill	251K	1	Valid	19.3K	—	13,044 ms	—	—	1	—	—
Prefill	256K	1	Valid	19.3K	—	13,350 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	291.7	294.6	83 ms	3.4 ms	1.0	2	7,561	25.9 s
Sustained decode	8K	2	Valid	427.0	748.6	108 ms	2.4 ms	2.5	2	8,076	18.9 s
Sustained decode	8K	4	Valid	676.0	1080.2	134 ms	2.6 ms	4.0	4	20,279	30.0 s
Sustained decode	8K	8	Valid	1,256.3	160.3	184 ms	6.2 ms	8.0	8	33,162	26.4 s
Sustained decode	8K	16	Valid	1,975.3	121.2	340 ms	8.3 ms	15.8	16	59,260	30.0 s
Sustained decode	8K	32	Valid	597.5	70.6	608 ms	14.2 ms	48.0	0	17,924	30.0 s
Sustained decode	8K	64	Valid	1,602.4	66.5	1,130 ms	15.0 ms	64.0	0	48,071	30.0 s
Sustained decode	8K	128	Valid	2,384.2	48.0	2,200 ms	20.8 ms	128.0	0	71,526	30.0 s
Sustained decode	16K	1	Valid	286.5	293.1	295 ms	3.4 ms	1.0	2	8,103	28.3 s
Sustained decode	16K	2	Valid	287.1	235.7	738 ms	4.3 ms	70.3	2	8,613	30.0 s
Sustained decode	16K	4	Valid	708.7	184.3	675 ms	5.4 ms	3.9	4	21,257	30.0 s
Sustained decode	16K	8	Valid	1,205.0	172.0	338 ms	5.8 ms	7.8	8	36,145	30.0 s
Sustained decode	16K	16	Valid	2,086.3	120.9	595 ms	8.3 ms	15.8	16	62,584	30.0 s
Sustained decode	16K	32	Valid	1,301.7	67.9	1,105 ms	14.7 ms	48.0	0	39,046	30.0 s
Sustained decode	16K	64	Valid	1,659.1	63.5	2,213 ms	15.7 ms	64.0	0	49,763	30.0 s
Sustained decode	16K	128	Valid	2,923.1	44.1	4,526 ms	22.7 ms	128.0	0	87,695	30.0 s
Sustained decode	32K	1	Valid	281.9	290.4	359 ms	3.4 ms	0.9	2	7,555	26.8 s
Sustained decode	32K	2	Valid	141.4	66.7	539 ms	15.0 ms	108.7	0	4,242	30.0 s
Sustained decode	32K	4	Valid	631.5	170.0	721 ms	5.9 ms	4.3	4	18,944	30.0 s
Sustained decode	32K	8	Valid	1,070.3	135.3	726 ms	7.4 ms	7.7	8	32,109	30.0 s
Sustained decode	32K	16	Valid	675.8	115.4	1,437 ms	8.7 ms	16.0	0	20,272	30.0 s
Sustained decode	32K	32	Valid	1,805.5	76.5	2,887 ms	13.1 ms	32.0	0	54,154	30.0 s
Sustained decode	32K	64	Valid	2,539.9	57.3	5,660 ms	17.4 ms	64.0	0	76,190	30.0 s
Sustained decode	32K	128	Valid	3,619.5	39.8	11,073 ms	25.1 ms	130.4	0	108,585	30.0 s
Sustained decode	64K	1	Valid	275.6	286.4	497 ms	3.5 ms	1.0	2	7,291	26.5 s
Sustained decode	64K	2	Valid	69.8	34.7	677 ms	28.8 ms	130.0	0	2,094	30.0 s
Sustained decode	64K	4	Valid	546.3	135.3	862 ms	7.4 ms	10.6	4	16,387	30.0 s
Sustained decode	64K	8	Valid	563.2	110.3	1,662 ms	9.1 ms	11.7	0	16,895	30.0 s
Sustained decode	64K	16	Valid	1,115.1	99.4	3,198 ms	10.1 ms	16.0	0	33,451	30.0 s
Sustained decode	64K	32	Valid	1,575.8	72.4	6,222 ms	13.8 ms	32.0	0	47,273	30.0 s
Sustained decode	64K	64	Valid	2,179.6	52.3	12,539 ms	19.1 ms	64.0	0	65,377	30.0 s
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	264.4	277.7	714 ms	3.6 ms	0.9	2	7,932	30.0 s
Sustained decode	128K	2	Valid	312.3	180.2	832 ms	5.9 ms	24.5	2	9,368	30.0 s
Sustained decode	128K	4	Valid	517.9	139.4	1,594 ms	7.2 ms	5.2	4	15,536	30.0 s
Sustained decode	128K	8	Valid	785.5	98.2	3,177 ms	10.2 ms	12.0	0	23,560	30.0 s
Sustained decode	128K	16	Valid	1,367.4	83.6	6,242 ms	12.0 ms	16.6	0	41,025	30.0 s
Sustained decode	128K	32	Valid	1,595.8	51.6	12,139 ms	19.4 ms	41.0	0	47,877	30.0 s
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	251.8	263.3	1,247 ms	3.8 ms	0.9	2	7,239	28.7 s
Sustained decode	251K	2	Valid	385.2	196.4	1,569 ms	5.2 ms	4.7	2	11,555	30.0 s
Sustained decode	251K	4	Valid	517.6	136.4	2,849 ms	7.3 ms	4.5	4	15,527	30.0 s
Sustained decode	251K	8	Valid	677.9	86.0	5,944 ms	11.6 ms	12.0	0	20,336	30.0 s
Sustained decode	251K	16	Valid	1,035.4	65.3	12,136 ms	15.3 ms	18.6	0	31,056	30.0 s
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.