Speed test matrix

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 on H200 NVL 141GB

sglang · 0.5.6.post3.dev9335+gdd15fb57b · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 20, 2026, 01:21 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9335+gdd15fb57b
Quantization: FP8
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: dd15fb57b5ef

Run

Served model: nemotron
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-FP8 --served-model-name nemotron --trust-remote-code --tp 1 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 262144 --mem-fraction-static 0.871010 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3

Prefill

prompt tok/s by context length

8K context34K241 ms TTFT

16K context45.6K356 ms TTFT

32K context44.5K726 ms TTFT

64K context40.6K1,588 ms TTFT

128K context34.3K3,756 ms TTFT

251K context25.5K9,878 ms TTFT

256K context25.4K10,115 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	367.8102 ms TTFT	648.6152 ms TTFT	1,209.7205 ms TTFT	1,147.0248 ms TTFT	2,467.6405 ms TTFT	4,355.1702 ms TTFT	2,654.81,344 ms TTFT	3,393.22,556 ms TTFT
16K	386.4224 ms TTFT	584.1309 ms TTFT	1,021.5442 ms TTFT	1,311.7457 ms TTFT	2,320.1725 ms TTFT	3,992.31,348 ms TTFT	2,621.02,748 ms TTFT	4,136.25,600 ms TTFT
32K	358.0279 ms TTFT	614.7461 ms TTFT	741.7489 ms TTFT	1,411.4799 ms TTFT	2,447.0909 ms TTFT	2,475.43,026 ms TTFT	2,854.16,334 ms TTFT	4,029.612,422 ms TTFT
64K	349.1399 ms TTFT	399.6530 ms TTFT	611.6806 ms TTFT	1,362.31,549 ms TTFT	1,933.23,000 ms TTFT	1,769.95,927 ms TTFT	3,598.411,848 ms TTFT	4,377.623,338 ms TTFT
128K	330.4680 ms TTFT	43.8992 ms TTFT	840.31,339 ms TTFT	1,205.82,842 ms TTFT	1,494.65,841 ms TTFT	1,375.711,744 ms TTFT	2,770.923,724 ms TTFT	Over capacity
251K	301.21,192 ms TTFT	349.91,639 ms TTFT	657.52,721 ms TTFT	914.46,209 ms TTFT	1,016.312,264 ms TTFT	1,445.225,013 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	34K	—	241 ms	—	—	1	—	—
Prefill	16K	1	Valid	45.6K	—	356 ms	—	—	1	—	—
Prefill	32K	1	Valid	44.5K	—	726 ms	—	—	1	—	—
Prefill	64K	1	Valid	40.6K	—	1,588 ms	—	—	1	—	—
Prefill	128K	1	Valid	34.3K	—	3,756 ms	—	—	1	—	—
Prefill	251K	1	Valid	25.5K	—	9,878 ms	—	—	1	—	—
Prefill	256K	1	Valid	25.4K	—	10,115 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	367.8	3548.8	102 ms	0.3 ms	1.0	3	11,008	29.9 s
Sustained decode	8K	2	Valid	648.6	514.6	152 ms	2.3 ms	2.0	4	16,033	24.7 s
Sustained decode	8K	4	Valid	1,209.7	2011.6	205 ms	0.5 ms	4.0	8	32,984	27.3 s
Sustained decode	8K	8	Valid	1,147.0	216.2	248 ms	4.6 ms	8.0	8	34,411	30.0 s
Sustained decode	8K	16	Valid	2,467.6	176.0	405 ms	5.7 ms	15.7	16	74,025	30.0 s
Sustained decode	8K	32	Valid	4,355.1	132.3	702 ms	7.6 ms	31.7	32	130,638	30.0 s
Sustained decode	8K	64	Valid	2,654.8	91.7	1,344 ms	10.9 ms	64.0	0	79,626	30.0 s
Sustained decode	8K	128	Valid	3,393.2	65.7	2,556 ms	15.2 ms	128.0	0	101,798	30.0 s
Sustained decode	16K	1	Valid	386.4	368.3	224 ms	2.7 ms	1.0	2	8,248	21.3 s
Sustained decode	16K	2	Valid	584.1	321.7	309 ms	3.1 ms	1.9	4	17,523	30.0 s
Sustained decode	16K	4	Valid	1,021.5	257.6	442 ms	3.9 ms	3.9	8	30,645	30.0 s
Sustained decode	16K	8	Valid	1,311.7	199.2	457 ms	5.0 ms	10.0	8	39,347	30.0 s
Sustained decode	16K	16	Valid	2,320.1	152.0	725 ms	6.6 ms	16.0	16	69,594	30.0 s
Sustained decode	16K	32	Valid	3,992.3	127.3	1,348 ms	7.9 ms	31.2	32	119,739	30.0 s
Sustained decode	16K	64	Valid	2,621.0	66.7	2,748 ms	15.0 ms	96.0	0	78,618	30.0 s
Sustained decode	16K	128	Valid	4,136.2	60.3	5,600 ms	16.6 ms	128.0	0	124,090	30.0 s
Sustained decode	32K	1	Valid	358.0	364.5	279 ms	2.7 ms	1.0	2	10,740	30.0 s
Sustained decode	32K	2	Valid	614.7	317.5	461 ms	3.1 ms	2.0	4	18,441	30.0 s
Sustained decode	32K	4	Valid	741.7	243.3	489 ms	4.1 ms	3.9	4	22,250	30.0 s
Sustained decode	32K	8	Valid	1,411.4	194.5	799 ms	5.1 ms	7.8	8	42,342	30.0 s
Sustained decode	32K	16	Valid	2,447.0	158.4	909 ms	6.3 ms	15.9	16	73,406	30.0 s
Sustained decode	32K	32	Valid	2,475.4	108.7	3,026 ms	9.2 ms	32.0	0	74,262	30.0 s
Sustained decode	32K	64	Valid	2,854.1	82.6	6,334 ms	12.1 ms	64.0	0	85,626	30.0 s
Sustained decode	32K	128	Valid	4,029.6	57.6	12,422 ms	17.4 ms	128.0	0	120,889	30.0 s
Sustained decode	64K	1	Valid	349.1	358.9	399 ms	2.8 ms	1.0	2	10,474	30.0 s
Sustained decode	64K	2	Valid	399.6	235.4	530 ms	4.4 ms	18.0	2	11,989	30.0 s
Sustained decode	64K	4	Valid	611.6	228.9	806 ms	4.4 ms	3.9	4	18,346	30.0 s
Sustained decode	64K	8	Valid	1,362.3	180.4	1,549 ms	5.5 ms	7.8	8	40,865	30.0 s
Sustained decode	64K	16	Valid	1,933.2	562.9	3,000 ms	1.8 ms	15.3	16	58,587	30.3 s
Sustained decode	64K	32	Valid	1,769.9	67.6	5,927 ms	14.8 ms	48.0	0	53,098	30.0 s
Sustained decode	64K	64	Valid	3,598.4	62.6	11,848 ms	16.0 ms	64.0	0	107,964	30.0 s
Sustained decode	64K	128	Valid	4,377.6	41.0	23,338 ms	24.4 ms	128.0	0	131,309	30.0 s
Sustained decode	128K	1	Valid	330.4	347.1	680 ms	2.9 ms	0.9	2	9,912	30.0 s
Sustained decode	128K	2	Valid	43.8	37.7	992 ms	26.5 ms	130.0	0	1,314	30.0 s
Sustained decode	128K	4	Valid	840.3	217.4	1,339 ms	4.6 ms	3.9	4	25,207	30.0 s
Sustained decode	128K	8	Valid	1,205.8	161.4	2,842 ms	6.2 ms	7.7	8	36,167	30.0 s
Sustained decode	128K	16	Valid	1,494.6	116.0	5,841 ms	8.6 ms	16.0	2	44,690	29.9 s
Sustained decode	128K	32	Valid	1,375.7	57.0	11,744 ms	17.6 ms	34.0	0	41,272	30.0 s
Sustained decode	128K	64	Valid	2,770.9	55.0	23,724 ms	18.2 ms	64.0	0	83,136	30.0 s
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	301.2	326.7	1,192 ms	3.1 ms	0.9	2	9,035	30.0 s
Sustained decode	251K	2	Valid	349.9	210.6	1,639 ms	5.2 ms	18.9	2	10,496	30.0 s
Sustained decode	251K	4	Valid	657.5	180.0	2,721 ms	5.6 ms	4.4	4	19,725	30.0 s
Sustained decode	251K	8	Valid	914.4	119.9	6,209 ms	8.3 ms	9.4	4	28,128	30.8 s
Sustained decode	251K	16	Valid	1,016.3	80.0	12,264 ms	12.5 ms	20.0	0	30,491	30.0 s
Sustained decode	251K	32	Valid	1,445.2	67.4	25,013 ms	14.8 ms	32.0	0	43,344	30.0 s
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.