Speed test matrix

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 4, 2026, 19:09 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: nemotron
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nvidia-nemotron/bin/vllm" serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --video-pruning-rate 0.5 --max-num-seqs 384 --allowed-local-media-path / --media-io-kwargs '{"video":{"fps":2,"num_frames":256}}'

Prefill

prompt tok/s by context length

8K context45.3K181 ms TTFT

16K context42.7K380 ms TTFT

32K context44.3K728 ms TTFT

64K context42.5K1,516 ms TTFT

128K context34.2K3,760 ms TTFT

251K context24.4K10,346 ms TTFT

256K context24K10,700 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	309.779 ms TTFT	591.4121 ms TTFT	647.2144 ms TTFT	1,156.2204 ms TTFT	2,139.5303 ms TTFT	1,285.5564 ms TTFT	1,419.6834 ms TTFT	2,257.01,312 ms TTFT
16K	308.188 ms TTFT	529.6104 ms TTFT	663.2128 ms TTFT	1,357.9141 ms TTFT	2,101.8286 ms TTFT	1,011.2505 ms TTFT	1,932.1683 ms TTFT	2,405.91,310 ms TTFT
32K	304.6104 ms TTFT	543.8131 ms TTFT	613.3197 ms TTFT	1,240.3254 ms TTFT	2,142.5241 ms TTFT	1,210.4643 ms TTFT	1,714.01,156 ms TTFT	2,467.21,822 ms TTFT
64K	299.3139 ms TTFT	496.5182 ms TTFT	786.0189 ms TTFT	1,121.7246 ms TTFT	1,243.2770 ms TTFT	1,486.71,401 ms TTFT	2,213.62,423 ms TTFT	3,462.03,930 ms TTFT
128K	285.7273 ms TTFT	365.3335 ms TTFT	695.5339 ms TTFT	1,006.1483 ms TTFT	1,442.81,483 ms TTFT	1,971.02,714 ms TTFT	2,536.85,303 ms TTFT	Over capacity
251K	264.0484 ms TTFT	447.0586 ms TTFT	606.0588 ms TTFT	949.31,294 ms TTFT	1,277.12,705 ms TTFT	1,599.35,009 ms TTFT	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	45.3K	—	181 ms	—	—	1	—	—
Prefill	16K	1	Valid	42.7K	—	380 ms	—	—	1	—	—
Prefill	32K	1	Valid	44.3K	—	728 ms	—	—	1	—	—
Prefill	64K	1	Valid	42.5K	—	1,516 ms	—	—	1	—	—
Prefill	128K	1	Valid	34.2K	—	3,760 ms	—	—	1	—	—
Prefill	251K	1	Valid	24.4K	—	10,346 ms	—	—	1	—	—
Prefill	256K	1	Valid	24K	—	10,700 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	309.7	1006.7	79 ms	1.0 ms	1.0	2	7,551	24.4 s
Sustained decode	8K	2	Valid	591.4	2789.1	121 ms	0.4 ms	2.0	4	17,750	30.0 s
Sustained decode	8K	4	Valid	647.2	193.5	144 ms	5.2 ms	4.0	4	19,415	30.0 s
Sustained decode	8K	8	Valid	1,156.2	190.2	204 ms	5.3 ms	8.0	8	34,685	30.0 s
Sustained decode	8K	16	Valid	2,139.5	129.4	303 ms	7.7 ms	16.0	16	64,187	30.0 s
Sustained decode	8K	32	Valid	1,285.5	91.7	564 ms	10.9 ms	32.0	0	38,558	30.0 s
Sustained decode	8K	64	Valid	1,419.6	68.3	834 ms	14.6 ms	64.0	0	42,585	30.0 s
Sustained decode	8K	128	Valid	2,257.0	50.0	1,312 ms	20.0 ms	128.0	0	67,701	30.0 s
Sustained decode	16K	1	Valid	308.1	316.1	88 ms	3.2 ms	1.0	2	7,338	23.8 s
Sustained decode	16K	2	Valid	529.6	960.5	104 ms	1.1 ms	2.0	4	15,886	30.0 s
Sustained decode	16K	4	Valid	663.2	193.7	128 ms	5.2 ms	4.0	4	19,897	30.0 s
Sustained decode	16K	8	Valid	1,357.9	175.4	141 ms	5.8 ms	8.0	8	40,737	30.0 s
Sustained decode	16K	16	Valid	2,101.8	123.2	286 ms	8.1 ms	15.7	16	63,043	30.0 s
Sustained decode	16K	32	Valid	1,011.2	87.8	505 ms	11.4 ms	32.0	0	30,334	30.0 s
Sustained decode	16K	64	Valid	1,932.1	64.0	683 ms	15.6 ms	64.0	0	57,960	30.0 s
Sustained decode	16K	128	Valid	2,405.9	50.1	1,310 ms	19.9 ms	128.0	0	72,153	30.0 s
Sustained decode	32K	1	Valid	304.6	307.2	104 ms	3.3 ms	1.0	2	7,483	24.6 s
Sustained decode	32K	2	Valid	543.8	541.9	131 ms	2.5 ms	1.9	4	15,506	28.5 s
Sustained decode	32K	4	Valid	613.3	371.5	197 ms	3.5 ms	4.0	4	18,394	30.0 s
Sustained decode	32K	8	Valid	1,240.3	159.8	254 ms	6.3 ms	8.0	8	37,205	30.0 s
Sustained decode	32K	16	Valid	2,142.5	122.7	241 ms	8.2 ms	15.9	16	64,279	30.0 s
Sustained decode	32K	32	Valid	1,210.4	81.7	643 ms	12.2 ms	32.0	0	36,306	30.0 s
Sustained decode	32K	64	Valid	1,714.0	59.6	1,156 ms	16.8 ms	64.0	0	51,405	30.0 s
Sustained decode	32K	128	Valid	2,467.2	43.6	1,822 ms	22.9 ms	128.0	0	74,013	30.0 s
Sustained decode	64K	1	Valid	299.3	302.1	139 ms	3.3 ms	1.0	2	8,978	30.0 s
Sustained decode	64K	2	Valid	496.5	260.5	182 ms	3.8 ms	2.0	4	14,896	30.0 s
Sustained decode	64K	4	Valid	786.0	185.9	189 ms	5.4 ms	4.0	4	23,579	30.0 s
Sustained decode	64K	8	Valid	1,121.7	147.2	246 ms	6.8 ms	8.0	8	33,647	30.0 s
Sustained decode	64K	16	Valid	1,243.2	109.5	770 ms	9.1 ms	16.0	0	37,294	30.0 s
Sustained decode	64K	32	Valid	1,486.7	77.5	1,401 ms	12.9 ms	32.0	0	44,592	30.0 s
Sustained decode	64K	64	Valid	2,213.6	54.2	2,423 ms	18.4 ms	64.0	0	66,395	30.0 s
Sustained decode	64K	128	Valid	3,462.0	35.8	3,930 ms	28.0 ms	128.0	0	103,802	30.0 s
Sustained decode	128K	1	Valid	285.7	310.2	273 ms	3.2 ms	1.0	2	7,501	26.3 s
Sustained decode	128K	2	Valid	365.3	242.4	335 ms	4.1 ms	2.0	2	10,959	30.0 s
Sustained decode	128K	4	Valid	695.5	175.5	339 ms	5.7 ms	3.9	4	20,865	30.0 s
Sustained decode	128K	8	Valid	1,006.1	127.8	483 ms	7.8 ms	7.9	8	30,178	30.0 s
Sustained decode	128K	16	Valid	1,442.8	89.9	1,483 ms	11.1 ms	16.0	0	43,278	30.0 s
Sustained decode	128K	32	Valid	1,971.0	62.6	2,714 ms	16.0 ms	32.0	0	59,115	30.0 s
Sustained decode	128K	64	Valid	2,536.8	41.7	5,303 ms	24.0 ms	64.0	0	76,103	30.0 s
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	264.0	272.8	484 ms	3.7 ms	1.0	2	7,921	30.0 s
Sustained decode	251K	2	Valid	447.0	228.6	586 ms	4.4 ms	1.9	2	13,409	30.0 s
Sustained decode	251K	4	Valid	606.0	163.0	588 ms	6.1 ms	3.9	4	18,178	30.0 s
Sustained decode	251K	8	Valid	949.3	119.5	1,294 ms	8.4 ms	8.0	3	28,477	30.0 s
Sustained decode	251K	16	Valid	1,277.1	79.9	2,705 ms	12.5 ms	16.0	0	38,301	30.0 s
Sustained decode	251K	32	Valid	1,599.3	51.7	5,009 ms	19.4 ms	32.0	0	47,986	30.0 s
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.