Speed test matrix

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 on H200 NVL 141GB

vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 4, 2026, 20:40 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.26.1rc1.dev638+g52be12cfa
Quantization: NVFP4+FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 52be12cfac0c

Run

Served model: nemotron
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-nvidia-nemotron/bin/vllm" serve nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 --served-model-name nemotron --trust-remote-code --tensor-parallel-size 1 --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 262144 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --video-pruning-rate 0.5 --max-num-seqs 384 --allowed-local-media-path / --media-io-kwargs '{"video":{"fps":2,"num_frames":256}}' --kv-cache-dtype fp8

Prefill

prompt tok/s by context length

8K context34.1K240 ms TTFT

16K context34.2K475 ms TTFT

32K context35.5K910 ms TTFT

64K context33.7K1,911 ms TTFT

128K context29.1K4,420 ms TTFT

251K context22.4K11,273 ms TTFT

256K context21.9K11,711 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	355.6100 ms TTFT	721.3143 ms TTFT	1,180.3244 ms TTFT	2,006.5337 ms TTFT	2,542.2531 ms TTFT	4,286.1Requests queued867 ms TTFT	1,621.81,612 ms TTFT	2,867.33,015 ms TTFT
16K	388.9126 ms TTFT	628.1163 ms TTFT	1,121.2224 ms TTFT	1,589.3351 ms TTFT	2,376.2505 ms TTFT	3,969.4Requests queued782 ms TTFT	2,409.61,395 ms TTFT	3,639.22,582 ms TTFT
32K	380.6122 ms TTFT	658.6144 ms TTFT	1,100.4207 ms TTFT	1,248.1242 ms TTFT	2,387.2358 ms TTFT	1,809.3695 ms TTFT	2,572.61,150 ms TTFT	3,807.02,179 ms TTFT
64K	372.4158 ms TTFT	541.4230 ms TTFT	1,059.6265 ms TTFT	1,546.3308 ms TTFT	2,316.8531 ms TTFT	2,321.31,299 ms TTFT	2,763.12,182 ms TTFT	3,549.43,556 ms TTFT
128K	325.5295 ms TTFT	585.7353 ms TTFT	831.4415 ms TTFT	1,373.0435 ms TTFT	1,918.21,106 ms TTFT	2,578.62,596 ms TTFT	3,076.25,155 ms TTFT	3,631.910,293 ms TTFT
251K	295.3564 ms TTFT	531.7Requests queued606 ms TTFT	748.7609 ms TTFT	1,156.6638 ms TTFT	1,480.22,774 ms TTFT	1,895.55,118 ms TTFT	2,236.59,903 ms TTFT	2,480.319,906 ms TTFT

Requests queued: the server could not run all requests at once, so some waited in a queue.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	34.1K	—	240 ms	—	—	1	—	—
Prefill	16K	1	Valid	34.2K	—	475 ms	—	—	1	—	—
Prefill	32K	1	Valid	35.5K	—	910 ms	—	—	1	—	—
Prefill	64K	1	Valid	33.7K	—	1,911 ms	—	—	1	—	—
Prefill	128K	1	Valid	29.1K	—	4,420 ms	—	—	1	—	—
Prefill	251K	1	Valid	22.4K	—	11,273 ms	—	—	1	—	—
Prefill	256K	1	Valid	21.9K	—	11,711 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	355.6	366.4	100 ms	2.7 ms	1.0	2	7,147	20.1 s
Sustained decode	8K	2	Valid	721.3	2463.5	143 ms	0.4 ms	2.0	4	17,438	24.2 s
Sustained decode	8K	4	Valid	1,180.3	287.6	244 ms	3.5 ms	4.0	8	31,303	26.5 s
Sustained decode	8K	8	Valid	2,006.5	1497.4	337 ms	0.7 ms	8.0	16	60,106	30.0 s
Sustained decode	8K	16	Valid	2,542.2	191.0	531 ms	5.2 ms	16.0	16	76,265	30.0 s
Sustained decode	8K	32	Requests queued	4,286.1	132.9	867 ms	7.5 ms	31.0	32	128,589	30.0 s
Sustained decode	8K	64	Valid	1,621.8	84.7	1,612 ms	11.8 ms	64.0	0	48,645	30.0 s
Sustained decode	8K	128	Valid	2,867.3	53.0	3,015 ms	18.9 ms	128.0	0	86,012	30.0 s
Sustained decode	16K	1	Valid	388.9	3904.6	126 ms	0.3 ms	1.0	2	9,502	24.4 s
Sustained decode	16K	2	Valid	628.1	328.2	163 ms	3.0 ms	2.0	4	17,938	28.6 s
Sustained decode	16K	4	Valid	1,121.2	282.5	224 ms	3.5 ms	4.0	8	33,636	30.0 s
Sustained decode	16K	8	Valid	1,589.3	239.2	351 ms	4.2 ms	8.0	8	47,675	30.0 s
Sustained decode	16K	16	Valid	2,376.2	181.9	505 ms	5.5 ms	16.0	16	71,282	30.0 s
Sustained decode	16K	32	Requests queued	3,969.4	126.2	782 ms	7.9 ms	31.8	32	119,068	30.0 s
Sustained decode	16K	64	Valid	2,409.6	82.0	1,395 ms	12.2 ms	64.0	0	72,285	30.0 s
Sustained decode	16K	128	Valid	3,639.2	52.0	2,582 ms	19.2 ms	128.0	0	109,155	30.0 s
Sustained decode	32K	1	Valid	380.6	1215.4	122 ms	0.8 ms	1.0	2	11,416	30.0 s
Sustained decode	32K	2	Valid	658.6	319.8	144 ms	3.1 ms	2.0	4	19,757	30.0 s
Sustained decode	32K	4	Valid	1,100.4	277.8	207 ms	3.6 ms	4.0	8	33,013	30.0 s
Sustained decode	32K	8	Valid	1,248.1	221.0	242 ms	4.5 ms	7.9	8	37,439	30.0 s
Sustained decode	32K	16	Valid	2,387.2	165.0	358 ms	6.1 ms	16.0	16	71,608	30.0 s
Sustained decode	32K	32	Valid	1,809.3	114.0	695 ms	8.8 ms	32.0	0	54,280	30.0 s
Sustained decode	32K	64	Valid	2,572.6	77.7	1,150 ms	12.9 ms	64.0	0	77,169	30.0 s
Sustained decode	32K	128	Valid	3,807.0	49.1	2,179 ms	20.3 ms	128.0	0	114,229	30.0 s
Sustained decode	64K	1	Valid	372.4	346.5	158 ms	2.9 ms	1.0	2	11,170	30.0 s
Sustained decode	64K	2	Valid	541.4	312.7	230 ms	3.2 ms	2.0	4	16,242	30.0 s
Sustained decode	64K	4	Valid	1,059.6	261.0	265 ms	3.8 ms	3.9	8	31,789	30.0 s
Sustained decode	64K	8	Valid	1,546.3	199.4	308 ms	5.0 ms	8.0	8	46,383	30.0 s
Sustained decode	64K	16	Valid	2,316.8	148.5	531 ms	6.7 ms	15.8	16	69,495	30.0 s
Sustained decode	64K	32	Valid	2,321.3	99.1	1,299 ms	10.1 ms	32.0	0	69,625	30.0 s
Sustained decode	64K	64	Valid	2,763.1	65.7	2,182 ms	15.2 ms	64.0	0	82,863	30.0 s
Sustained decode	64K	128	Valid	3,549.4	40.2	3,556 ms	24.9 ms	128.0	0	106,513	30.0 s
Sustained decode	128K	1	Valid	325.5	332.5	295 ms	3.0 ms	1.0	2	9,766	30.0 s
Sustained decode	128K	2	Valid	585.7	300.1	353 ms	3.3 ms	2.0	4	17,571	30.0 s
Sustained decode	128K	4	Valid	831.4	235.3	415 ms	4.2 ms	4.0	4	24,939	30.0 s
Sustained decode	128K	8	Valid	1,373.0	173.4	435 ms	5.8 ms	7.9	8	41,185	30.0 s
Sustained decode	128K	16	Valid	1,918.2	119.9	1,106 ms	8.3 ms	16.0	7	57,516	30.0 s
Sustained decode	128K	32	Valid	2,578.6	79.2	2,596 ms	12.6 ms	32.0	0	77,342	30.0 s
Sustained decode	128K	64	Valid	3,076.2	50.9	5,155 ms	19.6 ms	64.0	0	92,290	30.0 s
Sustained decode	128K	128	Valid	3,631.9	29.4	10,293 ms	34.0 ms	128.0	0	108,949	30.0 s
Sustained decode	251K	1	Valid	295.3	307.8	564 ms	3.2 ms	1.0	2	8,860	30.0 s
Sustained decode	251K	2	Requests queued	531.7	278.3	606 ms	3.6 ms	1.9	4	15,951	30.0 s
Sustained decode	251K	4	Valid	748.7	208.5	609 ms	4.8 ms	3.9	4	22,462	30.0 s
Sustained decode	251K	8	Valid	1,156.6	147.3	638 ms	6.8 ms	7.9	8	34,696	30.0 s
Sustained decode	251K	16	Valid	1,480.2	98.0	2,774 ms	10.2 ms	16.0	0	44,399	30.0 s
Sustained decode	251K	32	Valid	1,895.5	59.9	5,118 ms	16.7 ms	32.0	0	56,860	30.0 s
Sustained decode	251K	64	Valid	2,236.5	35.7	9,903 ms	28.0 ms	64.0	0	67,065	30.0 s
Sustained decode	251K	128	Valid	2,480.3	20.2	19,906 ms	49.6 ms	128.0	0	74,429	30.0 s

Requests queued: the server could not run all requests at once, so some waited in a queue.