Speed test matrix

NVIDIA-Nemotron-3-Super-120B-A12B-BF16 on H200 NVL 141GB ×2

sglang · 0.5.6.post3.dev9335+gdd15fb57b · BF168 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 19, 2026, 19:56 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
GPU count: 2
VRAM: 282 GB (2 × 141 GB)
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9335+gdd15fb57b
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: dd15fb57b5ef

Run

Served model: nemotron
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 --served-model-name nemotron --trust-remote-code --tp 2 --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder --context-length 1048576 --mem-fraction-static 0.861854 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3

Prefill

prompt tok/s by context length

8K context9,758840 ms TTFT

16K context10.2K1,593 ms TTFT

32K context10.3K3,126 ms TTFT

64K context10.5K6,163 ms TTFT

128K context10.2K12,589 ms TTFT

256K context9,62126,744 ms TTFT

512K context8,54860,188 ms TTFT

1019K context6,931147,706 ms TTFT

1024K context6,904149,021 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	125.664 ms TTFT	222.597 ms TTFT	343.4142 ms TTFT	477.8Requests queued140 ms TTFT	486.7Requests queued55,531 ms TTFT	480.9Requests queued48,588 ms TTFT	483.0Requests queued48,409 ms TTFT	483.6Requests queued14,018 ms TTFT
16K	124.8209 ms TTFT	200.21,420 ms TTFT	335.9462 ms TTFT	473.4Requests queued457 ms TTFT	471.0Requests queued57,651 ms TTFT	481.1Requests queued45,433 ms TTFT	478.4Requests queued45,247 ms TTFT	Over capacity
32K	124.4213 ms TTFT	137.1310 ms TTFT	312.23,992 ms TTFT	455.7Requests queued5,181 ms TTFT	460.0Requests queued40,851 ms TTFT	448.7Requests queued61,389 ms TTFT	Over capacity	Over capacity
64K	123.7272 ms TTFT	189.3395 ms TTFT	321.9529 ms TTFT	451.8Requests queued852 ms TTFT	466.1Requests queued61,863 ms TTFT	Over capacity	Over capacity	Over capacity
128K	122.1407 ms TTFT	180.3559 ms TTFT	309.6841 ms TTFT	443.8Requests queued1,104 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
256K	122.0517 ms TTFT	130.9849 ms TTFT	276.71,976 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
512K	118.1914 ms TTFT	149.51,646 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	111.62,466 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	9,758.0	—	840 ms	—	—	1	—	—
Prefill	16K	1	Valid	10.2K	—	1,593 ms	—	—	1	—	—
Prefill	32K	1	Valid	10.3K	—	3,126 ms	—	—	1	—	—
Prefill	64K	1	Valid	10.5K	—	6,163 ms	—	—	1	—	—
Prefill	128K	1	Valid	10.2K	—	12,589 ms	—	—	1	—	—
Prefill	256K	1	Valid	9,621.0	—	26,744 ms	—	—	1	—	—
Prefill	512K	1	Valid	8,548.0	—	60,188 ms	—	—	1	—	—
Prefill	1019K	1	Valid	6,931.0	—	147,706 ms	—	—	1	—	—
Prefill	1024K	1	Valid	6,904.0	—	149,021 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	125.6	126.3	64 ms	7.9 ms	1.0	1	3,770	30.0 s
Sustained decode	8K	2	Valid	222.5	110.9	97 ms	9.0 ms	2.0	0	6,674	30.0 s
Sustained decode	8K	4	Valid	343.4	86.3	142 ms	11.6 ms	4.0	0	10,304	30.0 s
Sustained decode	8K	8	Requests queued	477.8	68.7	140 ms	14.6 ms	6.9	7	14,330	30.0 s
Sustained decode	8K	16	Requests queued	486.7	70.1	55,531 ms	14.3 ms	7.0	6	14,584	30.0 s
Sustained decode	8K	32	Requests queued	480.9	68.9	48,588 ms	14.5 ms	7.0	4	14,424	30.0 s
Sustained decode	8K	64	Requests queued	483.0	69.3	48,409 ms	14.4 ms	7.0	4	14,492	30.0 s
Sustained decode	8K	128	Requests queued	483.6	69.4	14,018 ms	14.4 ms	7.0	0	14,509	30.0 s
Sustained decode	16K	1	Valid	124.8	126.2	209 ms	7.9 ms	1.0	1	3,744	30.0 s
Sustained decode	16K	2	Valid	200.2	95.2	1,420 ms	10.5 ms	3.1	0	6,006	30.0 s
Sustained decode	16K	4	Valid	335.9	82.1	462 ms	12.2 ms	4.2	0	10,080	30.0 s
Sustained decode	16K	8	Requests queued	473.4	68.3	457 ms	14.6 ms	6.9	7	14,189	30.0 s
Sustained decode	16K	16	Requests queued	471.0	68.1	57,651 ms	14.7 ms	7.0	6	14,123	30.0 s
Sustained decode	16K	32	Requests queued	481.1	69.0	45,433 ms	14.5 ms	7.0	4	14,430	30.0 s
Sustained decode	16K	64	Requests queued	478.4	68.6	45,247 ms	14.6 ms	7.0	4	14,346	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	124.4	125.9	213 ms	7.9 ms	1.0	1	3,731	30.0 s
Sustained decode	32K	2	Valid	137.1	68.4	310 ms	14.6 ms	5.0	0	4,108	30.0 s
Sustained decode	32K	4	Valid	312.2	75.1	3,992 ms	13.3 ms	4.8	0	9,368	30.0 s
Sustained decode	32K	8	Requests queued	455.7	66.2	5,181 ms	15.1 ms	7.0	7	13,664	30.0 s
Sustained decode	32K	16	Requests queued	460.0	66.3	40,851 ms	15.1 ms	7.0	7	13,798	30.0 s
Sustained decode	32K	32	Requests queued	448.7	66.5	61,389 ms	15.0 ms	7.0	5	13,460	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	123.7	125.6	272 ms	8.0 ms	1.0	1	3,710	30.0 s
Sustained decode	64K	2	Valid	189.3	91.8	395 ms	10.9 ms	3.3	0	5,674	30.0 s
Sustained decode	64K	4	Valid	321.9	78.6	529 ms	12.7 ms	4.5	0	9,656	30.0 s
Sustained decode	64K	8	Requests queued	451.8	66.9	852 ms	14.9 ms	6.9	14	13,561	30.0 s
Sustained decode	64K	16	Requests queued	466.1	67.4	61,863 ms	14.8 ms	6.9	14	13,979	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	122.1	123.4	407 ms	8.1 ms	1.0	1	3,664	30.0 s
Sustained decode	128K	2	Valid	180.3	87.1	559 ms	11.5 ms	3.9	0	5,410	30.0 s
Sustained decode	128K	4	Valid	309.6	75.9	841 ms	13.2 ms	4.6	0	9,276	30.0 s
Sustained decode	128K	8	Requests queued	443.8	64.2	1,104 ms	15.6 ms	7.0	17	13,311	30.0 s
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	122.0	122.0	517 ms	8.2 ms	1.0	1	3,595	29.5 s
Sustained decode	256K	2	Valid	130.9	65.3	849 ms	15.3 ms	6.0	0	3,922	30.0 s
Sustained decode	256K	4	Valid	276.7	68.5	1,976 ms	14.6 ms	5.9	0	8,296	30.0 s
Sustained decode	256K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	118.1	118.1	914 ms	8.5 ms	1.0	0	3,544	30.0 s
Sustained decode	512K	2	Valid	149.5	73.8	1,646 ms	13.5 ms	4.6	0	4,487	30.0 s
Sustained decode	512K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	111.6	111.6	2,466 ms	9.0 ms	1.0	0	3,349	30.0 s
Sustained decode	1019K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.