Speed test matrix

gemma-4-26B-A4B-it on H200 NVL 141GB

sglang · 0.5.6.post3.dev9269+g834400705 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K

by @keennaylmx dev importMeasured Sep 1, 2026, 19:53 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9269+g834400705
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 834400705f2d

Run

Served model: gemma
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-z-lab-sglang/bin/sglang" serve --model-path google/gemma-4-26B-A4B-it --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 262144 --mem-fraction-static 0.846617 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton --speculative-algorithm DFLASH --speculative-draft-model-path z-lab/gemma-4-26B-A4B-it-DFlash --speculative-num-draft-tokens 16 --speculative-draft-attention-backend fa4

Prefill

prompt tok/s by context length

8K context5,9581,377 ms TTFT

16K context10K1,621 ms TTFT

32K context10.9K2,960 ms TTFT

64K context6,9209,330 ms TTFT

128K context3,98132,399 ms TTFT

251K context2,204114,682 ms TTFT

256K context2,164119,116 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	10.9Requests queued374 ms TTFT	54.71,695 ms TTFT	96.1519 ms TTFT	101.4696 ms TTFT	291.8703 ms TTFT	Over capacity	Over capacity	Over capacity
16K	366.8Requests queued116 ms TTFT	32.8839 ms TTFT	43.7535 ms TTFT	123.8647 ms TTFT	211.7751 ms TTFT	Over capacity	Over capacity	Over capacity
32K	270.3178 ms TTFT	138.6407 ms TTFT	226.8416 ms TTFT	497.4776 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
64K	90.0311 ms TTFT	50.4556 ms TTFT	162.9924 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
128K	85.4744 ms TTFT	51.21,406 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
251K	12.91,743 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	5,958.0	—	1,377 ms	—	—	1	—	—
Prefill	16K	1	Valid	10K	—	1,621 ms	—	—	1	—	—
Prefill	32K	1	Valid	10.9K	—	2,960 ms	—	—	1	—	—
Prefill	64K	1	Valid	6,920.0	—	9,330 ms	—	—	1	—	—
Prefill	128K	1	Valid	3,981.0	—	32,399 ms	—	—	1	—	—
Prefill	251K	1	Valid	2,204.0	—	114,682 ms	—	—	1	—	—
Prefill	256K	1	Valid	2,164.0	—	119,116 ms	—	—	1	—	—
Sustained decode	8K	1	Requests queued	10.9	10.9	374 ms	92.1 ms	2.0	0	327	30.0 s
Sustained decode	8K	2	Valid	54.7	23.0	1,695 ms	54.6 ms	3.0	0	1,636	29.9 s
Sustained decode	8K	4	Valid	96.1	22.0	519 ms	61.8 ms	6.7	0	2,869	29.9 s
Sustained decode	8K	8	Valid	101.4	10.5	696 ms	95.6 ms	14.3	0	3,027	29.9 s
Sustained decode	8K	16	Valid	291.8	16.7	703 ms	59.7 ms	29.7	0	8,704	29.8 s
Sustained decode	8K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Requests queued	366.8	341.3	116 ms	2.9 ms	0.9	2	10,995	30.0 s
Sustained decode	16K	2	Valid	32.8	15.8	839 ms	65.7 ms	30.1	0	981	29.9 s
Sustained decode	16K	4	Valid	43.7	8.8	535 ms	114.5 ms	30.1	0	1,303	29.8 s
Sustained decode	16K	8	Valid	123.8	16.4	647 ms	61.8 ms	30.8	0	3,710	30.0 s
Sustained decode	16K	16	Valid	211.7	11.4	751 ms	87.7 ms	41.0	0	6,285	29.7 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	270.3	254.2	178 ms	3.9 ms	0.9	2	8,104	30.0 s
Sustained decode	32K	2	Valid	138.6	64.5	407 ms	15.5 ms	36.3	0	4,148	29.9 s
Sustained decode	32K	4	Valid	226.8	50.7	416 ms	22.7 ms	29.4	0	6,778	29.9 s
Sustained decode	32K	8	Valid	497.4	66.4	776 ms	15.1 ms	31.3	0	14,916	30.0 s
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	90.0	84.3	311 ms	11.9 ms	1.0	0	2,696	30.0 s
Sustained decode	64K	2	Valid	50.4	24.1	556 ms	46.0 ms	25.0	0	1,508	29.9 s
Sustained decode	64K	4	Valid	162.9	34.4	924 ms	40.4 ms	20.9	0	4,878	29.9 s
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	85.4	72.8	744 ms	13.7 ms	1.0	0	2,552	29.9 s
Sustained decode	128K	2	Valid	51.2	24.2	1,406 ms	70.6 ms	17.9	0	1,528	29.9 s
Sustained decode	128K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	1	Valid	12.9	12.5	1,743 ms	80.0 ms	1.0	0	385	29.9 s
Sustained decode	251K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	251K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.