Speed test matrix

granite-4.2-30b on H200 NVL 141GB

sglang · 0.5.18 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 19, 2026, 04:55 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.18
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2

Run

Served model: granite-4.2-30b
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-ibm-sglang/bin/sglang" serve --model-path ibm-granite/granite-4.2-30b --served-model-name granite-4.2-30b --tp 1 --reasoning-parser auto --tool-call-parser auto --context-length 131072 --mem-fraction-static 0.851009 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16

Prefill

prompt tok/s by context length

8K context8,556958 ms TTFT

16K context8,2321,969 ms TTFT

32K context7,3784,374 ms TTFT

64K context6,02710,683 ms TTFT

123K context4,44627,799 ms TTFT

128K context4,39129,294 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	38.0103 ms TTFT	58.9112 ms TTFT	121.8109 ms TTFT	257.9143 ms TTFT	504.1196 ms TTFT	Over capacity	Over capacity	Over capacity
16K	36.3116 ms TTFT	58.8139 ms TTFT	110.0160 ms TTFT	204.4193 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
32K	31.6141 ms TTFT	50.2162 ms TTFT	96.7243 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
64K	35.1188 ms TTFT	44.2273 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
123K	27.1306 ms TTFT	33.6421 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	8,556.0	—	958 ms	—	—	1	—	—
Prefill	16K	1	Valid	8,232.0	—	1,969 ms	—	—	1	—	—
Prefill	32K	1	Valid	7,378.0	—	4,374 ms	—	—	1	—	—
Prefill	64K	1	Valid	6,027.0	—	10,683 ms	—	—	1	—	—
Prefill	123K	1	Valid	4,446.0	—	27,799 ms	—	—	1	—	—
Prefill	128K	1	Valid	4,391.0	—	29,294 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	38.0	38.0	103 ms	26.3 ms	—	0	1,139	30.0 s
Sustained decode	8K	2	Valid	58.9	29.3	112 ms	34.2 ms	—	0	1,766	30.0 s
Sustained decode	8K	4	Valid	121.8	30.4	109 ms	32.9 ms	—	0	3,652	30.0 s
Sustained decode	8K	8	Valid	257.9	32.1	143 ms	31.1 ms	—	0	7,737	30.0 s
Sustained decode	8K	16	Valid	504.1	31.5	196 ms	31.7 ms	—	0	15,120	30.0 s
Sustained decode	8K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	36.3	36.3	116 ms	27.6 ms	—	0	1,087	30.0 s
Sustained decode	16K	2	Valid	58.8	29.4	139 ms	34.0 ms	—	0	1,762	30.0 s
Sustained decode	16K	4	Valid	110.0	27.4	160 ms	36.5 ms	—	0	3,296	30.0 s
Sustained decode	16K	8	Valid	204.4	25.5	193 ms	39.2 ms	—	0	6,128	30.0 s
Sustained decode	16K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	31.6	31.5	141 ms	31.8 ms	—	0	947	30.0 s
Sustained decode	32K	2	Valid	50.2	25.1	162 ms	39.8 ms	—	0	1,506	30.0 s
Sustained decode	32K	4	Valid	96.7	24.1	243 ms	41.4 ms	—	0	2,900	30.0 s
Sustained decode	32K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	35.1	35.1	188 ms	28.5 ms	—	0	1,051	30.0 s
Sustained decode	64K	2	Valid	44.2	22.1	273 ms	45.3 ms	—	0	1,326	30.0 s
Sustained decode	64K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	27.1	27.1	306 ms	36.9 ms	—	0	812	30.0 s
Sustained decode	123K	2	Valid	33.6	16.8	421 ms	59.5 ms	—	0	1,006	30.0 s
Sustained decode	123K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.