Speed test matrix

granite-4.2-8b on H200 NVL 141GB

sglang · 0.5.18 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 19, 2026, 05:31 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.18
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2

Run

Served model: granite-4.2-8b
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-ibm-sglang/bin/sglang" serve --model-path ibm-granite/granite-4.2-8b --served-model-name granite-4.2-8b --tp 1 --reasoning-parser auto --tool-call-parser auto --context-length 131072 --mem-fraction-static 0.859766 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16

Prefill

prompt tok/s by context length

8K context25.7K319 ms TTFT

16K context23.1K703 ms TTFT

32K context19.5K1,657 ms TTFT

64K context14.1K4,575 ms TTFT

123K context9,22813,394 ms TTFT

128K context9,07214,179 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	83.655 ms TTFT	243.068 ms TTFT	510.080 ms TTFT	732.0120 ms TTFT	1,390.9199 ms TTFT	1,809.1333 ms TTFT	Over capacity	Over capacity
16K	135.773 ms TTFT	93.291 ms TTFT	344.4121 ms TTFT	704.4184 ms TTFT	979.7399 ms TTFT	1,038.0718 ms TTFT	Over capacity	Over capacity
32K	116.0105 ms TTFT	56.2133 ms TTFT	123.7216 ms TTFT	232.9375 ms TTFT	451.7697 ms TTFT	Over capacity	Over capacity	Over capacity
64K	108.7136 ms TTFT	52.2274 ms TTFT	84.2406 ms TTFT	149.8765 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
123K	86.9253 ms TTFT	30.8448 ms TTFT	64.3777 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	25.7K	—	319 ms	—	—	1	—	—
Prefill	16K	1	Valid	23.1K	—	703 ms	—	—	1	—	—
Prefill	32K	1	Valid	19.5K	—	1,657 ms	—	—	1	—	—
Prefill	64K	1	Valid	14.1K	—	4,575 ms	—	—	1	—	—
Prefill	123K	1	Valid	9,228.0	—	13,394 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,072.0	—	14,179 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	83.6	83.6	55 ms	12.0 ms	—	0	2,509	30.0 s
Sustained decode	8K	2	Valid	243.0	116.5	68 ms	8.6 ms	—	0	7,290	30.0 s
Sustained decode	8K	4	Valid	510.0	130.0	80 ms	7.7 ms	—	4	15,298	30.0 s
Sustained decode	8K	8	Valid	732.0	92.3	120 ms	10.8 ms	—	0	21,952	30.0 s
Sustained decode	8K	16	Valid	1,390.9	85.0	199 ms	11.8 ms	—	0	41,728	30.0 s
Sustained decode	8K	32	Valid	1,809.1	55.9	333 ms	17.9 ms	—	0	54,272	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	135.7	134.2	73 ms	7.5 ms	—	1	4,072	30.0 s
Sustained decode	16K	2	Valid	93.2	47.0	91 ms	21.3 ms	—	0	2,794	30.0 s
Sustained decode	16K	4	Valid	344.4	80.5	121 ms	12.4 ms	—	0	10,332	30.0 s
Sustained decode	16K	8	Valid	704.4	86.4	184 ms	11.6 ms	—	0	21,128	30.0 s
Sustained decode	16K	16	Valid	979.7	60.0	399 ms	16.7 ms	—	0	29,392	30.0 s
Sustained decode	16K	32	Valid	1,038.0	32.8	718 ms	30.5 ms	—	0	31,136	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	116.0	116.0	105 ms	8.6 ms	—	0	3,478	30.0 s
Sustained decode	32K	2	Valid	56.2	27.8	133 ms	35.9 ms	—	0	1,686	30.0 s
Sustained decode	32K	4	Valid	123.7	31.1	216 ms	32.2 ms	—	0	3,708	30.0 s
Sustained decode	32K	8	Valid	232.9	28.7	375 ms	34.9 ms	—	0	6,984	30.0 s
Sustained decode	32K	16	Valid	451.7	28.4	697 ms	35.2 ms	—	0	13,552	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	108.7	108.8	136 ms	9.2 ms	—	0	3,262	30.0 s
Sustained decode	64K	2	Valid	52.2	26.1	274 ms	38.3 ms	—	0	1,566	30.0 s
Sustained decode	64K	4	Valid	84.2	21.1	406 ms	47.4 ms	—	0	2,524	30.0 s
Sustained decode	64K	8	Valid	149.8	18.5	765 ms	54.0 ms	—	0	4,488	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	86.9	85.7	253 ms	11.7 ms	—	0	2,606	30.0 s
Sustained decode	123K	2	Valid	30.8	15.4	448 ms	64.9 ms	—	0	922	30.0 s
Sustained decode	123K	4	Valid	64.3	15.8	777 ms	63.2 ms	—	0	1,928	30.0 s
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.