Speed test matrix

granite-4.2-3b on H200 NVL 141GB

vllm · 0.25.0 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 3, 2026, 09:08 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.25.0
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: granite-4.2-3b
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-ibm-vllm/bin/vllm" serve ibm-granite/granite-4.2-3b --served-model-name granite-4.2-3b --tensor-parallel-size 1 --reasoning-parser granite_thinking_parser --reasoning-parser-plugin /workspace/compute/recipes/nvidia-h200/ibm-granite/plugin/granite_thinking_parser.py --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 131072 --gpu-memory-utilization 0.866765 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16

Prefill

prompt tok/s by context length

8K context49.7K165 ms TTFT

16K context45.2K359 ms TTFT

32K context31.7K1,018 ms TTFT

64K context21.8K2,948 ms TTFT

123K context13.5K9,143 ms TTFT

128K context13.3K9,644 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	271.355 ms TTFT	510.064 ms TTFT	929.286 ms TTFT	1,552.9133 ms TTFT	2,414.6155 ms TTFT	3,266.1284 ms TTFT	4,193.1478 ms TTFT	Over capacity
16K	256.692 ms TTFT	473.387 ms TTFT	815.4135 ms TTFT	1,250.9168 ms TTFT	1,786.2258 ms TTFT	2,294.8462 ms TTFT	2,689.8645 ms TTFT	Over capacity
32K	235.8103 ms TTFT	414.5160 ms TTFT	647.1191 ms TTFT	910.9314 ms TTFT	1,203.4511 ms TTFT	1,408.1758 ms TTFT	Over capacity	Over capacity
64K	204.2151 ms TTFT	326.6232 ms TTFT	468.0340 ms TTFT	601.7526 ms TTFT	720.2921 ms TTFT	Over capacity	Over capacity	Over capacity
123K	161.1296 ms TTFT	239.0326 ms TTFT	313.7535 ms TTFT	369.6890 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	49.7K	—	165 ms	—	—	1	—	—
Prefill	16K	1	Valid	45.2K	—	359 ms	—	—	1	—	—
Prefill	32K	1	Valid	31.7K	—	1,018 ms	—	—	1	—	—
Prefill	64K	1	Valid	21.8K	—	2,948 ms	—	—	1	—	—
Prefill	123K	1	Valid	13.5K	—	9,143 ms	—	—	1	—	—
Prefill	128K	1	Valid	13.3K	—	9,644 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	271.3	272.4	55 ms	3.7 ms	1.0	2	8,139	30.0 s
Sustained decode	8K	2	Valid	510.0	256.5	64 ms	3.9 ms	2.0	4	15,298	30.0 s
Sustained decode	8K	4	Valid	929.2	233.9	86 ms	4.3 ms	4.0	4	27,873	30.0 s
Sustained decode	8K	8	Valid	1,552.9	196.8	133 ms	5.1 ms	8.0	8	46,584	30.0 s
Sustained decode	8K	16	Valid	2,414.6	156.3	155 ms	6.4 ms	16.0	16	72,440	30.0 s
Sustained decode	8K	32	Valid	3,266.1	103.6	284 ms	9.7 ms	32.0	0	97,985	30.0 s
Sustained decode	8K	64	Valid	4,193.1	66.4	478 ms	15.1 ms	64.0	0	125,824	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	256.6	258.5	92 ms	3.9 ms	1.0	2	7,702	30.0 s
Sustained decode	16K	2	Valid	473.3	238.6	87 ms	4.2 ms	2.0	4	14,198	30.0 s
Sustained decode	16K	4	Valid	815.4	205.5	135 ms	4.9 ms	4.0	4	24,463	30.0 s
Sustained decode	16K	8	Valid	1,250.9	159.2	168 ms	6.3 ms	8.0	8	37,525	30.0 s
Sustained decode	16K	16	Valid	1,786.2	112.5	258 ms	8.9 ms	16.0	0	53,584	30.0 s
Sustained decode	16K	32	Valid	2,294.8	72.0	462 ms	13.9 ms	32.0	0	68,832	30.0 s
Sustained decode	16K	64	Valid	2,689.8	42.2	645 ms	23.7 ms	64.0	0	80,704	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	235.8	237.1	103 ms	4.2 ms	1.0	1	7,075	30.0 s
Sustained decode	32K	2	Valid	414.5	209.0	160 ms	4.8 ms	1.9	2	12,436	30.0 s
Sustained decode	32K	4	Valid	647.1	164.1	191 ms	6.1 ms	4.0	4	19,414	30.0 s
Sustained decode	32K	8	Valid	910.9	114.3	314 ms	8.8 ms	8.0	0	27,320	30.0 s
Sustained decode	32K	16	Valid	1,203.4	75.4	511 ms	13.3 ms	16.0	0	36,095	30.0 s
Sustained decode	32K	32	Valid	1,408.1	44.0	758 ms	22.7 ms	32.0	0	42,240	30.0 s
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	204.2	205.6	151 ms	4.9 ms	1.0	1	6,124	30.0 s
Sustained decode	64K	2	Valid	326.6	165.5	232 ms	6.0 ms	2.0	2	9,799	30.0 s
Sustained decode	64K	4	Valid	468.0	117.4	340 ms	8.5 ms	4.0	0	14,048	30.0 s
Sustained decode	64K	8	Valid	601.7	75.4	526 ms	13.3 ms	8.0	0	18,056	30.0 s
Sustained decode	64K	16	Valid	720.2	45.0	921 ms	22.2 ms	16.0	0	21,600	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	161.1	163.6	296 ms	6.1 ms	1.0	1	4,832	30.0 s
Sustained decode	123K	2	Valid	239.0	121.0	326 ms	8.3 ms	2.0	2	7,168	30.0 s
Sustained decode	123K	4	Valid	313.7	78.6	535 ms	12.7 ms	4.0	0	9,408	30.0 s
Sustained decode	123K	8	Valid	369.6	46.2	890 ms	21.6 ms	8.0	0	11,088	30.0 s
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.