Speed test matrix

granite-4.2-8b on H200 NVL 141GB

vllm · 0.25.0 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 3, 2026, 09:33 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.25.0
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: granite-4.2-8b
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-ibm-vllm/bin/vllm" serve ibm-granite/granite-4.2-8b --served-model-name granite-4.2-8b --tensor-parallel-size 1 --reasoning-parser granite_thinking_parser --reasoning-parser-plugin /workspace/compute/recipes/nvidia-h200/ibm-granite/plugin/granite_thinking_parser.py --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 131072 --gpu-memory-utilization 0.864805 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16

Prefill

prompt tok/s by context length

8K context27.3K300 ms TTFT

16K context23.5K689 ms TTFT

32K context19.4K1,664 ms TTFT

64K context14.3K4,516 ms TTFT

123K context9,40113,143 ms TTFT

128K context9,18613,998 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	145.951 ms TTFT	281.162 ms TTFT	530.469 ms TTFT	920.7167 ms TTFT	1,472.7227 ms TTFT	2,135.2330 ms TTFT	Over capacity	Over capacity
16K	138.885 ms TTFT	263.978 ms TTFT	478.1119 ms TTFT	782.1200 ms TTFT	1,164.3358 ms TTFT	1,500.8386 ms TTFT	Over capacity	Over capacity
32K	126.8112 ms TTFT	234.7148 ms TTFT	400.8208 ms TTFT	595.9328 ms TTFT	787.2518 ms TTFT	Over capacity	Over capacity	Over capacity
64K	109.2149 ms TTFT	194.3212 ms TTFT	297.6359 ms TTFT	390.2557 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
123K	87.0275 ms TTFT	144.1375 ms TTFT	195.2597 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	27.3K	—	300 ms	—	—	1	—	—
Prefill	16K	1	Valid	23.5K	—	689 ms	—	—	1	—	—
Prefill	32K	1	Valid	19.4K	—	1,664 ms	—	—	1	—	—
Prefill	64K	1	Valid	14.3K	—	4,516 ms	—	—	1	—	—
Prefill	123K	1	Valid	9,401.0	—	13,143 ms	—	—	1	—	—
Prefill	128K	1	Valid	9,186.0	—	13,998 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	145.9	146.9	51 ms	6.8 ms	1.0	1	4,376	30.0 s
Sustained decode	8K	2	Valid	281.1	142.2	62 ms	7.0 ms	2.0	2	8,434	30.0 s
Sustained decode	8K	4	Valid	530.4	135.2	69 ms	7.4 ms	4.0	4	15,911	30.0 s
Sustained decode	8K	8	Valid	920.7	116.0	167 ms	8.6 ms	8.0	0	27,640	30.0 s
Sustained decode	8K	16	Valid	1,472.7	93.4	227 ms	10.7 ms	16.0	0	44,176	30.0 s
Sustained decode	8K	32	Valid	2,135.2	67.8	330 ms	14.7 ms	32.0	0	64,032	30.0 s
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	138.8	140.3	85 ms	7.1 ms	1.0	1	4,165	30.0 s
Sustained decode	16K	2	Valid	263.9	133.8	78 ms	7.5 ms	2.0	2	7,918	30.0 s
Sustained decode	16K	4	Valid	478.1	122.6	119 ms	8.2 ms	4.0	4	14,341	30.0 s
Sustained decode	16K	8	Valid	782.1	98.3	200 ms	10.2 ms	8.0	0	23,473	30.0 s
Sustained decode	16K	16	Valid	1,164.3	73.5	358 ms	13.6 ms	16.0	0	34,928	30.0 s
Sustained decode	16K	32	Valid	1,500.8	47.4	386 ms	21.1 ms	32.0	0	45,026	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	126.8	129.6	112 ms	7.7 ms	1.0	1	3,803	30.0 s
Sustained decode	32K	2	Valid	234.7	117.8	148 ms	8.5 ms	2.0	0	7,040	30.0 s
Sustained decode	32K	4	Valid	400.8	100.5	208 ms	10.0 ms	4.0	0	12,024	30.0 s
Sustained decode	32K	8	Valid	595.9	74.8	328 ms	13.4 ms	8.0	0	17,880	30.0 s
Sustained decode	32K	16	Valid	787.2	49.5	518 ms	20.2 ms	16.0	0	23,616	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	109.2	109.5	149 ms	9.1 ms	1.0	0	3,277	30.0 s
Sustained decode	64K	2	Valid	194.3	97.4	212 ms	10.3 ms	2.0	0	5,828	30.0 s
Sustained decode	64K	4	Valid	297.6	74.6	359 ms	13.4 ms	4.0	0	8,928	30.0 s
Sustained decode	64K	8	Valid	390.2	48.9	557 ms	20.5 ms	8.0	0	11,704	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	87.0	87.2	275 ms	11.5 ms	1.0	0	2,608	30.0 s
Sustained decode	123K	2	Valid	144.1	72.3	375 ms	13.8 ms	2.0	0	4,322	30.0 s
Sustained decode	123K	4	Valid	195.2	48.9	597 ms	20.4 ms	4.0	0	5,856	30.0 s
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.