Speed test matrix

granite-4.2-3b on RTX PRO 6000 Blackwell Server Edition 96GB

vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Oct 4, 2026, 10:39 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev452+g73a583112
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 73a5831127a9

Run

Served model: granite-4.2-3b
Revision: main
Benchmark tool: llm-inference-bench v0.7.6
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_rtxpro6k-ibm-granite-vllm-73a5831127a9/bin/vllm" serve ibm-granite/granite-4.2-3b --served-model-name granite-4.2-3b --tensor-parallel-size 1 --reasoning-parser granite_thinking_parser --enable-auto-tool-choice --tool-call-parser qwen3_coder --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --attention-backend FLASH_ATTN

Prefill

prompt tok/s by context length

8K context36.9K222 ms TTFT

16K context31.1K521 ms TTFT

32K context23K1,401 ms TTFT

64K context15K4,285 ms TTFT

123K context9,07013,627 ms TTFT

128K context8,84314,546 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	149.729 ms TTFT	268.532 ms TTFT	507.949 ms TTFT	Looped	1,529.6142 ms TTFT	2,320.9262 ms TTFT	3,104.3507 ms TTFT	Over capacity
16K	140.545 ms TTFT	247.962 ms TTFT	459.187 ms TTFT	806.8141 ms TTFT	1,217.1203 ms TTFT	1,698.2383 ms TTFT	Over capacity	Over capacity
32K	124.974 ms TTFT	216.592 ms TTFT	390.0133 ms TTFT	626.5207 ms TTFT	852.2385 ms TTFT	Over capacity	Over capacity	Over capacity
64K	102.8148 ms TTFT	172.0183 ms TTFT	299.1270 ms TTFT	430.7449 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
123K	77.1267 ms TTFT	107.7349 ms TTFT	197.5507 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	36.9K	—	222 ms	—	—	1	—	—
Prefill	16K	1	Valid	31.1K	—	521 ms	—	—	1	—	—
Prefill	32K	1	Valid	23K	—	1,401 ms	—	—	1	—	—
Prefill	64K	1	Valid	15K	—	4,285 ms	—	—	1	—	—
Prefill	123K	1	Valid	9,070.0	—	13,627 ms	—	—	1	—	—
Prefill	128K	1	Valid	8,843.0	—	14,546 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	149.7	150.6	29 ms	6.6 ms	1.0	1	4,487	30.0 s
Sustained decode	8K	2	Valid	268.5	136.2	32 ms	7.3 ms	2.0	2	8,054	30.0 s
Sustained decode	8K	4	Valid	507.9	129.9	49 ms	7.7 ms	4.0	4	15,235	30.0 s
Sustained decode	8K	8	Looped	—	—	—	—	—	0	0	—
Sustained decode	8K	16	Valid	1,529.6	96.5	142 ms	10.4 ms	16.0	0	45,888	30.0 s
Sustained decode	8K	32	Valid	2,320.9	73.0	262 ms	13.7 ms	32.0	0	69,600	30.0 s
Sustained decode	8K	64	Valid	3,104.3	49.1	507 ms	20.4 ms	64.0	0	93,122	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	140.5	141.6	45 ms	7.1 ms	1.0	1	4,214	30.0 s
Sustained decode	16K	2	Valid	247.9	125.6	62 ms	8.0 ms	2.0	2	7,435	30.0 s
Sustained decode	16K	4	Valid	459.1	115.3	87 ms	8.7 ms	4.0	0	13,772	30.0 s
Sustained decode	16K	8	Valid	806.8	101.4	141 ms	9.9 ms	8.0	0	24,200	30.0 s
Sustained decode	16K	16	Valid	1,217.1	76.3	203 ms	13.1 ms	16.0	0	36,513	30.0 s
Sustained decode	16K	32	Valid	1,698.2	53.0	383 ms	18.9 ms	32.0	0	50,942	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	124.9	126.9	74 ms	7.9 ms	1.0	1	3,747	30.0 s
Sustained decode	32K	2	Valid	216.5	108.5	92 ms	9.2 ms	2.0	0	6,494	30.0 s
Sustained decode	32K	4	Valid	390.0	97.8	133 ms	10.2 ms	4.0	0	11,696	30.0 s
Sustained decode	32K	8	Valid	626.5	78.4	207 ms	12.8 ms	8.0	0	18,792	30.0 s
Sustained decode	32K	16	Valid	852.2	53.1	385 ms	18.8 ms	16.0	0	25,552	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	102.8	102.9	148 ms	9.7 ms	1.0	0	3,082	30.0 s
Sustained decode	64K	2	Valid	172.0	86.1	183 ms	11.6 ms	2.0	0	5,158	30.0 s
Sustained decode	64K	4	Valid	299.1	74.9	270 ms	13.4 ms	4.0	0	8,968	30.0 s
Sustained decode	64K	8	Valid	430.7	53.7	449 ms	18.6 ms	8.0	0	12,920	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	77.1	77.2	267 ms	13.0 ms	1.0	0	2,313	30.0 s
Sustained decode	123K	2	Valid	107.7	54.0	349 ms	18.5 ms	2.0	0	3,230	30.0 s
Sustained decode	123K	4	Valid	197.5	49.4	507 ms	20.3 ms	4.0	0	5,923	30.0 s
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.