Speed test matrix

gemma-3n-E4B-it-FP8-dynamic on H200 NVL 141GB

vllm · 0.10.0 · FP83 contexts (8K–27K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 27K, 32K

by @keennaylmx dev importMeasured Sep 7, 2026, 13:06 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: vllm
Version: 0.10.0
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2

Run

Served model: gemma
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-gemma3n-vllm/bin/vllm" serve RedHatAI/gemma-3n-E4B-it-FP8-dynamic --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --max-model-len 32768 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-seqs 2 --enforce-eager

Prefill

prompt tok/s by context length

8K context32.8K250 ms TTFT

16K context31.9K510 ms TTFT

27K context34.3K797 ms TTFT

32K context34.5K939 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	59.9Requests queued46 ms TTFT	113.756 ms TTFT	114.7Requests queued35,841 ms TTFT	114.9Requests queued35,863 ms TTFT	115.1Requests queued35,791 ms TTFT	113.9Requests queued36,083 ms TTFT	114.2Requests queued36,067 ms TTFT	114.9Requests queued35,884 ms TTFT
16K	59.6Requests queued75 ms TTFT	113.7101 ms TTFT	114.4Requests queued35,920 ms TTFT	114.2Requests queued35,991 ms TTFT	113.5Requests queued36,238 ms TTFT	113.4Requests queued36,489 ms TTFT	113.7Requests queued36,220 ms TTFT	113.8Requests queued36,390 ms TTFT
27K	59.4Requests queued74 ms TTFT	113.879 ms TTFT	114.3Requests queued36,053 ms TTFT	113.9Requests queued36,163 ms TTFT	114.6Requests queued35,970 ms TTFT	114.1Requests queued36,147 ms TTFT	114.1Requests queued36,054 ms TTFT	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	32.8K	—	250 ms	—	—	1	—	—
Prefill	16K	1	Valid	31.9K	—	510 ms	—	—	1	—	—
Prefill	27K	1	Valid	34.3K	—	797 ms	—	—	1	—	—
Prefill	32K	1	Valid	34.5K	—	939 ms	—	—	1	—	—
Sustained decode	8K	1	Requests queued	59.9	59.9	46 ms	16.7 ms	1.0	0	1,796	30.0 s
Sustained decode	8K	2	Valid	113.7	80.5	56 ms	13.6 ms	2.0	0	3,410	30.0 s
Sustained decode	8K	4	Requests queued	114.7	86.1	35,841 ms	13.1 ms	2.0	2	3,438	30.0 s
Sustained decode	8K	8	Requests queued	114.9	94.8	35,863 ms	12.5 ms	2.0	2	3,444	30.0 s
Sustained decode	8K	16	Requests queued	115.1	57.6	35,791 ms	17.4 ms	2.0	2	3,452	30.0 s
Sustained decode	8K	32	Requests queued	113.9	56.9	36,083 ms	17.6 ms	2.0	2	3,416	30.0 s
Sustained decode	8K	64	Requests queued	114.2	69.9	36,067 ms	14.8 ms	2.0	2	3,427	30.0 s
Sustained decode	8K	128	Requests queued	114.9	106.8	35,884 ms	11.9 ms	2.0	2	3,448	30.0 s
Sustained decode	16K	1	Requests queued	59.6	59.6	75 ms	16.8 ms	1.0	0	1,788	30.0 s
Sustained decode	16K	2	Valid	113.7	71.0	101 ms	14.5 ms	2.0	0	3,410	30.0 s
Sustained decode	16K	4	Requests queued	114.4	89.7	35,920 ms	12.8 ms	2.0	2	3,432	30.0 s
Sustained decode	16K	8	Requests queued	114.2	102.0	35,991 ms	12.2 ms	2.0	2	3,426	30.0 s
Sustained decode	16K	16	Requests queued	113.5	56.8	36,238 ms	17.6 ms	2.0	2	3,404	30.0 s
Sustained decode	16K	32	Requests queued	113.4	56.7	36,489 ms	17.6 ms	2.0	2	3,402	30.0 s
Sustained decode	16K	64	Requests queued	113.7	99.0	36,220 ms	12.3 ms	2.0	2	3,412	30.0 s
Sustained decode	16K	128	Requests queued	113.8	56.9	36,390 ms	17.6 ms	2.0	2	3,414	30.0 s
Sustained decode	27K	1	Requests queued	59.4	59.4	74 ms	16.8 ms	1.0	0	1,782	30.0 s
Sustained decode	27K	2	Valid	113.8	56.8	79 ms	17.6 ms	2.0	0	3,414	30.0 s
Sustained decode	27K	4	Requests queued	114.3	81.2	36,053 ms	13.5 ms	2.0	2	3,428	30.0 s
Sustained decode	27K	8	Requests queued	113.9	57.0	36,163 ms	17.6 ms	2.0	2	3,416	30.0 s
Sustained decode	27K	16	Requests queued	114.6	104.6	35,970 ms	12.0 ms	2.0	2	3,436	30.0 s
Sustained decode	27K	32	Requests queued	114.1	90.8	36,147 ms	12.8 ms	2.0	2	3,422	30.0 s
Sustained decode	27K	64	Requests queued	114.1	92.3	36,054 ms	12.7 ms	2.0	2	3,420	30.0 s
Sustained decode	27K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.