Speed test matrix
gemma-4-E2B-it on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/gemma-4-E2B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 131072 --gpu-memory-utilization 0.854015 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser gemma4 --limit-mm-per-prompt '{"image":4,"audio":1}' --async-schedulingPrefill
prompt tok/s by context length8K context62K132 ms TTFT
16K context56.8K286 ms TTFT
32K context51.4K629 ms TTFT
64K context41.1K1,571 ms TTFT
123K context27.6K4,492 ms TTFT
128K context26.4K4,884 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 260.371 ms TTFT | 514.087 ms TTFT | 1,001.5113 ms TTFT | 1,903.3134 ms TTFT | 3,529.7180 ms TTFT | 5,982.0248 ms TTFT | 9,049.3310 ms TTFT | 13.1K1,051 ms TTFT |
| 16K | 256.197 ms TTFT | 507.0116 ms TTFT | 975.3118 ms TTFT | 1,809.1207 ms TTFT | 3,243.9205 ms TTFT | 5,150.8268 ms TTFT | 7,255.1869 ms TTFT | 9,854.81,600 ms TTFT |
| 32K | 250.1145 ms TTFT | 489.5180 ms TTFT | 923.6233 ms TTFT | 1,650.8247 ms TTFT | 2,761.0208 ms TTFT | 4,029.2302 ms TTFT | 5,330.11,784 ms TTFT | 6,468.13,480 ms TTFT |
| 64K | 242.1211 ms TTFT | 463.6289 ms TTFT | 839.3268 ms TTFT | 1,394.8279 ms TTFT | 2,077.7302 ms TTFT | 2,875.22,018 ms TTFT | 3,419.33,670 ms TTFT | 3,796.17,401 ms TTFT |
| 123K | 227.1473 ms TTFT | 415.8541 ms TTFT | 709.4517 ms TTFT | 1,063.6573 ms TTFT | 1,497.92,692 ms TTFT | 1,859.24,997 ms TTFT | 2,047.68,685 ms TTFT | 2,151.016,966 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 62K | — | 132 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 56.8K | — | 286 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 51.4K | — | 629 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 41.1K | — | 1,571 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 27.6K | — | 4,492 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 26.4K | — | 4,884 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 260.3 | 263.9 | 71 ms | 3.8 ms | 1.0 | 2 | 7,809 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 514.0 | 260.9 | 87 ms | 3.8 ms | 2.0 | 4 | 15,419 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,001.5 | 256.7 | 113 ms | 3.9 ms | 4.0 | 8 | 30,043 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,903.3 | 243.4 | 134 ms | 4.1 ms | 8.0 | 16 | 57,096 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,529.7 | 223.6 | 180 ms | 4.5 ms | 15.7 | 16 | 105,889 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 5,982.0 | 192.7 | 248 ms | 5.2 ms | 32.0 | 32 | 179,462 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 9,049.3 | 149.9 | 310 ms | 6.7 ms | 63.4 | 64 | 271,495 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 13.1K | 104.2 | 1,051 ms | 9.6 ms | 128.0 | 0 | 393,466 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 256.1 | 258.4 | 97 ms | 3.9 ms | 1.0 | 2 | 7,687 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 507.0 | 256.7 | 116 ms | 3.9 ms | 2.0 | 4 | 15,210 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 975.3 | 246.2 | 118 ms | 4.1 ms | 4.0 | 8 | 29,260 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,809.1 | 228.4 | 207 ms | 4.4 ms | 8.0 | 8 | 54,274 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,243.9 | 205.5 | 205 ms | 4.9 ms | 16.0 | 16 | 97,316 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,150.8 | 164.1 | 268 ms | 6.1 ms | 31.7 | 32 | 154,526 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,255.1 | 114.9 | 869 ms | 8.7 ms | 64.0 | 0 | 217,617 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 9,854.8 | 77.5 | 1,600 ms | 12.9 ms | 128.0 | 0 | 295,680 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 250.1 | 255.5 | 145 ms | 3.9 ms | 1.0 | 2 | 7,507 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 489.5 | 247.8 | 180 ms | 4.0 ms | 2.0 | 4 | 14,688 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 923.6 | 233.9 | 233 ms | 4.3 ms | 3.9 | 4 | 27,707 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,650.8 | 209.1 | 247 ms | 4.8 ms | 8.0 | 8 | 49,524 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,761.0 | 173.7 | 208 ms | 5.8 ms | 16.0 | 16 | 82,824 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,029.2 | 127.1 | 302 ms | 7.9 ms | 31.8 | 32 | 120,879 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,330.1 | 83.1 | 1,784 ms | 12.0 ms | 64.0 | 0 | 159,872 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,468.1 | 50.8 | 3,480 ms | 19.7 ms | 128.0 | 0 | 194,048 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 242.1 | 245.3 | 211 ms | 4.1 ms | 1.0 | 1 | 7,262 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 463.6 | 234.3 | 289 ms | 4.3 ms | 2.0 | 2 | 13,907 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 839.3 | 211.9 | 268 ms | 4.7 ms | 3.9 | 4 | 25,177 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,394.8 | 175.3 | 279 ms | 5.7 ms | 7.9 | 8 | 41,840 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,077.7 | 131.2 | 302 ms | 7.6 ms | 15.9 | 16 | 62,332 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,875.2 | 89.6 | 2,018 ms | 11.2 ms | 32.0 | 0 | 86,240 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,419.3 | 53.2 | 3,670 ms | 18.8 ms | 64.0 | 0 | 102,592 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,796.1 | 29.8 | 7,401 ms | 33.6 ms | 128.0 | 0 | 113,920 | 30.0 s |
| Sustained decode | 123K | 1 | Valid | 227.1 | 231.5 | 473 ms | 4.3 ms | 1.0 | 1 | 6,812 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 415.8 | 212.8 | 541 ms | 4.7 ms | 2.0 | 2 | 12,472 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 709.4 | 180.4 | 517 ms | 5.5 ms | 4.0 | 4 | 21,282 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 1,063.6 | 135.6 | 573 ms | 7.4 ms | 8.0 | 8 | 31,905 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,497.9 | 93.5 | 2,692 ms | 10.7 ms | 16.0 | 0 | 44,928 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,859.2 | 57.6 | 4,997 ms | 17.4 ms | 32.0 | 0 | 55,776 | 30.0 s |
| Sustained decode | 123K | 64 | Valid | 2,047.6 | 31.8 | 8,685 ms | 31.5 ms | 64.0 | 0 | 61,439 | 30.0 s |
| Sustained decode | 123K | 128 | Valid | 2,151.0 | 16.9 | 16,966 ms | 59.2 ms | 128.0 | 0 | 64,513 | 30.0 s |