Speed test matrix
gemma-4-E2B-it on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-gemma-vllm/bin/vllm" serve google/gemma-4-E2B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 131072 --gpu-memory-utilization 0.854015 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context57K144 ms TTFT
16K context55.5K293 ms TTFT
32K context51.1K633 ms TTFT
64K context40.5K1,594 ms TTFT
123K context27.4K4,520 ms TTFT
128K context26.4K4,879 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 260.284 ms TTFT | 516.282 ms TTFT | 1,004.0100 ms TTFT | 1,903.8149 ms TTFT | 3,528.8204 ms TTFT | 5,972.2279 ms TTFT | 9,011.6332 ms TTFT | 13.2K1,055 ms TTFT |
| 16K | 256.1104 ms TTFT | 506.6143 ms TTFT | 975.7134 ms TTFT | 1,811.8251 ms TTFT | 3,244.4249 ms TTFT | 5,169.9208 ms TTFT | 7,296.6940 ms TTFT | 9,859.51,630 ms TTFT |
| 32K | 250.6153 ms TTFT | 491.3140 ms TTFT | 926.1209 ms TTFT | 1,644.3368 ms TTFT | 2,770.6205 ms TTFT | 4,038.3384 ms TTFT | 5,363.82,034 ms TTFT | 6,483.43,482 ms TTFT |
| 64K | 242.7212 ms TTFT | 464.2275 ms TTFT | 841.7288 ms TTFT | 1,397.2286 ms TTFT | 2,072.8294 ms TTFT | 2,880.22,016 ms TTFT | 3,424.33,746 ms TTFT | 3,799.97,275 ms TTFT |
| 123K | 227.8419 ms TTFT | 418.1496 ms TTFT | 713.6459 ms TTFT | 1,065.2603 ms TTFT | 1,500.92,215 ms TTFT | 1,861.94,063 ms TTFT | 2,047.58,044 ms TTFT | 2,150.715,922 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 57K | — | 144 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 55.5K | — | 293 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 51.1K | — | 633 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 40.5K | — | 1,594 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 27.4K | — | 4,520 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 26.4K | — | 4,879 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 260.2 | 264.2 | 84 ms | 3.8 ms | 1.0 | 2 | 7,805 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 516.2 | 260.9 | 82 ms | 3.8 ms | 2.0 | 4 | 15,484 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,004.0 | 255.4 | 100 ms | 3.9 ms | 4.0 | 8 | 30,117 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,903.8 | 240.9 | 149 ms | 4.2 ms | 8.0 | 16 | 57,116 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,528.8 | 223.3 | 204 ms | 4.5 ms | 15.8 | 16 | 105,861 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 5,972.2 | 192.4 | 279 ms | 5.2 ms | 32.0 | 32 | 179,170 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 9,011.6 | 148.9 | 332 ms | 6.7 ms | 63.7 | 64 | 270,349 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 13.2K | 104.5 | 1,055 ms | 9.6 ms | 128.0 | 0 | 394,783 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 256.1 | 258.1 | 104 ms | 3.9 ms | 1.0 | 2 | 7,684 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 506.6 | 258.1 | 143 ms | 3.9 ms | 2.0 | 4 | 15,198 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 975.7 | 246.1 | 134 ms | 4.1 ms | 4.0 | 8 | 29,270 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,811.8 | 229.3 | 251 ms | 4.4 ms | 8.0 | 8 | 54,349 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,244.4 | 205.8 | 249 ms | 4.9 ms | 16.0 | 16 | 97,321 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,169.9 | 165.0 | 208 ms | 6.1 ms | 31.8 | 32 | 155,099 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,296.6 | 115.5 | 940 ms | 8.7 ms | 64.0 | 0 | 218,900 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 9,859.5 | 77.6 | 1,630 ms | 12.9 ms | 128.0 | 0 | 295,808 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 250.6 | 253.2 | 153 ms | 3.9 ms | 1.0 | 2 | 7,518 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 491.3 | 248.3 | 140 ms | 4.0 ms | 1.9 | 4 | 14,739 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 926.1 | 233.4 | 209 ms | 4.3 ms | 3.9 | 4 | 27,781 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,644.3 | 208.8 | 368 ms | 4.8 ms | 8.0 | 8 | 49,323 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,770.6 | 174.3 | 205 ms | 5.7 ms | 15.9 | 16 | 83,116 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,038.3 | 127.7 | 384 ms | 7.8 ms | 31.8 | 32 | 121,127 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,363.8 | 83.7 | 2,034 ms | 12.0 ms | 64.0 | 0 | 160,896 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,483.4 | 50.8 | 3,482 ms | 19.7 ms | 128.0 | 0 | 194,432 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 242.7 | 245.0 | 212 ms | 4.1 ms | 1.0 | 1 | 7,281 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 464.2 | 234.6 | 275 ms | 4.3 ms | 2.0 | 2 | 13,924 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 841.7 | 212.9 | 288 ms | 4.7 ms | 3.9 | 4 | 25,248 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,397.2 | 175.7 | 286 ms | 5.7 ms | 7.9 | 8 | 41,915 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,072.8 | 131.0 | 294 ms | 7.6 ms | 15.9 | 16 | 62,183 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,880.2 | 89.3 | 2,016 ms | 11.2 ms | 32.0 | 0 | 86,400 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,424.3 | 53.1 | 3,746 ms | 18.8 ms | 64.0 | 0 | 102,720 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 3,799.9 | 29.8 | 7,275 ms | 33.6 ms | 128.0 | 0 | 114,048 | 30.0 s |
| Sustained decode | 123K | 1 | Valid | 227.8 | 231.8 | 419 ms | 4.3 ms | 1.0 | 1 | 6,835 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 418.1 | 213.1 | 496 ms | 4.7 ms | 2.0 | 2 | 12,543 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 713.6 | 181.3 | 459 ms | 5.5 ms | 4.0 | 4 | 21,407 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 1,065.2 | 135.7 | 603 ms | 7.4 ms | 7.9 | 8 | 31,956 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,500.9 | 93.2 | 2,215 ms | 10.7 ms | 16.0 | 0 | 45,024 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,861.9 | 57.4 | 4,063 ms | 17.4 ms | 32.0 | 0 | 55,840 | 30.0 s |
| Sustained decode | 123K | 64 | Valid | 2,047.5 | 31.7 | 8,044 ms | 31.5 ms | 64.0 | 0 | 61,376 | 30.0 s |
| Sustained decode | 123K | 128 | Valid | 2,150.7 | 16.9 | 15,922 ms | 59.0 ms | 128.0 | 0 | 64,510 | 30.0 s |