Speed test matrix
gemma-4-E4B-it on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/gemma-4-E4B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 131072 --gpu-memory-utilization 0.853675 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser gemma4 --limit-mm-per-prompt '{"image":4,"audio":1}' --async-schedulingPrefill
prompt tok/s by context length8K context40.6K202 ms TTFT
16K context38.7K420 ms TTFT
32K context36.4K890 ms TTFT
64K context30.8K2,097 ms TTFT
123K context22.3K5,554 ms TTFT
128K context22.4K5,765 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 179.679 ms TTFT | 353.8102 ms TTFT | 683.6144 ms TTFT | 1,266.8111 ms TTFT | 2,263.1221 ms TTFT | 3,721.8376 ms TTFT | 5,843.4697 ms TTFT | 7,744.31,084 ms TTFT |
| 16K | 177.6103 ms TTFT | 346.0164 ms TTFT | 657.1156 ms TTFT | 1,184.4240 ms TTFT | 2,016.4232 ms TTFT | 3,154.5603 ms TTFT | 4,491.2910 ms TTFT | 5,487.11,688 ms TTFT |
| 32K | 173.1147 ms TTFT | 332.1180 ms TTFT | 613.5206 ms TTFT | 1,048.3251 ms TTFT | 1,687.2630 ms TTFT | 2,381.51,119 ms TTFT | 2,956.41,684 ms TTFT | 3,477.03,409 ms TTFT |
| 64K | 165.5234 ms TTFT | 307.6283 ms TTFT | 537.6272 ms TTFT | 862.9668 ms TTFT | 1,249.81,140 ms TTFT | 1,591.22,017 ms TTFT | 1,836.94,000 ms TTFT | Over capacity |
| 123K | 151.8465 ms TTFT | 268.0493 ms TTFT | 443.5773 ms TTFT | 641.81,284 ms TTFT | 791.52,267 ms TTFT | 980.74,113 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 40.6K | — | 202 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 38.7K | — | 420 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 36.4K | — | 890 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.8K | — | 2,097 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 22.3K | — | 5,554 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.4K | — | 5,765 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 179.6 | 180.3 | 79 ms | 5.5 ms | 1.0 | 1 | 5,386 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 353.8 | 177.9 | 102 ms | 5.6 ms | 2.0 | 2 | 10,614 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 683.6 | 172.2 | 144 ms | 5.8 ms | 4.0 | 4 | 20,504 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,266.8 | 159.9 | 111 ms | 6.3 ms | 8.0 | 8 | 38,003 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,263.1 | 145.1 | 221 ms | 6.9 ms | 15.9 | 16 | 67,893 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,721.8 | 117.3 | 376 ms | 8.5 ms | 32.0 | 0 | 111,636 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,843.4 | 92.1 | 697 ms | 10.9 ms | 64.0 | 0 | 175,296 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,744.3 | 61.2 | 1,084 ms | 16.3 ms | 128.0 | 0 | 232,304 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 177.6 | 178.7 | 103 ms | 5.6 ms | 1.0 | 1 | 5,329 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 346.0 | 174.8 | 164 ms | 5.7 ms | 2.0 | 2 | 10,377 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 657.1 | 165.9 | 156 ms | 6.0 ms | 4.0 | 4 | 19,713 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,184.4 | 150.1 | 240 ms | 6.7 ms | 7.9 | 8 | 35,530 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,016.4 | 128.4 | 232 ms | 7.8 ms | 16.0 | 16 | 60,492 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,154.5 | 99.4 | 603 ms | 10.1 ms | 32.0 | 0 | 94,624 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,491.2 | 70.6 | 910 ms | 14.2 ms | 64.0 | 0 | 134,711 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,487.1 | 43.5 | 1,688 ms | 23.0 ms | 128.0 | 0 | 164,605 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 173.1 | 174.5 | 147 ms | 5.7 ms | 1.0 | 1 | 5,194 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 332.1 | 167.5 | 180 ms | 6.0 ms | 2.0 | 2 | 9,963 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 613.5 | 155.0 | 206 ms | 6.5 ms | 3.9 | 4 | 18,403 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,048.3 | 133.3 | 251 ms | 7.5 ms | 8.0 | 8 | 31,447 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,687.2 | 106.1 | 630 ms | 9.4 ms | 16.0 | 0 | 50,617 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,381.5 | 74.7 | 1,119 ms | 13.4 ms | 32.0 | 0 | 71,452 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,956.4 | 47.5 | 1,684 ms | 21.0 ms | 64.0 | 0 | 88,684 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,477.0 | 27.3 | 3,409 ms | 36.6 ms | 128.0 | 0 | 104,280 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 165.5 | 167.5 | 234 ms | 6.0 ms | 1.0 | 1 | 4,965 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 307.6 | 155.9 | 283 ms | 6.4 ms | 2.0 | 2 | 9,226 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 537.6 | 136.1 | 272 ms | 7.3 ms | 4.0 | 4 | 16,127 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 862.9 | 108.0 | 668 ms | 9.3 ms | 8.0 | 0 | 25,880 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,249.8 | 78.1 | 1,140 ms | 12.8 ms | 16.0 | 0 | 37,487 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,591.2 | 49.7 | 2,017 ms | 20.1 ms | 32.0 | 0 | 47,719 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,836.9 | 28.7 | 4,000 ms | 34.8 ms | 64.0 | 0 | 55,085 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 151.8 | 155.5 | 465 ms | 6.4 ms | 1.0 | 1 | 4,555 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 268.0 | 137.4 | 493 ms | 7.3 ms | 2.0 | 2 | 8,039 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 443.5 | 111.0 | 773 ms | 9.0 ms | 4.0 | 0 | 13,304 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 641.8 | 80.3 | 1,284 ms | 12.5 ms | 8.0 | 0 | 19,248 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 791.5 | 52.6 | 2,267 ms | 19.0 ms | 16.0 | 0 | 23,744 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 980.7 | 30.7 | 4,113 ms | 32.6 ms | 32.0 | 0 | 29,407 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.