Speed test matrix
gemma-4-E4B-it on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-gemma-vllm/bin/vllm" serve google/gemma-4-E4B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 131072 --gpu-memory-utilization 0.853675 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context41.5K198 ms TTFT
16K context38.6K421 ms TTFT
32K context36.9K877 ms TTFT
64K context30.8K2,095 ms TTFT
123K context22.3K5,557 ms TTFT
128K context22.2K5,815 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 180.186 ms TTFT | 355.2100 ms TTFT | 685.2128 ms TTFT | 1,272.0154 ms TTFT | 2,273.8179 ms TTFT | 3,514.8392 ms TTFT | 5,717.2933 ms TTFT | 7,759.31,060 ms TTFT |
| 16K | 177.7116 ms TTFT | 347.5127 ms TTFT | 659.2168 ms TTFT | 1,188.5197 ms TTFT | 2,022.4242 ms TTFT | 3,168.9662 ms TTFT | 4,494.71,212 ms TTFT | 5,519.21,801 ms TTFT |
| 32K | 173.6143 ms TTFT | 333.7172 ms TTFT | 613.3200 ms TTFT | 1,047.9259 ms TTFT | 1,693.3577 ms TTFT | 2,394.51,114 ms TTFT | 3,040.51,733 ms TTFT | 3,474.53,287 ms TTFT |
| 64K | 165.3239 ms TTFT | 307.7291 ms TTFT | 538.4289 ms TTFT | 862.5674 ms TTFT | 1,195.51,206 ms TTFT | 1,549.32,103 ms TTFT | 1,801.83,771 ms TTFT | Over capacity |
| 123K | 152.4449 ms TTFT | 267.7497 ms TTFT | 324.4810 ms TTFT | 641.31,322 ms TTFT | 849.42,262 ms TTFT | 986.14,085 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 41.5K | — | 198 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 38.6K | — | 421 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 36.9K | — | 877 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 30.8K | — | 2,095 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 22.3K | — | 5,557 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.2K | — | 5,815 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 180.1 | 181.0 | 86 ms | 5.5 ms | 1.0 | 1 | 5,403 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 355.2 | 178.8 | 100 ms | 5.6 ms | 2.0 | 2 | 10,661 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 685.2 | 172.6 | 128 ms | 5.8 ms | 4.0 | 4 | 20,554 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,272.0 | 160.9 | 154 ms | 6.2 ms | 7.9 | 8 | 38,158 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,273.8 | 145.2 | 179 ms | 6.9 ms | 15.8 | 16 | 68,216 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,514.8 | 117.5 | 392 ms | 8.5 ms | 32.0 | 0 | 105,445 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,717.2 | 92.5 | 933 ms | 10.8 ms | 64.0 | 0 | 171,484 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,759.3 | 61.2 | 1,060 ms | 16.3 ms | 128.0 | 0 | 232,742 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 177.7 | 178.8 | 116 ms | 5.6 ms | 1.0 | 1 | 5,330 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 347.5 | 174.9 | 127 ms | 5.7 ms | 2.0 | 2 | 10,423 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 659.2 | 166.3 | 168 ms | 6.0 ms | 3.9 | 4 | 19,775 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,188.5 | 150.6 | 197 ms | 6.6 ms | 7.9 | 8 | 35,661 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,022.4 | 129.5 | 242 ms | 7.7 ms | 16.0 | 16 | 60,665 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,168.9 | 99.5 | 662 ms | 10.1 ms | 32.0 | 0 | 95,069 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,494.7 | 70.5 | 1,212 ms | 14.2 ms | 64.0 | 0 | 134,835 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,519.2 | 43.6 | 1,801 ms | 23.0 ms | 128.0 | 0 | 165,493 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 173.6 | 175.1 | 143 ms | 5.7 ms | 1.0 | 1 | 5,208 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 333.7 | 168.1 | 172 ms | 6.0 ms | 2.0 | 2 | 10,010 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 613.3 | 155.0 | 200 ms | 6.5 ms | 3.9 | 4 | 18,395 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,047.9 | 133.2 | 259 ms | 7.5 ms | 8.0 | 8 | 31,432 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,693.3 | 106.0 | 577 ms | 9.4 ms | 16.0 | 0 | 50,816 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,394.5 | 75.0 | 1,114 ms | 13.3 ms | 32.0 | 0 | 71,840 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,040.5 | 47.9 | 1,733 ms | 20.9 ms | 64.0 | 0 | 91,186 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,474.5 | 27.4 | 3,287 ms | 36.5 ms | 128.0 | 0 | 104,226 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 165.3 | 167.5 | 239 ms | 6.0 ms | 1.0 | 1 | 4,958 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 307.7 | 156.0 | 291 ms | 6.4 ms | 2.0 | 2 | 9,229 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 538.4 | 136.1 | 289 ms | 7.3 ms | 4.0 | 4 | 16,149 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 862.5 | 107.9 | 674 ms | 9.3 ms | 8.0 | 0 | 25,888 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,195.5 | 78.5 | 1,206 ms | 12.7 ms | 16.0 | 0 | 35,855 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,549.3 | 49.8 | 2,103 ms | 20.1 ms | 32.0 | 0 | 46,463 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,801.8 | 28.8 | 3,771 ms | 34.7 ms | 64.0 | 0 | 54,048 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 152.4 | 156.0 | 449 ms | 6.4 ms | 1.0 | 1 | 4,572 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 267.7 | 137.4 | 497 ms | 7.3 ms | 2.0 | 2 | 8,032 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 324.4 | 111.0 | 810 ms | 9.0 ms | 4.0 | 0 | 9,730 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 641.3 | 80.2 | 1,322 ms | 12.5 ms | 8.0 | 0 | 19,239 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 849.4 | 52.9 | 2,262 ms | 18.9 ms | 16.0 | 0 | 25,472 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 986.1 | 30.7 | 4,085 ms | 32.6 ms | 32.0 | 0 | 29,565 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.