Speed test matrix
gemma-4-12B-it on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/gemma-4-12B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.849406 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser gemma4 --limit-mm-per-prompt '{"image":4,"audio":1}' --async-schedulingPrefill
prompt tok/s by context length8K context19.4K423 ms TTFT
16K context18.3K887 ms TTFT
32K context16.7K1,933 ms TTFT
64K context13.7K4,721 ms TTFT
128K context9,78513,182 ms TTFT
251K context6,33739,894 ms TTFT
256K context6,27341,104 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 106.166 ms TTFT | 210.0111 ms TTFT | 408.1152 ms TTFT | 778.4186 ms TTFT | 1,426.5277 ms TTFT | 2,370.3402 ms TTFT | 3,491.4779 ms TTFT | Over capacity |
| 16K | 105.5106 ms TTFT | 208.9154 ms TTFT | 403.5202 ms TTFT | 762.3241 ms TTFT | 1,375.3326 ms TTFT | 2,229.4605 ms TTFT | 3,186.71,099 ms TTFT | Over capacity |
| 32K | 104.6151 ms TTFT | 206.2205 ms TTFT | 394.7274 ms TTFT | 730.6398 ms TTFT | 1,279.8547 ms TTFT | 1,988.91,027 ms TTFT | Over capacity | Over capacity |
| 64K | 103.3216 ms TTFT | 201.4319 ms TTFT | 378.1378 ms TTFT | 676.7694 ms TTFT | 1,126.11,201 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 100.5445 ms TTFT | 192.7597 ms TTFT | 349.8911 ms TTFT | 588.41,365 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 95.8998 ms TTFT | 178.21,393 ms TTFT | 305.02,110 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.4K | — | 423 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.3K | — | 887 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16.7K | — | 1,933 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.7K | — | 4,721 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,785.0 | — | 13,182 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,337.0 | — | 39,894 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,273.0 | — | 41,104 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 106.1 | 106.2 | 66 ms | 9.4 ms | 1.0 | 0 | 3,184 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 210.0 | 105.1 | 111 ms | 9.5 ms | 2.0 | 0 | 6,299 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 408.1 | 102.2 | 152 ms | 9.8 ms | 4.0 | 0 | 12,244 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 778.4 | 97.6 | 186 ms | 10.2 ms | 8.0 | 0 | 23,368 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,426.5 | 89.6 | 277 ms | 11.2 ms | 16.0 | 0 | 42,786 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,370.3 | 74.4 | 402 ms | 13.4 ms | 32.0 | 0 | 71,102 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,491.4 | 54.9 | 779 ms | 18.2 ms | 64.0 | 0 | 104,786 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 105.5 | 105.6 | 106 ms | 9.5 ms | 1.0 | 0 | 3,164 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 208.9 | 104.6 | 154 ms | 9.6 ms | 2.0 | 0 | 6,266 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 403.5 | 101.0 | 202 ms | 9.9 ms | 4.0 | 0 | 12,101 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 762.3 | 95.5 | 241 ms | 10.5 ms | 8.0 | 0 | 22,864 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,375.3 | 86.2 | 326 ms | 11.6 ms | 16.0 | 0 | 41,260 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,229.4 | 69.9 | 605 ms | 14.3 ms | 32.0 | 0 | 66,880 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,186.7 | 50.1 | 1,099 ms | 20.0 ms | 64.0 | 0 | 95,616 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 104.6 | 104.8 | 151 ms | 9.5 ms | 1.0 | 0 | 3,137 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 206.2 | 103.3 | 205 ms | 9.7 ms | 2.0 | 0 | 6,186 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 394.7 | 98.8 | 274 ms | 10.1 ms | 4.0 | 0 | 11,840 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 730.6 | 91.5 | 398 ms | 10.9 ms | 8.0 | 0 | 21,917 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,279.8 | 80.0 | 547 ms | 12.5 ms | 16.0 | 0 | 38,384 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,988.9 | 62.1 | 1,027 ms | 16.1 ms | 32.0 | 0 | 59,649 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 103.3 | 103.5 | 216 ms | 9.7 ms | 1.0 | 0 | 3,097 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 201.4 | 100.9 | 319 ms | 9.9 ms | 2.0 | 0 | 6,044 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 378.1 | 94.6 | 378 ms | 10.6 ms | 4.0 | 0 | 11,344 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 676.7 | 84.4 | 694 ms | 11.8 ms | 8.0 | 0 | 20,298 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,126.1 | 69.9 | 1,201 ms | 14.3 ms | 16.0 | 0 | 33,775 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 100.5 | 100.8 | 445 ms | 9.9 ms | 1.0 | 0 | 3,015 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 192.7 | 96.6 | 597 ms | 10.4 ms | 2.0 | 0 | 5,780 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 349.8 | 87.4 | 911 ms | 11.4 ms | 4.0 | 0 | 10,496 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 588.4 | 73.4 | 1,365 ms | 13.6 ms | 8.0 | 0 | 17,664 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 95.8 | 96.3 | 998 ms | 10.4 ms | 1.0 | 0 | 2,874 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 178.2 | 89.5 | 1,393 ms | 11.2 ms | 2.0 | 0 | 5,346 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 305.0 | 76.4 | 2,110 ms | 13.1 ms | 4.0 | 0 | 9,148 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.