Speed test matrix
gemma-4-31B-it on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/gemma-4-31B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.850454 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser gemma4 --limit-mm-per-prompt '{"image":4}' --async-schedulingPrefill
prompt tok/s by context length8K context7,8921,039 ms TTFT
16K context7,5272,160 ms TTFT
32K context6,5894,913 ms TTFT
64K context5,25512,286 ms TTFT
128K context3,72334,649 ms TTFT
251K context2,442103,499 ms TTFT
256K context2,414106,795 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 50.367 ms TTFT | 98.4108 ms TTFT | 189.1159 ms TTFT | 353.0214 ms TTFT | 616.9258 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 49.797 ms TTFT | 96.6148 ms TTFT | 183.9205 ms TTFT | 335.4263 ms TTFT | 566.6460 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 48.8157 ms TTFT | 93.4189 ms TTFT | 173.8272 ms TTFT | 304.1466 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 47.3236 ms TTFT | 87.7313 ms TTFT | 156.7498 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 44.0495 ms TTFT | 77.8607 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 39.3996 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,892.0 | — | 1,039 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,527.0 | — | 2,160 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,589.0 | — | 4,913 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,255.0 | — | 12,286 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,723.0 | — | 34,649 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,442.0 | — | 103,499 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,414.0 | — | 106,795 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 50.3 | 50.2 | 67 ms | 19.9 ms | 1.0 | 0 | 1,508 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 98.4 | 49.2 | 108 ms | 20.3 ms | 2.0 | 0 | 2,951 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 189.1 | 47.3 | 159 ms | 21.1 ms | 4.0 | 0 | 5,670 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 353.0 | 44.2 | 214 ms | 22.6 ms | 8.0 | 0 | 10,592 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 616.9 | 38.7 | 258 ms | 25.8 ms | 16.0 | 0 | 18,505 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 49.7 | 49.7 | 97 ms | 20.1 ms | 1.0 | 0 | 1,490 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 96.6 | 48.4 | 148 ms | 20.7 ms | 2.0 | 0 | 2,898 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 183.9 | 46.0 | 205 ms | 21.7 ms | 4.0 | 0 | 5,519 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 335.4 | 42.0 | 263 ms | 23.8 ms | 8.0 | 0 | 10,061 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 566.6 | 35.5 | 460 ms | 28.1 ms | 16.0 | 0 | 16,989 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 48.8 | 48.8 | 157 ms | 20.5 ms | 1.0 | 0 | 1,463 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 93.4 | 46.8 | 189 ms | 21.4 ms | 2.0 | 0 | 2,800 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 173.8 | 43.5 | 272 ms | 23.0 ms | 4.0 | 0 | 5,210 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 304.1 | 38.1 | 466 ms | 26.3 ms | 8.0 | 0 | 9,120 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 47.3 | 47.3 | 236 ms | 21.2 ms | 1.0 | 0 | 1,416 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 87.7 | 43.9 | 313 ms | 22.8 ms | 2.0 | 0 | 2,630 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 156.7 | 39.3 | 498 ms | 25.5 ms | 4.0 | 0 | 4,701 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 44.0 | 44.2 | 495 ms | 22.6 ms | 1.0 | 0 | 1,321 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 77.8 | 38.9 | 607 ms | 25.7 ms | 2.0 | 0 | 2,333 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 39.3 | 39.5 | 996 ms | 25.3 ms | 1.0 | 0 | 1,178 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.