Speed test matrix
gemma-4-31B-it-FP8-block on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/gemma-4-31B-it-FP8-block --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.846765 --host 0.0.0.0 --port 8000 --api-key <redacted> --limit-mm-per-prompt '{"image":4,"audio":1}' --async-schedulingPrefill
prompt tok/s by context length8K context11.4K721 ms TTFT
16K context10.3K1,577 ms TTFT
32K context8,6813,729 ms TTFT
64K context6,5009,933 ms TTFT
128K context4,33029,789 ms TTFT
251K context2,69593,791 ms TTFT
256K context2,65797,025 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 74.761 ms TTFT | 144.3119 ms TTFT | 272.0159 ms TTFT | 497.5240 ms TTFT | 822.0281 ms TTFT | 1,189.1463 ms TTFT | Over capacity | Over capacity |
| 16K | 73.7110 ms TTFT | 140.9132 ms TTFT | 261.0210 ms TTFT | 463.1302 ms TTFT | 735.4466 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 71.7152 ms TTFT | 134.0205 ms TTFT | 241.7282 ms TTFT | 406.0396 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 68.2223 ms TTFT | 122.3313 ms TTFT | 209.3450 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 61.9447 ms TTFT | 103.7600 ms TTFT | 165.4903 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 52.81,017 ms TTFT | 80.91,394 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.4K | — | 721 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.3K | — | 1,577 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,681.0 | — | 3,729 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,500.0 | — | 9,933 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,330.0 | — | 29,789 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,695.0 | — | 93,791 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,657.0 | — | 97,025 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 74.7 | 74.8 | 61 ms | 13.4 ms | 1.0 | 0 | 2,242 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 144.3 | 72.3 | 119 ms | 13.8 ms | 2.0 | 0 | 4,329 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 272.0 | 68.2 | 159 ms | 14.7 ms | 4.0 | 0 | 8,160 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 497.5 | 62.5 | 240 ms | 16.0 ms | 8.0 | 0 | 14,931 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 822.0 | 51.6 | 281 ms | 19.4 ms | 16.0 | 0 | 24,666 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,189.1 | 37.4 | 463 ms | 26.7 ms | 32.0 | 0 | 35,674 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 73.7 | 73.8 | 110 ms | 13.6 ms | 1.0 | 0 | 2,211 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 140.9 | 70.5 | 132 ms | 14.2 ms | 2.0 | 0 | 4,228 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 261.0 | 65.5 | 210 ms | 15.3 ms | 4.0 | 0 | 7,827 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 463.1 | 58.2 | 302 ms | 17.2 ms | 8.0 | 0 | 13,887 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 735.4 | 46.2 | 466 ms | 21.6 ms | 16.0 | 0 | 22,060 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 71.7 | 71.8 | 152 ms | 13.9 ms | 1.0 | 0 | 2,149 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 134.0 | 67.1 | 205 ms | 14.9 ms | 2.0 | 0 | 4,020 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 241.7 | 60.6 | 282 ms | 16.5 ms | 4.0 | 0 | 7,251 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 406.0 | 50.9 | 396 ms | 19.6 ms | 8.0 | 0 | 12,178 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 68.2 | 68.3 | 223 ms | 14.6 ms | 1.0 | 0 | 2,045 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 122.3 | 61.3 | 313 ms | 16.3 ms | 2.0 | 0 | 3,666 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 209.3 | 52.5 | 450 ms | 19.1 ms | 4.0 | 0 | 6,276 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 61.9 | 62.1 | 447 ms | 16.1 ms | 1.0 | 0 | 1,858 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 103.7 | 52.0 | 600 ms | 19.2 ms | 2.0 | 0 | 3,112 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 165.4 | 41.5 | 903 ms | 24.1 ms | 4.0 | 0 | 4,959 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 52.8 | 53.1 | 1,017 ms | 18.8 ms | 1.0 | 0 | 1,583 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 80.9 | 40.6 | 1,394 ms | 24.6 ms | 2.0 | 0 | 2,426 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.