Speed test matrix
gemma-4-26B-A4B-it on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/gemma-4-26B-A4B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.878238 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser gemma4 --limit-mm-per-prompt '{"image":4}' --async-schedulingPrefill
prompt tok/s by context length8K context40.2K204 ms TTFT
16K context35K464 ms TTFT
32K context31.6K1,025 ms TTFT
64K context25.4K2,541 ms TTFT
128K context17.3K7,457 ms TTFT
251K context10.7K23,681 ms TTFT
256K context10.4K24,815 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 201.074 ms TTFT | 361.882 ms TTFT | 529.7120 ms TTFT | 842.4198 ms TTFT | 1,461.0259 ms TTFT | 2,136.8422 ms TTFT | 3,008.6670 ms TTFT | 4,389.31,017 ms TTFT |
| 16K | 198.9115 ms TTFT | 358.1126 ms TTFT | 550.8176 ms TTFT | 943.0230 ms TTFT | 1,380.2379 ms TTFT | 2,153.1637 ms TTFT | 2,902.61,045 ms TTFT | Over capacity |
| 32K | 195.8151 ms TTFT | 334.5187 ms TTFT | 522.6184 ms TTFT | 761.8423 ms TTFT | 1,274.5779 ms TTFT | 1,576.61,042 ms TTFT | Over capacity | Over capacity |
| 64K | 188.4242 ms TTFT | 315.3280 ms TTFT | 474.6424 ms TTFT | 700.5703 ms TTFT | 1,177.21,199 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 175.0449 ms TTFT | 262.8497 ms TTFT | 313.2861 ms TTFT | 625.11,388 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 153.5983 ms TTFT | 237.31,403 ms TTFT | 325.62,168 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 40.2K | — | 204 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 35K | — | 464 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.6K | — | 1,025 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.4K | — | 2,541 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.3K | — | 7,457 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.7K | — | 23,681 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.4K | — | 24,815 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 201.0 | 201.7 | 74 ms | 5.0 ms | 1.0 | 1 | 6,031 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 361.8 | 174.2 | 82 ms | 5.7 ms | 2.0 | 2 | 10,855 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 529.7 | 136.1 | 120 ms | 7.4 ms | 4.0 | 4 | 15,887 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 842.4 | 118.6 | 198 ms | 8.4 ms | 8.0 | 0 | 25,279 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,461.0 | 93.6 | 259 ms | 10.7 ms | 16.0 | 0 | 43,827 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,136.8 | 71.9 | 422 ms | 13.9 ms | 32.0 | 0 | 64,093 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,008.6 | 52.1 | 670 ms | 19.2 ms | 64.0 | 0 | 90,245 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,389.3 | 35.7 | 1,017 ms | 28.0 ms | 128.0 | 0 | 131,724 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 198.9 | 480.5 | 115 ms | 3.2 ms | 1.0 | 1 | 5,966 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 358.1 | 174.6 | 126 ms | 5.7 ms | 2.0 | 2 | 10,744 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 550.8 | 135.1 | 176 ms | 7.4 ms | 4.0 | 4 | 16,521 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 943.0 | 117.7 | 230 ms | 8.5 ms | 8.0 | 0 | 28,287 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,380.2 | 92.3 | 379 ms | 10.8 ms | 16.0 | 0 | 41,400 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,153.1 | 68.5 | 637 ms | 14.6 ms | 32.0 | 0 | 64,593 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,902.6 | 47.1 | 1,045 ms | 21.2 ms | 64.0 | 0 | 87,049 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 195.8 | 197.1 | 151 ms | 5.1 ms | 1.0 | 1 | 5,873 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 334.5 | 168.5 | 187 ms | 5.9 ms | 2.0 | 2 | 10,033 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 522.6 | 130.5 | 184 ms | 7.7 ms | 4.0 | 4 | 15,676 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 761.8 | 109.1 | 423 ms | 9.2 ms | 8.0 | 0 | 22,850 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,274.5 | 83.9 | 779 ms | 11.9 ms | 16.0 | 0 | 38,234 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,576.6 | 60.0 | 1,042 ms | 16.7 ms | 32.0 | 0 | 47,282 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 188.4 | 190.5 | 242 ms | 5.2 ms | 1.0 | 1 | 5,651 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 315.3 | 159.5 | 280 ms | 6.3 ms | 2.0 | 2 | 9,458 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 474.6 | 118.8 | 424 ms | 8.4 ms | 4.0 | 0 | 14,236 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 700.5 | 99.7 | 703 ms | 10.0 ms | 8.0 | 0 | 21,014 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,177.2 | 73.0 | 1,199 ms | 13.7 ms | 16.0 | 0 | 35,308 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 175.0 | 180.7 | 449 ms | 5.5 ms | 1.0 | 1 | 4,197 | 24.0 s |
| Sustained decode | 128K | 2 | Valid | 262.8 | 151.1 | 497 ms | 6.6 ms | 2.0 | 2 | 7,883 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 313.2 | 104.7 | 861 ms | 9.5 ms | 4.0 | 0 | 9,399 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 625.1 | 78.0 | 1,388 ms | 12.8 ms | 8.0 | 0 | 18,752 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 153.5 | 160.8 | 983 ms | 6.2 ms | 1.0 | 1 | 4,608 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 237.3 | 120.5 | 1,403 ms | 8.3 ms | 2.0 | 1 | 7,119 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 325.6 | 81.9 | 2,168 ms | 12.2 ms | 4.0 | 0 | 9,767 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.