Speed test matrix
gemma-4-26B-A4B-it-NVFP4 on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/gemma-4-26B-A4B-it-NVFP4 --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.878533 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser gemma4 --limit-mm-per-prompt '{"image":4}' --async-schedulingPrefill
prompt tok/s by context length8K context24.7K332 ms TTFT
16K context23.5K690 ms TTFT
32K context22.5K1,440 ms TTFT
64K context19.2K3,362 ms TTFT
128K context14.2K9,086 ms TTFT
251K context9,39326,912 ms TTFT
256K context9,19428,041 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 205.469 ms TTFT | 304.0100 ms TTFT | 710.2102 ms TTFT | 1,192.9104 ms TTFT | 1,890.2255 ms TTFT | 2,601.1394 ms TTFT | 3,371.3651 ms TTFT | 4,355.91,126 ms TTFT |
| 16K | 203.7104 ms TTFT | 377.0128 ms TTFT | 678.6159 ms TTFT | 1,133.4155 ms TTFT | 1,813.0390 ms TTFT | 2,315.7595 ms TTFT | 2,886.4944 ms TTFT | 3,922.71,717 ms TTFT |
| 32K | 198.8150 ms TTFT | 367.2165 ms TTFT | 636.7229 ms TTFT | 1,058.9340 ms TTFT | 1,486.1676 ms TTFT | 1,917.01,172 ms TTFT | 2,333.31,713 ms TTFT | Over capacity |
| 64K | 192.3219 ms TTFT | 347.6260 ms TTFT | 595.1252 ms TTFT | 934.0624 ms TTFT | 1,291.71,165 ms TTFT | 1,513.52,048 ms TTFT | Over capacity | Over capacity |
| 128K | 191.1419 ms TTFT | 306.3491 ms TTFT | 488.8640 ms TTFT | 717.11,438 ms TTFT | 937.52,534 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 157.3964 ms TTFT | 251.01,068 ms TTFT | 377.82,108 ms TTFT | 504.43,554 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 24.7K | — | 332 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 23.5K | — | 690 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.5K | — | 1,440 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 19.2K | — | 3,362 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 14.2K | — | 9,086 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 9,393.0 | — | 26,912 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,194.0 | — | 28,041 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 205.4 | 206.3 | 69 ms | 4.8 ms | 1.0 | 1 | 6,162 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 304.0 | 194.4 | 100 ms | 5.1 ms | 2.0 | 2 | 9,120 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 710.2 | 171.7 | 102 ms | 5.8 ms | 4.0 | 4 | 21,305 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,192.9 | 148.6 | 104 ms | 6.7 ms | 8.0 | 8 | 35,787 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,890.2 | 118.0 | 255 ms | 8.5 ms | 16.0 | 0 | 56,704 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,601.1 | 83.2 | 394 ms | 12.0 ms | 32.0 | 0 | 78,036 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,371.3 | 54.6 | 651 ms | 18.3 ms | 64.0 | 0 | 101,119 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,355.9 | 37.7 | 1,126 ms | 26.5 ms | 128.0 | 0 | 130,664 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 203.7 | 204.6 | 104 ms | 4.9 ms | 1.0 | 1 | 6,110 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 377.0 | 189.9 | 128 ms | 5.3 ms | 2.0 | 2 | 11,311 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 678.6 | 171.1 | 159 ms | 5.8 ms | 4.0 | 4 | 20,357 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,133.4 | 146.1 | 155 ms | 6.8 ms | 8.0 | 8 | 33,999 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,813.0 | 112.1 | 390 ms | 8.9 ms | 16.0 | 0 | 54,383 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,315.7 | 77.7 | 595 ms | 12.9 ms | 32.0 | 0 | 69,474 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,886.4 | 49.3 | 944 ms | 20.3 ms | 64.0 | 0 | 86,557 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,922.7 | 32.1 | 1,717 ms | 31.1 ms | 128.0 | 0 | 117,649 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 198.8 | 200.1 | 150 ms | 5.0 ms | 1.0 | 1 | 5,964 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 367.2 | 185.0 | 165 ms | 5.4 ms | 2.0 | 2 | 11,015 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 636.7 | 160.8 | 229 ms | 6.2 ms | 3.9 | 4 | 19,098 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,058.9 | 132.3 | 340 ms | 7.6 ms | 8.0 | 8 | 31,761 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,486.1 | 99.3 | 676 ms | 10.1 ms | 16.0 | 0 | 44,580 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,917.0 | 66.1 | 1,172 ms | 15.1 ms | 32.0 | 0 | 57,506 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,333.3 | 39.8 | 1,713 ms | 25.1 ms | 64.0 | 0 | 69,969 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 192.3 | 194.4 | 219 ms | 5.1 ms | 1.0 | 1 | 5,773 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 347.6 | 175.3 | 260 ms | 5.7 ms | 2.0 | 2 | 10,426 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 595.1 | 146.0 | 252 ms | 6.8 ms | 4.0 | 4 | 17,852 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 934.0 | 116.5 | 624 ms | 8.6 ms | 8.0 | 0 | 28,016 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,291.7 | 80.6 | 1,165 ms | 12.4 ms | 16.0 | 0 | 38,751 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,513.5 | 50.1 | 2,048 ms | 19.9 ms | 32.0 | 0 | 45,382 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 191.1 | 1871.9 | 419 ms | 2.9 ms | 1.0 | 1 | 5,736 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 306.3 | 156.3 | 491 ms | 6.4 ms | 1.9 | 2 | 9,188 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 488.8 | 124.4 | 640 ms | 8.0 ms | 4.0 | 4 | 14,646 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 717.1 | 89.7 | 1,438 ms | 11.1 ms | 8.0 | 0 | 21,513 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 937.5 | 58.2 | 2,534 ms | 17.2 ms | 16.0 | 0 | 28,126 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 157.3 | 164.1 | 964 ms | 6.1 ms | 1.0 | 1 | 4,718 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 251.0 | 142.8 | 1,068 ms | 7.1 ms | 1.9 | 2 | 7,531 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 377.8 | 94.9 | 2,108 ms | 10.5 ms | 4.0 | 0 | 11,331 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 504.4 | 64.2 | 3,554 ms | 15.6 ms | 8.0 | 0 | 15,137 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.