Speed test matrix
gemma-4-12B-it-NVFP4 on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/gemma-4-12B-it-NVFP4 --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.850077 --host 0.0.0.0 --port 8000 --api-key <redacted> --reasoning-parser gemma4 --limit-mm-per-prompt '{"image":4,"audio":1}' --async-schedulingPrefill
prompt tok/s by context length8K context11.2K733 ms TTFT
16K context10.9K1,497 ms TTFT
32K context10.3K3,147 ms TTFT
64K context9,1317,071 ms TTFT
128K context7,28117,716 ms TTFT
251K context5,20948,527 ms TTFT
256K context5,14350,127 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 142.866 ms TTFT | 283.875 ms TTFT | 548.1119 ms TTFT | 1,011.8196 ms TTFT | 1,746.6274 ms TTFT | 2,647.5396 ms TTFT | 3,338.3694 ms TTFT | 3,875.81,108 ms TTFT |
| 16K | 142.495 ms TTFT | 279.8115 ms TTFT | 538.5163 ms TTFT | 981.5219 ms TTFT | 1,666.5402 ms TTFT | 2,470.8629 ms TTFT | 3,059.71,020 ms TTFT | Over capacity |
| 32K | 140.2152 ms TTFT | 274.4169 ms TTFT | 522.0197 ms TTFT | 931.8415 ms TTFT | 1,531.5672 ms TTFT | 2,181.8885 ms TTFT | Over capacity | Over capacity |
| 64K | 137.4222 ms TTFT | 265.2274 ms TTFT | 491.9275 ms TTFT | 849.4748 ms TTFT | 1,311.11,210 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 131.6461 ms TTFT | 248.1494 ms TTFT | 446.1893 ms TTFT | 715.31,449 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 121.81,004 ms TTFT | 228.51,414 ms TTFT | 379.22,222 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.2K | — | 733 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.9K | — | 1,497 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.3K | — | 3,147 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,131.0 | — | 7,071 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,281.0 | — | 17,716 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 5,209.0 | — | 48,527 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 5,143.0 | — | 50,127 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 142.8 | 143.4 | 66 ms | 7.0 ms | 1.0 | 1 | 4,283 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 283.8 | 142.5 | 75 ms | 7.0 ms | 2.0 | 2 | 8,516 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 548.1 | 138.4 | 119 ms | 7.2 ms | 3.9 | 4 | 16,443 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,011.8 | 128.1 | 196 ms | 7.8 ms | 8.0 | 8 | 30,349 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,746.6 | 109.8 | 274 ms | 9.1 ms | 16.0 | 0 | 52,399 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,647.5 | 83.1 | 396 ms | 12.0 ms | 32.0 | 0 | 79,429 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,338.3 | 52.4 | 694 ms | 19.1 ms | 64.0 | 0 | 100,163 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,875.8 | 30.5 | 1,108 ms | 32.8 ms | 128.0 | 0 | 116,238 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 142.4 | 143.5 | 95 ms | 7.0 ms | 1.0 | 1 | 4,271 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 279.8 | 141.4 | 115 ms | 7.1 ms | 2.0 | 2 | 8,398 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 538.5 | 136.3 | 163 ms | 7.3 ms | 4.0 | 4 | 16,152 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 981.5 | 123.7 | 219 ms | 8.1 ms | 8.0 | 8 | 29,445 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,666.5 | 104.7 | 402 ms | 9.6 ms | 16.0 | 0 | 49,999 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,470.8 | 77.4 | 629 ms | 12.9 ms | 32.0 | 0 | 74,112 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,059.7 | 48.0 | 1,020 ms | 20.8 ms | 64.0 | 0 | 91,787 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 140.2 | 141.7 | 152 ms | 7.1 ms | 1.0 | 1 | 4,207 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 274.4 | 138.8 | 169 ms | 7.2 ms | 2.0 | 2 | 8,229 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 522.0 | 131.6 | 197 ms | 7.6 ms | 4.0 | 4 | 15,657 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 931.8 | 116.8 | 415 ms | 8.6 ms | 8.0 | 0 | 27,952 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,531.5 | 95.8 | 672 ms | 10.4 ms | 16.0 | 0 | 45,936 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,181.8 | 68.0 | 885 ms | 14.7 ms | 32.0 | 0 | 65,440 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 137.4 | 139.8 | 222 ms | 7.2 ms | 1.0 | 1 | 4,120 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 265.2 | 135.3 | 274 ms | 7.4 ms | 1.9 | 2 | 7,955 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 491.9 | 124.2 | 275 ms | 8.1 ms | 4.0 | 4 | 14,755 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 849.4 | 106.0 | 748 ms | 9.4 ms | 8.0 | 0 | 25,480 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,311.1 | 81.6 | 1,210 ms | 12.3 ms | 16.0 | 0 | 39,330 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 131.6 | 136.9 | 461 ms | 7.3 ms | 1.0 | 1 | 3,947 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 248.1 | 128.9 | 494 ms | 7.8 ms | 2.0 | 2 | 7,446 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 446.1 | 111.7 | 893 ms | 9.0 ms | 4.0 | 0 | 13,388 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 715.3 | 89.4 | 1,449 ms | 11.2 ms | 8.0 | 0 | 21,456 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 121.8 | 251.7 | 1,004 ms | 5.3 ms | 1.0 | 1 | 3,654 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 228.5 | 114.8 | 1,414 ms | 8.7 ms | 2.0 | 0 | 6,856 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 379.2 | 95.0 | 2,222 ms | 10.5 ms | 4.0 | 0 | 11,380 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.