Speed test matrix
gemma-4-31B-it-NVFP4 on H200 NVL 141GB
vllm · 0.27.2rc1.dev113+g5cecfc013 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.27.2rc1.dev113+g5cecfc013
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
5cecfc013750
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-vllm/bin/vllm" serve RedHatAI/gemma-4-31B-it-NVFP4 --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.850957 --host 0.0.0.0 --port 8000 --api-key <redacted> --limit-mm-per-prompt '{"image":4,"audio":1}' --async-schedulingPrefill
prompt tok/s by context length8K context4,4861,829 ms TTFT
16K context4,3683,723 ms TTFT
32K context4,0258,041 ms TTFT
64K context3,51718,360 ms TTFT
128K context2,76546,648 ms TTFT
251K context1,997126,593 ms TTFT
256K context1,975130,522 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 79.148 ms TTFT | 152.9104 ms TTFT | 286.6158 ms TTFT | 518.1225 ms TTFT | 817.1267 ms TTFT | 1,143.4459 ms TTFT | Over capacity | Over capacity |
| 16K | 77.2110 ms TTFT | 148.0157 ms TTFT | 272.9207 ms TTFT | 479.7299 ms TTFT | 730.3478 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 75.0152 ms TTFT | 140.5206 ms TTFT | 251.9286 ms TTFT | 418.7433 ms TTFT | 602.1581 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 71.1224 ms TTFT | 127.7317 ms TTFT | 217.2473 ms TTFT | 336.4820 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 64.5450 ms TTFT | 108.4628 ms TTFT | 171.5891 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 54.5999 ms TTFT | 83.51,401 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,486.0 | — | 1,829 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,368.0 | — | 3,723 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,025.0 | — | 8,041 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,517.0 | — | 18,360 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,765.0 | — | 46,648 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,997.0 | — | 126,593 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,975.0 | — | 130,522 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 79.1 | 79.1 | 48 ms | 12.6 ms | 1.0 | 0 | 2,373 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 152.9 | 76.5 | 104 ms | 13.1 ms | 2.0 | 0 | 4,588 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 286.6 | 71.9 | 158 ms | 13.9 ms | 4.0 | 0 | 8,596 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 518.1 | 65.1 | 225 ms | 15.4 ms | 8.0 | 0 | 15,537 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 817.1 | 51.4 | 267 ms | 19.5 ms | 16.0 | 0 | 24,528 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,143.4 | 35.9 | 459 ms | 27.9 ms | 32.0 | 0 | 34,281 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 77.2 | 77.3 | 110 ms | 12.9 ms | 1.0 | 0 | 2,314 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 148.0 | 74.1 | 157 ms | 13.5 ms | 2.0 | 0 | 4,440 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 272.9 | 68.5 | 207 ms | 14.6 ms | 4.0 | 0 | 8,184 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 479.7 | 60.2 | 299 ms | 16.6 ms | 8.0 | 0 | 14,393 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 730.3 | 45.9 | 478 ms | 21.8 ms | 16.0 | 0 | 21,908 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 75.0 | 75.1 | 152 ms | 13.3 ms | 1.0 | 0 | 2,251 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 140.5 | 70.4 | 206 ms | 14.2 ms | 2.0 | 0 | 4,215 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 251.9 | 63.2 | 286 ms | 15.8 ms | 4.0 | 0 | 7,554 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 418.7 | 52.5 | 433 ms | 19.1 ms | 8.0 | 0 | 12,560 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 602.1 | 37.7 | 581 ms | 26.6 ms | 16.0 | 0 | 18,052 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 71.1 | 71.2 | 224 ms | 14.0 ms | 1.0 | 0 | 2,132 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 127.7 | 64.0 | 317 ms | 15.6 ms | 2.0 | 0 | 3,828 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 217.2 | 54.4 | 473 ms | 18.4 ms | 4.0 | 0 | 6,516 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 336.4 | 42.0 | 820 ms | 23.8 ms | 8.0 | 0 | 10,092 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 64.5 | 64.7 | 450 ms | 15.5 ms | 1.0 | 0 | 1,934 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 108.4 | 54.3 | 628 ms | 18.4 ms | 2.0 | 0 | 3,252 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 171.5 | 43.0 | 891 ms | 23.3 ms | 4.0 | 0 | 5,144 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 54.5 | 54.8 | 999 ms | 18.2 ms | 1.0 | 0 | 1,636 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 83.5 | 42.0 | 1,401 ms | 23.8 ms | 2.0 | 0 | 2,503 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.