Speed test matrix
Gemma-4-26B-A4B-NVFP4 on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-vllm/bin/vllm" serve nvidia/Gemma-4-26B-A4B-NVFP4 --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.878348 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context20.1K409 ms TTFT
16K context15K1,082 ms TTFT
32K context9,6773,345 ms TTFT
64K context5,60311,523 ms TTFT
128K context3,01742,753 ms TTFT
251K context1,611156,937 ms TTFT
256K context1,584162,771 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 167.575 ms TTFT | 318.998 ms TTFT | 565.4114 ms TTFT | 1,007.9141 ms TTFT | 1,681.1279 ms TTFT | 1,503.9392 ms TTFT | 2,708.4680 ms TTFT | 3,935.31,027 ms TTFT |
| 16K | 149.9128 ms TTFT | 290.2138 ms TTFT | 519.0140 ms TTFT | 897.8257 ms TTFT | 1,469.6372 ms TTFT | 1,076.7704 ms TTFT | 1,923.21,108 ms TTFT | 3,217.21,770 ms TTFT |
| 32K | 135.1183 ms TTFT | 259.4215 ms TTFT | 464.2304 ms TTFT | 743.1435 ms TTFT | 1,226.0708 ms TTFT | 639.61,142 ms TTFT | 1,284.71,823 ms TTFT | 2,215.73,278 ms TTFT |
| 64K | 120.7281 ms TTFT | 116.8406 ms TTFT | 408.5448 ms TTFT | 764.0717 ms TTFT | 1,043.21,130 ms TTFT | 395.81,960 ms TTFT | 776.43,515 ms TTFT | Over capacity |
| 128K | 99.1632 ms TTFT | 191.9819 ms TTFT | 329.21,153 ms TTFT | 529.41,782 ms TTFT | 727.73,210 ms TTFT | 218.15,685 ms TTFT | Over capacity | Over capacity |
| 251K | 73.61,305 ms TTFT | 143.81,656 ms TTFT | 239.32,287 ms TTFT | 457.03,693 ms TTFT | 568.56,616 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.1K | — | 409 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15K | — | 1,082 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,677.0 | — | 3,345 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,603.0 | — | 11,523 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,017.0 | — | 42,753 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,611.0 | — | 156,937 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,584.0 | — | 162,771 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 167.5 | 169.2 | 75 ms | 5.9 ms | 1.0 | 1 | 5,024 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 318.9 | 161.0 | 98 ms | 6.2 ms | 2.0 | 2 | 9,565 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 565.4 | 142.8 | 114 ms | 7.0 ms | 3.9 | 4 | 16,959 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,007.9 | 129.4 | 141 ms | 7.7 ms | 8.0 | 8 | 30,231 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,681.1 | 105.6 | 279 ms | 9.5 ms | 16.0 | 0 | 50,434 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,503.9 | 50.9 | 392 ms | 19.7 ms | 32.0 | 0 | 45,095 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,708.4 | 42.6 | 680 ms | 23.5 ms | 64.0 | 0 | 81,223 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,935.3 | 33.5 | 1,027 ms | 29.8 ms | 128.0 | 0 | 118,062 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 149.9 | 152.7 | 128 ms | 6.6 ms | 1.0 | 1 | 4,495 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 290.2 | 147.1 | 138 ms | 6.8 ms | 2.0 | 2 | 8,705 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 519.0 | 131.7 | 140 ms | 7.6 ms | 4.0 | 4 | 15,572 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 897.8 | 112.7 | 257 ms | 8.9 ms | 8.0 | 0 | 26,928 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,469.6 | 92.1 | 372 ms | 10.9 ms | 16.0 | 0 | 44,081 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,076.7 | 36.3 | 704 ms | 27.5 ms | 32.0 | 0 | 32,302 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,923.2 | 31.7 | 1,108 ms | 31.5 ms | 64.0 | 0 | 57,700 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,217.2 | 26.3 | 1,770 ms | 38.1 ms | 128.0 | 0 | 96,566 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 135.1 | 137.3 | 183 ms | 7.3 ms | 1.0 | 1 | 4,053 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 259.4 | 131.7 | 215 ms | 7.6 ms | 2.0 | 2 | 7,781 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 464.2 | 116.1 | 304 ms | 8.6 ms | 4.0 | 0 | 13,924 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 743.1 | 105.4 | 435 ms | 9.5 ms | 8.0 | 0 | 22,291 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,226.0 | 81.2 | 708 ms | 12.3 ms | 16.0 | 0 | 36,780 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 639.6 | 22.9 | 1,142 ms | 43.7 ms | 32.0 | 0 | 19,166 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,284.7 | 21.0 | 1,823 ms | 47.5 ms | 64.0 | 0 | 38,530 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,215.7 | 18.1 | 3,278 ms | 55.3 ms | 128.0 | 0 | 66,413 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 120.7 | 121.0 | 281 ms | 8.3 ms | 1.0 | 0 | 3,620 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 116.8 | 123.8 | 406 ms | 8.1 ms | 2.0 | 0 | 3,504 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 408.5 | 102.1 | 448 ms | 9.8 ms | 4.0 | 0 | 12,253 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 764.0 | 94.9 | 717 ms | 10.5 ms | 8.0 | 0 | 22,917 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,043.2 | 68.5 | 1,130 ms | 14.6 ms | 16.0 | 0 | 31,291 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 395.8 | 13.1 | 1,960 ms | 76.5 ms | 32.0 | 0 | 11,873 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 776.4 | 12.5 | 3,515 ms | 80.2 ms | 64.0 | 0 | 23,292 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 99.1 | 99.4 | 632 ms | 10.1 ms | 1.0 | 0 | 2,972 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 191.9 | 96.1 | 819 ms | 10.4 ms | 2.0 | 0 | 5,758 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 329.2 | 81.9 | 1,153 ms | 12.2 ms | 4.0 | 0 | 9,872 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 529.4 | 74.6 | 1,782 ms | 13.4 ms | 8.0 | 0 | 15,876 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 727.7 | 50.5 | 3,210 ms | 19.8 ms | 16.0 | 0 | 21,830 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 218.1 | 7.2 | 5,685 ms | 139.3 ms | 32.0 | 0 | 6,538 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 73.6 | 79.8 | 1,305 ms | 12.5 ms | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 143.8 | 72.0 | 1,656 ms | 13.9 ms | 2.0 | 0 | 4,313 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 239.3 | 59.3 | 2,287 ms | 16.8 ms | 4.0 | 0 | 7,178 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 457.0 | 55.6 | 3,693 ms | 18.0 ms | 8.0 | 0 | 13,704 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 568.5 | 33.8 | 6,616 ms | 29.6 ms | 16.0 | 0 | 17,040 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.