Speed test matrix
gemma-4-12B-it-int4-AutoRound on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-intel-vllm/bin/vllm" serve Intel/gemma-4-12B-it-int4-AutoRound --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.849155 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context18.7K439 ms TTFT
16K context17.7K920 ms TTFT
32K context16.3K1,980 ms TTFT
64K context13.3K4,838 ms TTFT
128K context9,61513,414 ms TTFT
251K context6,27340,299 ms TTFT
256K context6,20641,548 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 127.969 ms TTFT | 250.680 ms TTFT | 481.2104 ms TTFT | 896.6158 ms TTFT | 1,602.7235 ms TTFT | 2,466.3413 ms TTFT | 3,382.5641 ms TTFT | 4,246.41,027 ms TTFT |
| 16K | 127.0103 ms TTFT | 247.6139 ms TTFT | 474.5170 ms TTFT | 867.6190 ms TTFT | 1,527.2417 ms TTFT | 2,289.9551 ms TTFT | 3,061.7925 ms TTFT | Over capacity |
| 32K | 125.4149 ms TTFT | 242.9207 ms TTFT | 459.6308 ms TTFT | 828.5410 ms TTFT | 1,403.4682 ms TTFT | 2,010.2973 ms TTFT | Over capacity | Over capacity |
| 64K | 123.1226 ms TTFT | 235.8316 ms TTFT | 437.2443 ms TTFT | 756.3651 ms TTFT | 1,215.61,213 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 119.0457 ms TTFT | 223.3625 ms TTFT | 398.3933 ms TTFT | 646.51,391 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 111.1987 ms TTFT | 203.31,359 ms TTFT | 341.62,122 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.7K | — | 439 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 17.7K | — | 920 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16.3K | — | 1,980 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.3K | — | 4,838 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,615.0 | — | 13,414 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,273.0 | — | 40,299 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,206.0 | — | 41,548 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 127.9 | 128.9 | 69 ms | 7.8 ms | 1.0 | 1 | 3,836 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 250.6 | 126.3 | 80 ms | 7.9 ms | 2.0 | 2 | 7,515 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 481.2 | 122.2 | 104 ms | 8.2 ms | 3.9 | 4 | 14,441 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 896.6 | 112.5 | 158 ms | 8.9 ms | 8.0 | 0 | 26,899 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,602.7 | 100.7 | 235 ms | 9.9 ms | 16.0 | 0 | 48,080 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,466.3 | 77.3 | 413 ms | 12.9 ms | 32.0 | 0 | 73,987 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,382.5 | 53.1 | 641 ms | 18.8 ms | 64.0 | 0 | 101,574 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,246.4 | 33.5 | 1,027 ms | 29.9 ms | 128.0 | 0 | 127,360 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 127.0 | 128.7 | 103 ms | 7.8 ms | 1.0 | 1 | 3,809 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 247.6 | 125.9 | 139 ms | 7.9 ms | 2.0 | 2 | 7,430 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 474.5 | 118.9 | 170 ms | 8.4 ms | 4.0 | 1 | 14,244 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 867.6 | 108.8 | 190 ms | 9.2 ms | 8.0 | 0 | 26,026 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,527.2 | 95.9 | 417 ms | 10.4 ms | 16.0 | 0 | 45,808 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,289.9 | 71.7 | 551 ms | 13.9 ms | 32.0 | 0 | 68,706 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,061.7 | 48.1 | 925 ms | 20.8 ms | 64.0 | 0 | 91,843 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 125.4 | 129.7 | 149 ms | 7.7 ms | 1.0 | 1 | 3,761 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 242.9 | 121.6 | 207 ms | 8.2 ms | 2.0 | 0 | 7,290 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 459.6 | 115.2 | 308 ms | 8.7 ms | 4.0 | 0 | 13,789 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 828.5 | 103.8 | 410 ms | 9.6 ms | 8.0 | 0 | 24,849 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,403.4 | 87.8 | 682 ms | 11.4 ms | 16.0 | 0 | 42,095 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,010.2 | 62.5 | 973 ms | 16.0 ms | 32.0 | 0 | 60,289 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 123.1 | 123.4 | 226 ms | 8.1 ms | 1.0 | 1 | 3,676 | 29.9 s |
| Sustained decode | 64K | 2 | Valid | 235.8 | 118.0 | 316 ms | 8.5 ms | 2.0 | 0 | 7,074 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 437.2 | 109.4 | 443 ms | 9.1 ms | 4.0 | 0 | 13,116 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 756.3 | 94.4 | 651 ms | 10.6 ms | 8.0 | 0 | 22,688 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,215.6 | 75.4 | 1,213 ms | 13.3 ms | 16.0 | 0 | 36,463 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 119.0 | 119.4 | 457 ms | 8.4 ms | 1.0 | 0 | 3,569 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 223.3 | 112.0 | 625 ms | 8.9 ms | 2.0 | 0 | 6,700 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 398.3 | 99.8 | 933 ms | 10.0 ms | 4.0 | 0 | 11,948 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 646.5 | 80.7 | 1,391 ms | 12.4 ms | 8.0 | 0 | 19,394 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 111.1 | 111.6 | 987 ms | 9.0 ms | 1.0 | 0 | 3,333 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 203.3 | 102.0 | 1,359 ms | 9.8 ms | 2.0 | 0 | 6,098 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 341.6 | 85.6 | 2,122 ms | 11.7 ms | 4.0 | 0 | 10,244 | 30.0 s |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.