Speed test matrix
gemma-4-26B-A4B-it on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve google/gemma-4-26B-A4B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.713818 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/gemma-4-26B-A4B-it-DFlash","num_speculative_tokens":15}'Prefill
prompt tok/s by context length8K context37.3K220 ms TTFT
16K context34.3K475 ms TTFT
32K context32.8K986 ms TTFT
64K context25.9K2,488 ms TTFT
128K context17.9K7,215 ms TTFT
251K context10.9K23,117 ms TTFT
256K context10.8K23,845 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 628.991 ms TTFT | 935.991 ms TTFT | 1,826.9101 ms TTFT | 2,188.9110 ms TTFT | 3,602.5117 ms TTFT | 5,076.7345 ms TTFT | 7,271.6716 ms TTFT | 7,770.2Requests queued1,299 ms TTFT |
| 16K | 1,188.9125 ms TTFT | 867.4126 ms TTFT | 1,791.1128 ms TTFT | 2,967.1154 ms TTFT | 6,328.6156 ms TTFT | 8,258.9191 ms TTFT | 9,948.0477 ms TTFT | 10.1KRequests queued1,899 ms TTFT |
| 32K | 764.6187 ms TTFT | 1,487.5176 ms TTFT | 2,118.3191 ms TTFT | 3,211.7212 ms TTFT | 5,666.4216 ms TTFT | 6,891.9261 ms TTFT | 8,301.0674 ms TTFT | Over capacity |
| 64K | 780.0263 ms TTFT | 1,266.3272 ms TTFT | 1,655.0288 ms TTFT | 2,685.5312 ms TTFT | 3,158.8343 ms TTFT | 4,972.0796 ms TTFT | Over capacity | Over capacity |
| 128K | 507.1508 ms TTFT | 1,325.7Requests queued553 ms TTFT | 1,534.4544 ms TTFT | 1,741.7582 ms TTFT | 2,753.6857 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 352.31,089 ms TTFT | 460.81,175 ms TTFT | 965.2Requests queued1,236 ms TTFT | 1,143.91,528 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 37.3K | — | 220 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.3K | — | 475 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 32.8K | — | 986 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 25.9K | — | 2,488 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.9K | — | 7,215 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 10.9K | — | 23,117 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.8K | — | 23,845 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 628.9 | 752.0 | 91 ms | 1.3 ms | 1.0 | 4 | 18,867 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 935.9 | 425.5 | 91 ms | 2.4 ms | 2.0 | 6 | 28,072 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,826.9 | 490.9 | 101 ms | 2.0 ms | 4.0 | 14 | 54,799 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,188.9 | 240.5 | 110 ms | 4.2 ms | 8.0 | 15 | 65,660 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,602.5 | 209.0 | 117 ms | 4.8 ms | 16.0 | 22 | 108,073 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 5,076.7 | 124.1 | 345 ms | 8.1 ms | 31.9 | 28 | 152,231 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 7,271.6 | 74.3 | 716 ms | 13.5 ms | 63.9 | 36 | 218,159 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 7,770.2 | 38.1 | 1,299 ms | 26.2 ms | 121.3 | 19 | 232,574 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 1,188.9 | 1313.9 | 125 ms | 0.8 ms | 1.0 | 9 | 35,666 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 867.4 | 361.9 | 126 ms | 2.8 ms | 2.0 | 6 | 26,020 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,791.1 | 711.9 | 128 ms | 1.4 ms | 4.0 | 14 | 53,724 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,967.1 | 491.1 | 154 ms | 2.0 ms | 7.9 | 20 | 89,003 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 6,328.6 | 465.6 | 156 ms | 2.1 ms | 15.8 | 45 | 189,720 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 8,258.9 | 306.2 | 191 ms | 3.3 ms | 31.8 | 51 | 247,650 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 9,948.0 | 154.9 | 477 ms | 6.5 ms | 63.8 | 50 | 298,207 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 10.1K | 76.6 | 1,899 ms | 13.1 ms | 111.7 | 31 | 300,991 | 29.8 s |
| Sustained decode | 32K | 1 | Valid | 764.6 | 962.4 | 187 ms | 1.0 ms | 1.0 | 6 | 22,936 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 1,487.5 | 912.9 | 176 ms | 1.1 ms | 2.0 | 14 | 44,618 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 2,118.3 | 698.4 | 191 ms | 1.4 ms | 4.0 | 20 | 63,522 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 3,211.7 | 555.7 | 212 ms | 1.8 ms | 7.8 | 23 | 96,332 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 5,666.4 | 431.1 | 216 ms | 2.3 ms | 15.9 | 37 | 169,882 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 6,891.9 | 236.2 | 261 ms | 4.2 ms | 31.8 | 40 | 206,727 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 8,301.0 | 112.5 | 674 ms | 8.9 ms | 63.7 | 49 | 248,612 | 29.9 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 780.0 | 982.8 | 263 ms | 1.0 ms | 1.0 | 6 | 23,392 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 1,266.3 | 886.9 | 272 ms | 1.1 ms | 2.0 | 10 | 37,925 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 1,655.0 | 564.5 | 288 ms | 1.8 ms | 4.0 | 13 | 49,637 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,685.5 | 460.9 | 312 ms | 2.2 ms | 7.9 | 18 | 80,542 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,158.8 | 172.0 | 343 ms | 5.8 ms | 15.9 | 19 | 94,747 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,972.0 | 178.0 | 796 ms | 5.6 ms | 31.7 | 23 | 149,104 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 507.1 | 1040.3 | 508 ms | 1.0 ms | 0.9 | 5 | 15,211 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 1,325.7 | 795.7 | 553 ms | 1.3 ms | 1.9 | 9 | 39,758 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,534.4 | 535.8 | 544 ms | 1.9 ms | 3.8 | 11 | 46,023 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,741.7 | 283.2 | 582 ms | 3.5 ms | 7.9 | 11 | 52,228 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 2,753.6 | 192.0 | 857 ms | 5.2 ms | 15.7 | 17 | 82,597 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 352.3 | 437.4 | 1,089 ms | 2.5 ms | 0.9 | 3 | 10,568 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 460.8 | 357.9 | 1,175 ms | 2.8 ms | 1.9 | 3 | 13,822 | 30.0 s |
| Sustained decode | 251K | 4 | Requests queued | 965.2 | 334.4 | 1,236 ms | 3.0 ms | 3.8 | 7 | 28,945 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 1,143.9 | 135.4 | 1,528 ms | 7.4 ms | 7.8 | 7 | 34,284 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.