Speed test matrix
gemma-4-12B-it on H200 NVL 141GB
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve google/gemma-4-12B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.830641 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/gemma4-12B-it-DFlash","num_speculative_tokens":15}'Prefill
prompt tok/s by context length8K context19.5K420 ms TTFT
16K context17.9K910 ms TTFT
32K context16.5K1,961 ms TTFT
64K context13.7K4,724 ms TTFT
128K context9,98812,915 ms TTFT
251K context6,51538,804 ms TTFT
256K context6,40740,242 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 456.0114 ms TTFT | 882.698 ms TTFT | 1,798.0100 ms TTFT | 2,665.7109 ms TTFT | 4,104.0133 ms TTFT | 4,803.7294 ms TTFT | 5,195.81,387 ms TTFT | 4,650.61,779 ms TTFT |
| 16K | 317.4168 ms TTFT | 743.8126 ms TTFT | 1,475.7149 ms TTFT | 2,519.2132 ms TTFT | 3,165.9196 ms TTFT | 3,741.5399 ms TTFT | 4,605.31,312 ms TTFT | 3,566.52,228 ms TTFT |
| 32K | 310.4203 ms TTFT | 801.3197 ms TTFT | 1,520.2205 ms TTFT | 2,076.6222 ms TTFT | 2,515.7332 ms TTFT | 3,134.8813 ms TTFT | 2,860.12,097 ms TTFT | 2,578.93,880 ms TTFT |
| 64K | 564.6314 ms TTFT | 915.9333 ms TTFT | 1,330.0330 ms TTFT | 1,967.2394 ms TTFT | 2,350.6637 ms TTFT | 2,595.81,764 ms TTFT | 2,921.83,808 ms TTFT | 2,303.77,714 ms TTFT |
| 128K | 395.5557 ms TTFT | 411.06,762 ms TTFT | 1,067.6661 ms TTFT | 1,084.41,120 ms TTFT | 1,711.41,837 ms TTFT | 1,747.46,062 ms TTFT | 1,496.310,893 ms TTFT | Over capacity |
| 251K | 244.91,204 ms TTFT | 588.7Requests queued1,277 ms TTFT | 763.61,544 ms TTFT | 657.22,834 ms TTFT | 704.86,680 ms TTFT | 1,084.515,286 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.5K | — | 420 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 17.9K | — | 910 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 16.5K | — | 1,961 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.7K | — | 4,724 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,988.0 | — | 12,915 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,515.0 | — | 38,804 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,407.0 | — | 40,242 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 456.0 | 507.0 | 114 ms | 2.0 ms | 1.0 | 3 | 13,681 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 882.6 | 491.5 | 98 ms | 2.0 ms | 2.0 | 7 | 26,475 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,798.0 | 469.9 | 100 ms | 2.1 ms | 4.0 | 14 | 53,921 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,665.7 | 423.2 | 109 ms | 2.4 ms | 8.0 | 17 | 79,951 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,104.0 | 281.3 | 133 ms | 3.6 ms | 15.9 | 27 | 123,097 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,803.7 | 136.8 | 294 ms | 7.3 ms | 31.8 | 28 | 143,955 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,195.8 | 79.9 | 1,387 ms | 12.5 ms | 64.0 | 5 | 155,793 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,650.6 | 34.8 | 1,779 ms | 28.7 ms | 128.0 | 0 | 139,251 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 317.4 | 450.5 | 168 ms | 2.2 ms | 1.0 | 2 | 9,521 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 743.8 | 471.7 | 126 ms | 2.1 ms | 1.9 | 5 | 22,307 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,475.7 | 444.2 | 149 ms | 2.3 ms | 3.9 | 11 | 44,263 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,519.2 | 352.6 | 132 ms | 2.8 ms | 8.0 | 17 | 75,568 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,165.9 | 196.2 | 196 ms | 5.1 ms | 15.9 | 20 | 94,969 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,741.5 | 99.0 | 399 ms | 10.1 ms | 31.9 | 17 | 112,131 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,605.3 | 74.4 | 1,312 ms | 13.4 ms | 64.0 | 2 | 137,972 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,566.5 | 27.6 | 2,228 ms | 36.3 ms | 128.0 | 0 | 106,796 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 310.4 | 242.8 | 203 ms | 4.1 ms | 1.0 | 2 | 9,308 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 801.3 | 472.7 | 197 ms | 2.1 ms | 2.0 | 6 | 24,035 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,520.2 | 389.1 | 205 ms | 2.6 ms | 4.0 | 11 | 45,602 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,076.6 | 256.9 | 222 ms | 4.0 ms | 8.0 | 14 | 62,273 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,515.7 | 128.7 | 332 ms | 7.8 ms | 15.9 | 15 | 75,451 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,134.8 | 87.4 | 813 ms | 11.5 ms | 31.8 | 12 | 94,003 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,860.1 | 41.0 | 2,097 ms | 24.4 ms | 64.0 | 0 | 85,553 | 29.9 s |
| Sustained decode | 32K | 128 | Valid | 2,578.9 | 16.4 | 3,880 ms | 60.9 ms | 128.0 | 0 | 77,014 | 29.9 s |
| Sustained decode | 64K | 1 | Valid | 564.6 | 536.4 | 314 ms | 1.9 ms | 1.0 | 4 | 16,933 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 915.9 | 504.4 | 333 ms | 2.0 ms | 2.0 | 6 | 27,469 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,330.0 | 385.3 | 330 ms | 2.6 ms | 3.9 | 9 | 39,883 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,967.2 | 288.8 | 394 ms | 3.5 ms | 8.0 | 13 | 59,016 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,350.6 | 152.9 | 637 ms | 6.5 ms | 15.9 | 12 | 70,515 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,595.8 | 89.7 | 1,764 ms | 11.1 ms | 32.0 | 3 | 77,821 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,921.8 | 44.5 | 3,808 ms | 22.5 ms | 64.0 | 0 | 87,412 | 29.9 s |
| Sustained decode | 64K | 128 | Valid | 2,303.7 | 15.9 | 7,714 ms | 62.9 ms | 128.0 | 0 | 68,833 | 29.9 s |
| Sustained decode | 128K | 1 | Valid | 395.5 | 375.1 | 557 ms | 2.8 ms | 0.9 | 3 | 11,865 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 411.0 | 178.3 | 6,762 ms | 5.6 ms | 2.0 | 2 | 12,330 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 1,067.6 | 304.2 | 661 ms | 3.3 ms | 3.9 | 7 | 32,026 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,084.4 | 143.5 | 1,120 ms | 7.9 ms | 7.9 | 4 | 32,516 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,711.4 | 95.7 | 1,837 ms | 10.5 ms | 15.8 | 9 | 51,246 | 29.9 s |
| Sustained decode | 128K | 32 | Valid | 1,747.4 | 57.4 | 6,062 ms | 17.4 ms | 32.0 | 0 | 52,334 | 29.9 s |
| Sustained decode | 128K | 64 | Valid | 1,496.3 | 19.4 | 10,893 ms | 51.5 ms | 64.0 | 0 | 44,821 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 244.9 | 221.1 | 1,204 ms | 4.5 ms | 0.9 | 2 | 7,346 | 30.0 s |
| Sustained decode | 251K | 2 | Requests queued | 588.7 | 318.6 | 1,277 ms | 3.1 ms | 1.9 | 4 | 17,658 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 763.6 | 222.9 | 1,544 ms | 4.5 ms | 3.9 | 4 | 22,890 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 657.2 | 57.1 | 2,834 ms | 17.5 ms | 7.9 | 3 | 19,707 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 704.8 | 31.5 | 6,680 ms | 31.7 ms | 16.0 | 0 | 21,137 | 30.0 s |
| Sustained decode | 251K | 32 | Valid | 1,084.5 | 34.6 | 15,286 ms | 28.9 ms | 32.0 | 0 | 32,523 | 30.0 s |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.