Speed test matrix
gemma-4-31B-it on H200 NVL 141GB ×2
vllm · 0.26.1rc1.dev914+gc3dabcddc · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev914+gc3dabcddc
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
c3dabcddc328
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-z-lab-vllm/bin/vllm" serve google/gemma-4-31B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 2 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.724274 --host 0.0.0.0 --port 8000 --api-key <redacted> --max-num-batched-tokens 32768 --attention-backend flash_attn --speculative-config '{"method":"dflash","model":"z-lab/gemma-4-31B-it-DFlash","num_speculative_tokens":15}'Prefill
prompt tok/s by context length8K context14.4K570 ms TTFT
16K context13.8K1,177 ms TTFT
32K context12.2K2,661 ms TTFT
64K context10.2K6,351 ms TTFT
128K context7,43317,353 ms TTFT
251K context4,88151,792 ms TTFT
256K context4,82853,405 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 501.489 ms TTFT | 766.5103 ms TTFT | 602.4120 ms TTFT | 1,023.8179 ms TTFT | 2,103.9232 ms TTFT | 2,740.9393 ms TTFT | 2,242.81,075 ms TTFT | 2,432.11,495 ms TTFT |
| 16K | 125.7659 ms TTFT | 372.4680 ms TTFT | 548.0152 ms TTFT | 863.6193 ms TTFT | 2,021.8254 ms TTFT | 2,607.5510 ms TTFT | 2,718.81,309 ms TTFT | 2,378.42,322 ms TTFT |
| 32K | 113.42,657 ms TTFT | 357.12,722 ms TTFT | 569.92,739 ms TTFT | 927.95,211 ms TTFT | 1,311.7Requests queued15,315 ms TTFT | 1,253.8Requests queued30,369 ms TTFT | 1,414.7Requests queued30,431 ms TTFT | 1,462.1Requests queued30,475 ms TTFT |
| 64K | 332.66,338 ms TTFT | 463.16,428 ms TTFT | 501.1Requests queued12,563 ms TTFT | 599.9Requests queued25,408 ms TTFT | 520.7Requests queued63,358 ms TTFT | 527.4Requests queued64,051 ms TTFT | 562.1Requests queued68,618 ms TTFT | Over capacity |
| 128K | 382.0572 ms TTFT | 362.49,097 ms TTFT | 876.9659 ms TTFT | 1,315.6734 ms TTFT | 1,820.31,951 ms TTFT | 1,880.54,920 ms TTFT | Over capacity | Over capacity |
| 251K | 367.551,829 ms TTFT | 27.3Requests queued51,986 ms TTFT | 416.0Requests queued25,727 ms TTFT | 93.0Requests queued29,868 ms TTFT | 14.0Requests queued31,070 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.4K | — | 570 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.8K | — | 1,177 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.2K | — | 2,661 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.2K | — | 6,351 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,433.0 | — | 17,353 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,881.0 | — | 51,792 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,828.0 | — | 53,405 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 501.4 | 575.9 | 89 ms | 1.7 ms | 1.0 | 4 | 15,038 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 766.5 | 457.0 | 103 ms | 2.2 ms | 2.0 | 5 | 22,985 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 602.4 | 118.9 | 120 ms | 8.4 ms | 4.0 | 2 | 18,065 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,023.8 | 102.7 | 179 ms | 9.7 ms | 8.0 | 3 | 30,696 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,103.9 | 143.2 | 232 ms | 7.0 ms | 15.9 | 9 | 63,058 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,740.9 | 65.7 | 393 ms | 15.2 ms | 32.0 | 13 | 81,953 | 29.9 s |
| Sustained decode | 8K | 64 | Valid | 2,242.8 | 20.2 | 1,075 ms | 49.6 ms | 64.0 | 1 | 67,294 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,432.1 | 10.8 | 1,495 ms | 92.8 ms | 128.0 | 0 | 72,636 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 125.7 | 173.6 | 659 ms | 6.2 ms | 1.0 | 1 | 3,769 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 372.4 | 196.0 | 680 ms | 5.8 ms | 2.0 | 2 | 11,172 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 548.0 | 119.0 | 152 ms | 8.4 ms | 4.0 | 3 | 16,434 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 863.6 | 92.0 | 193 ms | 10.9 ms | 8.0 | 2 | 25,905 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,021.8 | 80.7 | 254 ms | 12.5 ms | 16.0 | 10 | 60,607 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,607.5 | 48.5 | 510 ms | 20.7 ms | 32.0 | 10 | 78,219 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,718.8 | 38.2 | 1,309 ms | 26.4 ms | 64.0 | 0 | 81,332 | 29.9 s |
| Sustained decode | 16K | 128 | Valid | 2,378.4 | 11.4 | 2,322 ms | 87.4 ms | 128.0 | 0 | 71,113 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 113.4 | 116.5 | 2,657 ms | 8.6 ms | 1.0 | 0 | 3,402 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 357.1 | 327.0 | 2,722 ms | 5.0 ms | 2.0 | 2 | 10,711 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 569.9 | 238.3 | 2,739 ms | 4.2 ms | 4.0 | 3 | 17,090 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 927.9 | 116.5 | 5,211 ms | 8.6 ms | 7.9 | 6 | 27,818 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 1,311.7 | 48.7 | 15,315 ms | 20.5 ms | 15.9 | 5 | 39,350 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 1,253.8 | 46.8 | 30,369 ms | 21.4 ms | 18.3 | 6 | 36,112 | 28.8 s |
| Sustained decode | 32K | 64 | Requests queued | 1,414.7 | 40.2 | 30,431 ms | 24.9 ms | 18.5 | 5 | 42,378 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 1,462.1 | 49.1 | 30,475 ms | 20.4 ms | 18.5 | 5 | 43,846 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 332.6 | 510.1 | 6,338 ms | 2.0 ms | 0.8 | 2 | 9,974 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 463.1 | 321.6 | 6,428 ms | 3.5 ms | 1.9 | 3 | 12,769 | 27.6 s |
| Sustained decode | 64K | 4 | Requests queued | 501.1 | 105.3 | 12,563 ms | 9.5 ms | 3.7 | 4 | 14,597 | 29.1 s |
| Sustained decode | 64K | 8 | Requests queued | 599.9 | 107.9 | 25,408 ms | 9.3 ms | 6.0 | 11 | 16,790 | 28.0 s |
| Sustained decode | 64K | 16 | Requests queued | 520.7 | 115.8 | 63,358 ms | 8.6 ms | 6.0 | 10 | 15,501 | 29.8 s |
| Sustained decode | 64K | 32 | Requests queued | 527.4 | 113.4 | 64,051 ms | 8.8 ms | 6.0 | 10 | 14,805 | 28.1 s |
| Sustained decode | 64K | 64 | Requests queued | 562.1 | 106.6 | 68,618 ms | 9.4 ms | 5.9 | 11 | 16,639 | 29.6 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 382.0 | 442.0 | 572 ms | 2.3 ms | 1.0 | 3 | 11,310 | 29.6 s |
| Sustained decode | 128K | 2 | Valid | 362.4 | 229.0 | 9,097 ms | 6.4 ms | 2.0 | 2 | 10,869 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 876.9 | 236.7 | 659 ms | 4.2 ms | 4.0 | 5 | 26,301 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 1,315.6 | 174.5 | 734 ms | 5.7 ms | 7.9 | 7 | 39,447 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,820.3 | 99.8 | 1,951 ms | 10.0 ms | 15.9 | 6 | 54,500 | 29.9 s |
| Sustained decode | 128K | 32 | Valid | 1,880.5 | 55.1 | 4,920 ms | 18.2 ms | 32.0 | 1 | 56,218 | 29.9 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 367.5 | 323.0 | 51,829 ms | 3.1 ms | 0.9 | 1 | 3,357 | 9.1 s |
| Sustained decode | 251K | 2 | Requests queued | 27.3 | 27.6 | 51,986 ms | 36.2 ms | 1.0 | 0 | 816 | 29.9 s |
| Sustained decode | 251K | 4 | Requests queued | 416.0 | 96.4 | 25,727 ms | 10.4 ms | 2.9 | 3 | 12,460 | 30.0 s |
| Sustained decode | 251K | 8 | Requests queued | 93.0 | 30.9 | 29,868 ms | 32.4 ms | 3.0 | 2 | 2,788 | 30.0 s |
| Sustained decode | 251K | 16 | Requests queued | 14.0 | 84.5 | 31,070 ms | 11.8 ms | 1.0 | 4 | 419 | 29.9 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.