Speed test matrix
gemma-4-12B-it on H200 NVL 141GB
sglang · 0.0.0.dev16952+gd5bd07850 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev16952+gd5bd07850
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/fechaMe/sglang
- Build
- cuda-13.2
- Commit
d5bd07850f62
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-redhat-sglang-pr-35809/bin/sglang" serve --model-path RedHatAI/gemma-4-12B-it --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 262144 --mem-fraction-static 0.855556 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context10.1K816 ms TTFT
16K context9,7021,676 ms TTFT
32K context6,8414,732 ms TTFT
64K context4,31914,951 ms TTFT
128K context2,48851,837 ms TTFT
251K context1,374183,967 ms TTFT
256K context1,350190,966 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 89.8Requests queued98 ms TTFT | 33.5198 ms TTFT | 66.71,347 ms TTFT | 132.7236 ms TTFT | 256.8403 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 79.0121 ms TTFT | 31.7248 ms TTFT | 66.0283 ms TTFT | 468.2397 ms TTFT | 1,015.5764 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 64.4192 ms TTFT | 116.2270 ms TTFT | 252.4405 ms TTFT | 487.9792 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 46.9344 ms TTFT | 93.7498 ms TTFT | 191.8832 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 30.3767 ms TTFT | 63.21,309 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 17.2Requests queued1,952 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.1K | — | 816 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,702.0 | — | 1,676 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,841.0 | — | 4,732 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,319.0 | — | 14,951 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,488.0 | — | 51,837 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,374.0 | — | 183,967 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,350.0 | — | 190,966 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 89.8 | 89.8 | 98 ms | 11.1 ms | 1.0 | 0 | 2,693 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 33.5 | 16.7 | 198 ms | 59.8 ms | 3.0 | 0 | 1,004 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 66.7 | 16.7 | 1,347 ms | 60.0 ms | 7.0 | 0 | 2,000 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 132.7 | 16.6 | 236 ms | 60.3 ms | 15.0 | 0 | 3,976 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 256.8 | 16.0 | 403 ms | 62.4 ms | 31.0 | 0 | 7,695 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 79.0 | 79.2 | 121 ms | 12.6 ms | 1.0 | 0 | 2,371 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 31.7 | 15.8 | 248 ms | 63.1 ms | 33.0 | 0 | 952 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 66.0 | 16.4 | 283 ms | 61.0 ms | 37.0 | 0 | 1,980 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 468.2 | 57.9 | 397 ms | 17.3 ms | 38.5 | 0 | 14,040 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,015.5 | 62.9 | 764 ms | 15.9 ms | 25.8 | 0 | 30,464 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 64.4 | 64.5 | 192 ms | 15.5 ms | 1.0 | 0 | 1,932 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 116.2 | 58.1 | 270 ms | 17.2 ms | 16.4 | 0 | 3,486 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 252.4 | 63.2 | 405 ms | 15.8 ms | 5.8 | 0 | 7,573 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 487.9 | 61.2 | 792 ms | 16.4 ms | 11.3 | 0 | 14,632 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 46.9 | 46.9 | 344 ms | 21.3 ms | 1.0 | 0 | 1,406 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 93.7 | 46.9 | 498 ms | 21.3 ms | 10.0 | 0 | 2,810 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 191.8 | 47.9 | 832 ms | 20.9 ms | 6.0 | 0 | 5,752 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 30.3 | 30.3 | 767 ms | 33.1 ms | 1.0 | 0 | 907 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 63.2 | 31.7 | 1,309 ms | 31.6 ms | 7.0 | 0 | 1,894 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Requests queued | 17.2 | 18.0 | 1,952 ms | 55.5 ms | 0.9 | 0 | 515 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.