Speed test matrix
gemma-4-26B-A4B-it on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-redhatai-sglang/bin/sglang" serve --model-path RedHatAI/gemma-4-26B-A4B-it --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 262144 --mem-fraction-static 0.852949 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context15.8K520 ms TTFT
16K context15.2K1,072 ms TTFT
32K context10.9K2,964 ms TTFT
64K context6,9329,314 ms TTFT
128K context3,98532,363 ms TTFT
251K context2,211114,338 ms TTFT
256K context2,162119,252 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 157.7Requests queued82 ms TTFT | 52.9153 ms TTFT | 99.6176 ms TTFT | 189.4215 ms TTFT | 933.7401 ms TTFT | 1,659.8767 ms TTFT | Over capacity | Over capacity |
| 16K | 138.0111 ms TTFT | 118.5159 ms TTFT | 424.2220 ms TTFT | 849.5398 ms TTFT | 1,336.8791 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 110.7199 ms TTFT | 181.9259 ms TTFT | 400.7414 ms TTFT | 732.2796 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 79.3312 ms TTFT | 151.8488 ms TTFT | 284.7840 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 50.7698 ms TTFT | 96.01,208 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 29.71,719 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 15.8K | — | 520 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.2K | — | 1,072 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.9K | — | 2,964 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,932.0 | — | 9,314 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,985.0 | — | 32,363 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,211.0 | — | 114,338 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,162.0 | — | 119,252 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 157.7 | 160.1 | 82 ms | 6.2 ms | 1.0 | 1 | 4,731 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 52.9 | 26.5 | 153 ms | 37.8 ms | 3.0 | 0 | 1,588 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 99.6 | 24.9 | 176 ms | 40.1 ms | 7.0 | 0 | 2,988 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 189.4 | 23.7 | 215 ms | 42.2 ms | 15.0 | 0 | 5,680 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 933.7 | 54.0 | 401 ms | 18.5 ms | 29.7 | 0 | 28,000 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,659.8 | 51.7 | 767 ms | 19.4 ms | 53.6 | 0 | 49,792 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 138.0 | 140.2 | 111 ms | 7.1 ms | 1.0 | 1 | 4,141 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 118.5 | 58.3 | 159 ms | 17.2 ms | 34.5 | 0 | 3,554 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 424.2 | 102.8 | 220 ms | 9.7 ms | 5.1 | 0 | 12,724 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 849.5 | 104.4 | 398 ms | 9.6 ms | 8.0 | 0 | 25,480 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,336.8 | 81.9 | 791 ms | 12.2 ms | 16.0 | 0 | 40,096 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 110.7 | 111.0 | 199 ms | 9.0 ms | 1.0 | 0 | 3,322 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 181.9 | 88.6 | 259 ms | 11.3 ms | 8.9 | 0 | 5,458 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 400.7 | 98.1 | 414 ms | 10.2 ms | 4.4 | 0 | 12,021 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 732.2 | 90.0 | 796 ms | 11.1 ms | 8.3 | 0 | 21,960 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 79.3 | 79.4 | 312 ms | 12.6 ms | 1.0 | 0 | 2,379 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 151.8 | 74.4 | 488 ms | 13.4 ms | 4.4 | 0 | 4,554 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 284.7 | 70.4 | 840 ms | 14.2 ms | 5.1 | 0 | 8,540 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 50.7 | 50.8 | 698 ms | 19.7 ms | 1.0 | 0 | 1,522 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 96.0 | 48.0 | 1,208 ms | 20.8 ms | 5.6 | 0 | 2,880 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 29.7 | 29.7 | 1,719 ms | 33.7 ms | 1.0 | 0 | 890 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.