Speed test matrix
gemma-4-26B-A4B-it-FP8-dynamic on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-redhatai-sglang/bin/sglang" serve --model-path RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 262144 --mem-fraction-static 0.852746 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context18.1K453 ms TTFT
16K context17.6K922 ms TTFT
32K context12.1K2,670 ms TTFT
64K context7,4228,699 ms TTFT
128K context4,12931,240 ms TTFT
251K context2,247112,504 ms TTFT
256K context2,204116,978 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 172.1Requests queued88 ms TTFT | 54.7154 ms TTFT | 105.4738 ms TTFT | 201.1298 ms TTFT | 1,315.2401 ms TTFT | 2,382.1766 ms TTFT | Over capacity | Over capacity |
| 16K | 148.2111 ms TTFT | 213.6155 ms TTFT | 530.3219 ms TTFT | 857.0397 ms TTFT | 1,633.6766 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 116.8172 ms TTFT | 230.0259 ms TTFT | 442.6412 ms TTFT | 811.6791 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 82.6301 ms TTFT | 167.8486 ms TTFT | 312.1858 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 51.6708 ms TTFT | 101.71,239 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 30.11,783 ms TTFT | 65.33,150 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.1K | — | 453 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 17.6K | — | 922 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.1K | — | 2,670 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,422.0 | — | 8,699 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,129.0 | — | 31,240 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,247.0 | — | 112,504 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,204.0 | — | 116,978 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 172.1 | 174.0 | 88 ms | 5.7 ms | 1.0 | 1 | 5,162 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 54.7 | 27.3 | 154 ms | 36.6 ms | 3.0 | 0 | 1,640 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 105.4 | 26.4 | 738 ms | 38.0 ms | 7.0 | 0 | 3,160 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 201.1 | 25.1 | 298 ms | 39.8 ms | 15.0 | 0 | 6,032 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,315.2 | 77.2 | 401 ms | 12.9 ms | 28.2 | 0 | 39,457 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,382.1 | 73.3 | 766 ms | 13.6 ms | 40.3 | 0 | 71,456 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 148.2 | 150.5 | 111 ms | 6.6 ms | 1.0 | 1 | 4,445 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 213.6 | 102.3 | 155 ms | 9.8 ms | 18.0 | 0 | 6,408 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 530.3 | 135.2 | 219 ms | 7.4 ms | 3.9 | 4 | 15,909 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 857.0 | 107.5 | 397 ms | 9.3 ms | 11.6 | 0 | 25,704 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,633.6 | 101.8 | 766 ms | 9.8 ms | 16.0 | 0 | 49,009 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 116.8 | 117.3 | 172 ms | 8.5 ms | 1.0 | 0 | 3,504 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 230.0 | 111.5 | 259 ms | 9.0 ms | 3.1 | 0 | 6,900 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 442.6 | 110.8 | 412 ms | 9.0 ms | 4.0 | 0 | 13,276 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 811.6 | 102.3 | 791 ms | 9.8 ms | 8.0 | 0 | 24,344 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 82.6 | 82.7 | 301 ms | 12.1 ms | 1.0 | 0 | 2,477 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 167.8 | 82.4 | 486 ms | 12.1 ms | 2.3 | 0 | 5,034 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 312.1 | 77.7 | 858 ms | 12.9 ms | 4.7 | 0 | 9,363 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 51.6 | 51.7 | 708 ms | 19.4 ms | 1.0 | 0 | 1,548 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 101.7 | 50.9 | 1,239 ms | 19.6 ms | 4.7 | 0 | 3,052 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 30.1 | 30.1 | 1,783 ms | 33.3 ms | 1.0 | 0 | 901 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 65.3 | 33.0 | 3,150 ms | 30.3 ms | 4.0 | 0 | 1,958 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.