Speed test matrix
gemma-4-26B-A4B-it on H200 NVL 141GB
sglang · 0.5.6.post3.dev5791+gf7d4e44d8 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev5791+gf7d4e44d8
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
f7d4e44d82ac
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-gemma-sglang/bin/sglang" serve --model-path google/gemma-4-26B-A4B-it --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 262144 --mem-fraction-static 0.849655 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context18.4K446 ms TTFT
16K context15.1K1,079 ms TTFT
32K context10.9K2,978 ms TTFT
64K context6,9659,270 ms TTFT
128K context4,03431,972 ms TTFT
251K context2,245112,619 ms TTFT
256K context2,200117,201 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 148.688 ms TTFT | 278.596 ms TTFT | 504.6137 ms TTFT | 841.4225 ms TTFT | 1,322.9401 ms TTFT | 2,174.0764 ms TTFT | Over capacity | Over capacity |
| 16K | 130.4118 ms TTFT | 247.2134 ms TTFT | 444.4219 ms TTFT | 816.3396 ms TTFT | 1,386.7762 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 105.9176 ms TTFT | 207.1250 ms TTFT | 380.0414 ms TTFT | 706.6784 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 77.4326 ms TTFT | 155.0477 ms TTFT | 295.8828 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 49.6726 ms TTFT | 103.51,165 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 29.71,751 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.4K | — | 446 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.1K | — | 1,079 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.9K | — | 2,978 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,965.0 | — | 9,270 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,034.0 | — | 31,972 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,245.0 | — | 112,619 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,200.0 | — | 117,201 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 148.6 | 150.7 | 88 ms | 6.6 ms | 1.0 | 1 | 4,456 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 278.5 | 141.2 | 96 ms | 7.1 ms | 2.0 | 2 | 8,354 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 504.6 | 127.2 | 137 ms | 7.9 ms | 4.0 | 4 | 15,135 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 841.4 | 105.1 | 225 ms | 9.5 ms | 8.0 | 0 | 25,240 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,322.9 | 83.1 | 401 ms | 12.0 ms | 16.0 | 0 | 39,680 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,174.0 | 68.2 | 764 ms | 14.7 ms | 32.0 | 0 | 65,216 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 130.4 | 133.0 | 118 ms | 7.5 ms | 1.0 | 1 | 3,910 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 247.2 | 125.5 | 134 ms | 8.0 ms | 2.0 | 2 | 7,415 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 444.4 | 111.5 | 219 ms | 9.0 ms | 4.0 | 0 | 13,328 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 816.3 | 102.2 | 396 ms | 9.8 ms | 8.0 | 0 | 24,489 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,386.7 | 86.4 | 762 ms | 11.6 ms | 16.0 | 0 | 41,598 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 105.9 | 106.1 | 176 ms | 9.4 ms | 1.0 | 0 | 3,177 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 207.1 | 103.9 | 250 ms | 9.6 ms | 2.0 | 0 | 6,214 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 380.0 | 95.5 | 414 ms | 10.5 ms | 4.0 | 0 | 11,400 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 706.6 | 88.6 | 784 ms | 11.3 ms | 8.0 | 0 | 21,192 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 77.4 | 77.5 | 326 ms | 12.9 ms | 1.0 | 0 | 2,321 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 155.0 | 77.7 | 477 ms | 12.9 ms | 2.0 | 0 | 4,650 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 295.8 | 74.3 | 828 ms | 13.5 ms | 4.0 | 0 | 8,872 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 49.6 | 49.6 | 726 ms | 20.2 ms | 1.0 | 0 | 1,486 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 103.5 | 51.9 | 1,165 ms | 19.3 ms | 2.0 | 0 | 3,104 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 29.7 | 29.7 | 1,751 ms | 33.7 ms | 1.0 | 0 | 890 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.