Speed test matrix
gemma-4-E2B-it on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-sglang/bin/sglang" serve --model-path RedHatAI/gemma-4-E2B-it --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 131072 --mem-fraction-static 0.853280 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context33K248 ms TTFT
16K context26.8K608 ms TTFT
32K context18.3K1,768 ms TTFT
64K context11K5,890 ms TTFT
123K context6,23919,867 ms TTFT
128K context6,02321,413 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 175.2Requests queued81 ms TTFT | 76.9124 ms TTFT | 239.5163 ms TTFT | 1,343.7212 ms TTFT | 2,700.2407 ms TTFT | 4,530.1745 ms TTFT | 6,596.91,514 ms TTFT | 11.2K2,899 ms TTFT |
| 16K | 140.9115 ms TTFT | 243.8151 ms TTFT | 570.8209 ms TTFT | 1,122.9363 ms TTFT | 2,192.9740 ms TTFT | 3,257.31,508 ms TTFT | 6,236.32,927 ms TTFT | 7,658.85,721 ms TTFT |
| 32K | 103.9181 ms TTFT | 104.8266 ms TTFT | 417.1416 ms TTFT | 873.1801 ms TTFT | 1,815.31,565 ms TTFT | 3,300.23,037 ms TTFT | 4,444.56,020 ms TTFT | Over capacity |
| 64K | 68.1319 ms TTFT | 96.2510 ms TTFT | 282.0846 ms TTFT | 590.21,720 ms TTFT | 1,155.63,220 ms TTFT | 1,510.36,416 ms TTFT | Over capacity | Over capacity |
| 123K | 41.5774 ms TTFT | 90.41,161 ms TTFT | 177.62,026 ms TTFT | 368.24,116 ms TTFT | 697.38,385 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33K | — | 248 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 26.8K | — | 608 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 18.3K | — | 1,768 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11K | — | 5,890 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 6,239.0 | — | 19,867 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,023.0 | — | 21,413 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 175.2 | 177.9 | 81 ms | 5.6 ms | 1.0 | 1 | 5,256 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 76.9 | 38.4 | 124 ms | 26.0 ms | 3.0 | 0 | 2,306 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 239.5 | 57.1 | 163 ms | 17.5 ms | 6.8 | 0 | 7,184 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,343.7 | 171.7 | 212 ms | 5.8 ms | 8.9 | 8 | 40,306 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,700.2 | 173.7 | 407 ms | 5.8 ms | 18.4 | 16 | 81,011 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,530.1 | 145.8 | 745 ms | 6.9 ms | 38.3 | 32 | 135,887 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,596.9 | 104.0 | 1,514 ms | 9.6 ms | 91.7 | 0 | 197,952 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 11.2K | 86.0 | 2,899 ms | 11.6 ms | 130.1 | 0 | 337,456 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 140.9 | 144.5 | 115 ms | 6.9 ms | 1.0 | 1 | 4,227 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 243.8 | 115.8 | 151 ms | 8.6 ms | 40.4 | 0 | 7,315 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 570.8 | 147.3 | 209 ms | 6.8 ms | 4.0 | 4 | 17,123 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,122.9 | 146.3 | 363 ms | 6.8 ms | 10.1 | 8 | 33,686 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,192.9 | 145.6 | 740 ms | 6.9 ms | 20.6 | 16 | 65,790 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,257.3 | 103.0 | 1,508 ms | 9.7 ms | 46.4 | 0 | 97,728 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 6,236.3 | 96.7 | 2,927 ms | 10.3 ms | 64.0 | 0 | 187,072 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 7,658.8 | 60.1 | 5,721 ms | 16.6 ms | 144.7 | 0 | 229,760 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 103.9 | 104.2 | 181 ms | 9.6 ms | 1.0 | 0 | 3,117 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 104.8 | 52.6 | 266 ms | 19.0 ms | 130.0 | 0 | 3,142 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 417.1 | 104.5 | 416 ms | 9.6 ms | 5.1 | 0 | 12,512 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 873.1 | 109.4 | 801 ms | 9.1 ms | 8.0 | 0 | 26,192 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,815.3 | 114.5 | 1,565 ms | 8.7 ms | 16.0 | 0 | 54,448 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,300.2 | 106.1 | 3,037 ms | 9.4 ms | 32.0 | 0 | 99,008 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,444.5 | 70.2 | 6,020 ms | 14.2 ms | 64.0 | 0 | 133,312 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 68.1 | 68.2 | 319 ms | 14.7 ms | 1.0 | 0 | 2,043 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 96.2 | 47.9 | 510 ms | 20.9 ms | 62.6 | 0 | 2,884 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 282.0 | 70.8 | 846 ms | 14.1 ms | 5.9 | 0 | 8,456 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 590.2 | 74.4 | 1,720 ms | 13.4 ms | 10.1 | 0 | 17,704 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,155.6 | 73.5 | 3,220 ms | 13.6 ms | 20.1 | 0 | 34,656 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,510.3 | 49.1 | 6,416 ms | 20.4 ms | 45.2 | 0 | 45,312 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 41.5 | 41.5 | 774 ms | 24.1 ms | 1.0 | 0 | 1,245 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 90.4 | 45.3 | 1,161 ms | 22.1 ms | 34.0 | 0 | 2,712 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 177.6 | 44.6 | 2,026 ms | 22.4 ms | 14.5 | 0 | 5,324 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 368.2 | 47.3 | 4,116 ms | 21.1 ms | 12.7 | 0 | 11,040 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 697.3 | 46.9 | 8,385 ms | 21.3 ms | 24.9 | 0 | 20,912 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.