Speed test matrix
gemma-4-E2B-it on H200 NVL 141GB
sglang · 0.5.6.post3.dev5791+gf7d4e44d8 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev5791+gf7d4e44d8
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
f7d4e44d82ac
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-gemma-sglang/bin/sglang" serve --model-path google/gemma-4-E2B-it --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 131072 --mem-fraction-static 0.851332 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context35.3K232 ms TTFT
16K context29K561 ms TTFT
32K context19.3K1,674 ms TTFT
64K context11.4K5,668 ms TTFT
123K context6,41719,315 ms TTFT
128K context6,21720,747 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 171.884 ms TTFT | 343.898 ms TTFT | 681.5133 ms TTFT | 1,354.0218 ms TTFT | 2,685.5391 ms TTFT | 5,027.0721 ms TTFT | 7,787.41,443 ms TTFT | 11.3K2,821 ms TTFT |
| 16K | 140.2123 ms TTFT | 285.0136 ms TTFT | 569.4222 ms TTFT | 1,143.1362 ms TTFT | 2,249.8672 ms TTFT | 4,136.41,156 ms TTFT | 6,215.92,909 ms TTFT | 8,289.75,796 ms TTFT |
| 32K | 104.5187 ms TTFT | 215.2266 ms TTFT | 436.4427 ms TTFT | 877.4807 ms TTFT | 1,760.51,539 ms TTFT | 3,315.53,018 ms TTFT | 4,465.05,973 ms TTFT | Over capacity |
| 64K | 69.1322 ms TTFT | 146.2474 ms TTFT | 295.0833 ms TTFT | 603.51,599 ms TTFT | 1,252.23,159 ms TTFT | 2,329.56,269 ms TTFT | Over capacity | Over capacity |
| 123K | 42.1699 ms TTFT | 90.91,149 ms TTFT | 184.42,123 ms TTFT | 398.64,185 ms TTFT | 809.18,311 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 35.3K | — | 232 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29K | — | 561 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.3K | — | 1,674 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.4K | — | 5,668 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 6,417.0 | — | 19,315 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,217.0 | — | 20,747 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 171.8 | 174.3 | 84 ms | 5.7 ms | 1.0 | 1 | 5,154 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 343.8 | 174.9 | 98 ms | 5.7 ms | 1.9 | 2 | 10,315 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 681.5 | 173.8 | 133 ms | 5.8 ms | 4.0 | 4 | 20,444 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,354.0 | 173.4 | 218 ms | 5.8 ms | 8.0 | 8 | 40,622 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,685.5 | 170.9 | 391 ms | 5.9 ms | 16.0 | 16 | 80,561 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 5,027.0 | 160.3 | 721 ms | 6.2 ms | 32.0 | 32 | 150,816 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 7,787.4 | 123.7 | 1,443 ms | 8.1 ms | 64.0 | 64 | 231,100 | 29.7 s |
| Sustained decode | 8K | 128 | Valid | 11.3K | 89.6 | 2,821 ms | 11.2 ms | 128.0 | 0 | 339,136 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 140.2 | 143.5 | 123 ms | 7.0 ms | 1.0 | 1 | 4,205 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 285.0 | 146.2 | 136 ms | 6.8 ms | 2.0 | 2 | 8,549 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 569.4 | 146.6 | 222 ms | 6.8 ms | 4.0 | 4 | 17,081 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,143.1 | 147.8 | 362 ms | 6.8 ms | 7.9 | 8 | 34,295 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,249.8 | 145.1 | 672 ms | 6.9 ms | 15.8 | 16 | 67,487 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,136.4 | 134.8 | 1,156 ms | 7.4 ms | 31.6 | 32 | 124,098 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 6,215.9 | 99.5 | 2,909 ms | 10.0 ms | 64.0 | 0 | 186,480 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 8,289.7 | 67.6 | 5,796 ms | 14.8 ms | 128.0 | 0 | 248,704 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 104.5 | 104.8 | 187 ms | 9.5 ms | 1.0 | 0 | 3,134 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 215.2 | 108.1 | 266 ms | 9.2 ms | 2.0 | 0 | 6,456 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 436.4 | 109.9 | 427 ms | 9.1 ms | 4.0 | 0 | 13,092 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 877.4 | 110.9 | 807 ms | 9.0 ms | 8.0 | 0 | 26,321 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,760.5 | 111.9 | 1,539 ms | 8.9 ms | 16.0 | 0 | 52,817 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 3,315.5 | 106.5 | 3,018 ms | 9.4 ms | 32.0 | 0 | 99,456 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,465.0 | 73.3 | 5,973 ms | 13.6 ms | 64.0 | 0 | 133,952 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 69.1 | 69.2 | 322 ms | 14.5 ms | 1.0 | 0 | 2,072 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 146.2 | 73.3 | 474 ms | 13.6 ms | 2.0 | 0 | 4,386 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 295.0 | 74.1 | 833 ms | 13.5 ms | 4.0 | 0 | 8,848 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 603.5 | 76.2 | 1,599 ms | 13.1 ms | 8.0 | 0 | 18,104 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,252.2 | 80.3 | 3,159 ms | 12.5 ms | 16.0 | 0 | 37,568 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,329.5 | 76.6 | 6,269 ms | 13.1 ms | 32.0 | 0 | 69,888 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 42.1 | 42.1 | 699 ms | 23.8 ms | 1.0 | 0 | 1,262 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 90.9 | 45.5 | 1,149 ms | 22.0 ms | 2.0 | 0 | 2,724 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 184.4 | 46.6 | 2,123 ms | 21.5 ms | 4.0 | 0 | 5,532 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 398.6 | 51.4 | 4,185 ms | 19.5 ms | 8.0 | 0 | 11,952 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 809.1 | 54.4 | 8,311 ms | 18.4 ms | 16.0 | 0 | 24,272 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.