Speed test matrix
gemma-4-12B-it on H200 NVL 141GB
sglang · 0.5.6.post3.dev5791+gf7d4e44d8 · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev5791+gf7d4e44d8
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
f7d4e44d82ac
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-gemma-sglang/bin/sglang" serve --model-path google/gemma-4-12B-it --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 262144 --mem-fraction-static 0.852244 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context11.8K696 ms TTFT
16K context9,5891,696 ms TTFT
32K context6,7654,785 ms TTFT
64K context4,31114,975 ms TTFT
128K context2,51151,370 ms TTFT
251K context1,392181,603 ms TTFT
256K context1,369188,370 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 85.694 ms TTFT | 171.8124 ms TTFT | 339.7176 ms TTFT | 665.0231 ms TTFT | 1,234.2396 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 76.2131 ms TTFT | 154.3169 ms TTFT | 307.5244 ms TTFT | 605.1394 ms TTFT | 1,130.2755 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 62.3205 ms TTFT | 128.5273 ms TTFT | 258.8417 ms TTFT | 515.8788 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 45.7346 ms TTFT | 95.9502 ms TTFT | 196.3803 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 29.8788 ms TTFT | 63.41,237 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 17.0Requests queued1,894 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.8K | — | 696 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,589.0 | — | 1,696 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,765.0 | — | 4,785 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,311.0 | — | 14,975 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,511.0 | — | 51,370 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,392.0 | — | 181,603 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,369.0 | — | 188,370 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 85.6 | 85.8 | 94 ms | 11.7 ms | 1.0 | 0 | 2,568 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 171.8 | 86.2 | 124 ms | 11.6 ms | 2.0 | 0 | 5,152 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 339.7 | 85.3 | 176 ms | 11.7 ms | 4.0 | 0 | 10,188 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 665.0 | 83.5 | 231 ms | 12.0 ms | 8.0 | 0 | 19,952 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,234.2 | 77.5 | 396 ms | 12.9 ms | 16.0 | 0 | 37,024 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 76.2 | 76.3 | 131 ms | 13.1 ms | 1.0 | 0 | 2,285 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 154.3 | 77.3 | 169 ms | 12.9 ms | 2.0 | 0 | 4,626 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 307.5 | 77.1 | 244 ms | 13.0 ms | 4.0 | 0 | 9,220 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 605.1 | 75.9 | 394 ms | 13.2 ms | 8.0 | 0 | 18,152 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,130.2 | 70.9 | 755 ms | 14.1 ms | 16.0 | 0 | 33,904 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 62.3 | 62.4 | 205 ms | 16.0 ms | 1.0 | 0 | 1,869 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 128.5 | 64.3 | 273 ms | 15.6 ms | 2.0 | 0 | 3,854 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 258.8 | 64.9 | 417 ms | 15.4 ms | 4.0 | 0 | 7,760 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 515.8 | 64.7 | 788 ms | 15.4 ms | 8.0 | 0 | 15,472 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 45.7 | 45.7 | 346 ms | 21.9 ms | 1.0 | 0 | 1,370 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 95.9 | 48.0 | 502 ms | 20.9 ms | 2.0 | 0 | 2,876 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 196.3 | 49.2 | 803 ms | 20.3 ms | 4.0 | 0 | 5,888 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 29.8 | 29.8 | 788 ms | 33.6 ms | 1.0 | 0 | 893 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 63.4 | 31.8 | 1,237 ms | 31.5 ms | 2.0 | 0 | 1,900 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Requests queued | 17.0 | 17.8 | 1,894 ms | 56.1 ms | 0.9 | 0 | 511 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.