Speed test matrix
gemma-4-12B-it-int4-AutoRound on H200 NVL 141GB
sglang · 0.5.6.post3.dev5791+gf7d4e44d8 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev5791+gf7d4e44d8
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
f7d4e44d82ac
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-intel-sglang/bin/sglang" serve --model-path Intel/gemma-4-12B-it-int4-AutoRound --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 262144 --mem-fraction-static 0.851795 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context8,267992 ms TTFT
16K context7,2532,242 ms TTFT
32K context5,4775,910 ms TTFT
64K context3,74917,224 ms TTFT
128K context2,31055,824 ms TTFT
251K context1,331189,875 ms TTFT
256K context1,308197,072 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 116.290 ms TTFT | 231.0115 ms TTFT | 455.9165 ms TTFT | 884.3237 ms TTFT | 1,574.1415 ms TTFT | 2,337.2778 ms TTFT | Over capacity | Over capacity |
| 16K | 99.8127 ms TTFT | 202.0161 ms TTFT | 400.4230 ms TTFT | 781.7403 ms TTFT | 1,400.0764 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 77.4198 ms TTFT | 161.1273 ms TTFT | 319.7417 ms TTFT | 639.9792 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 53.6351 ms TTFT | 113.3507 ms TTFT | 231.7823 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 32.8786 ms TTFT | 70.71,252 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 18.0Requests queued1,872 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,267.0 | — | 992 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,253.0 | — | 2,242 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,477.0 | — | 5,910 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,749.0 | — | 17,224 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,310.0 | — | 55,824 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,331.0 | — | 189,875 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,308.0 | — | 197,072 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 116.2 | 116.7 | 90 ms | 8.6 ms | 1.0 | 0 | 3,485 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 231.0 | 116.3 | 115 ms | 8.6 ms | 2.0 | 0 | 6,930 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 455.9 | 114.9 | 165 ms | 8.7 ms | 4.0 | 0 | 13,676 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 884.3 | 111.5 | 237 ms | 9.0 ms | 8.0 | 0 | 26,528 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,574.1 | 99.1 | 415 ms | 10.1 ms | 16.0 | 0 | 47,216 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,337.2 | 73.5 | 778 ms | 13.6 ms | 32.0 | 0 | 70,112 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 99.8 | 100.1 | 127 ms | 10.0 ms | 1.0 | 0 | 2,992 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 202.0 | 101.5 | 161 ms | 9.9 ms | 2.0 | 0 | 6,058 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 400.4 | 100.8 | 230 ms | 9.9 ms | 4.0 | 0 | 12,012 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 781.7 | 98.5 | 403 ms | 10.2 ms | 8.0 | 0 | 23,448 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,400.0 | 88.3 | 764 ms | 11.3 ms | 16.0 | 0 | 41,999 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 77.4 | 77.6 | 198 ms | 12.9 ms | 1.0 | 0 | 2,322 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 161.1 | 80.7 | 273 ms | 12.4 ms | 2.0 | 0 | 4,834 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 319.7 | 80.2 | 417 ms | 12.5 ms | 4.0 | 0 | 9,592 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 639.9 | 80.6 | 792 ms | 12.4 ms | 8.0 | 0 | 19,192 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 53.6 | 53.6 | 351 ms | 18.6 ms | 1.0 | 0 | 1,607 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 113.3 | 56.7 | 507 ms | 17.6 ms | 2.0 | 0 | 3,398 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 231.7 | 58.1 | 823 ms | 17.2 ms | 4.0 | 0 | 6,948 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 32.8 | 32.9 | 786 ms | 30.4 ms | 1.0 | 0 | 985 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 70.7 | 35.4 | 1,252 ms | 28.2 ms | 2.0 | 0 | 2,118 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Requests queued | 18.0 | 18.8 | 1,872 ms | 53.1 ms | 0.9 | 0 | 540 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.