Speed test matrix
Gemma-4-31B-IT-NVFP4 on H200 NVL 141GB ×2
sglang · 0.5.6.post3.dev9335+gdd15fb57b · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9335+gdd15fb57b
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
dd15fb57b5ef
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-nvidia-sglang/bin/sglang" serve --model-path nvidia/Gemma-4-31B-IT-NVFP4 --served-model-name gemma --trust-remote-code --tp 2 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 262144 --mem-fraction-static 0.839976 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context6,5891,244 ms TTFT
16K context6,0272,695 ms TTFT
32K context5,0916,353 ms TTFT
64K context3,80816,945 ms TTFT
128K context2,54850,597 ms TTFT
251K context1,561161,869 ms TTFT
256K context1,536167,742 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | Looped | Looped | Looped | Looped | Looped | Over capacity | Over capacity |
| 16K | 68.9122 ms TTFT | Looped | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
| 32K | 55.1188 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 39.1339 ms TTFT | 83.3516 ms TTFT | 170.5858 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 24.9795 ms TTFT | 54.01,294 ms TTFT | 112.22,380 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 14.51,821 ms TTFT | 32.03,246 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,589.0 | — | 1,244 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,027.0 | — | 2,695 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,091.0 | — | 6,353 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,808.0 | — | 16,945 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,548.0 | — | 50,597 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,561.0 | — | 161,869 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,536.0 | — | 167,742 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 68.9 | 69.1 | 122 ms | 14.5 ms | 1.0 | 0 | 2,067 | 30.0 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 55.1 | 55.1 | 188 ms | 18.1 ms | 1.0 | 0 | 1,652 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 39.1 | 39.1 | 339 ms | 25.6 ms | 1.0 | 0 | 1,173 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 83.3 | 41.7 | 516 ms | 24.0 ms | 2.0 | 0 | 2,499 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 170.5 | 42.7 | 858 ms | 23.4 ms | 4.0 | 0 | 5,116 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 24.9 | 24.9 | 795 ms | 40.1 ms | 1.0 | 0 | 747 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 54.0 | 27.1 | 1,294 ms | 36.9 ms | 2.0 | 0 | 1,620 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 112.2 | 28.4 | 2,380 ms | 35.2 ms | 4.0 | 0 | 3,364 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 14.5 | 14.5 | 1,821 ms | 68.9 ms | 1.0 | 0 | 435 | 29.9 s |
| Sustained decode | 251K | 2 | Valid | 32.0 | 16.2 | 3,246 ms | 61.7 ms | 2.0 | 0 | 960 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.