Speed test matrix
granite-4.2-3b on H200 NVL 141GB
sglang · 0.5.18 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.18
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
Run
- Served model
- granite-4.2-3b
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-ibm-sglang/bin/sglang" serve --model-path ibm-granite/granite-4.2-3b --served-model-name granite-4.2-3b --tp 1 --reasoning-parser auto --tool-call-parser auto --context-length 131072 --mem-fraction-static 0.857703 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16
Prefill
prompt tok/s by context length8K context47.5K172 ms TTFT
16K context42.2K384 ms TTFT
32K context33K978 ms TTFT
64K context22.1K2,910 ms TTFT
123K context13.7K8,997 ms TTFT
128K context13.5K9,491 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 202.147 ms TTFT | 358.762 ms TTFT | 835.682 ms TTFT | 1,476.5107 ms TTFT | 2,001.1186 ms TTFT | 2,427.3332 ms TTFT | 2,734.5658 ms TTFT | Over capacity |
| 16K | 240.466 ms TTFT | 104.194 ms TTFT | 506.4134 ms TTFT | 1,199.4160 ms TTFT | 1,103.3339 ms TTFT | 1,650.4649 ms TTFT | 1,814.41,254 ms TTFT | Over capacity |
| 32K | 211.490 ms TTFT | 48.9163 ms TTFT | 118.1229 ms TTFT | 211.7386 ms TTFT | 663.3708 ms TTFT | 732.31,331 ms TTFT | Over capacity | Over capacity |
| 64K | 191.4148 ms TTFT | 45.0252 ms TTFT | 89.2416 ms TTFT | 177.5745 ms TTFT | 253.61,458 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 144.8256 ms TTFT | 32.0479 ms TTFT | 63.7780 ms TTFT | 100.81,515 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 47.5K | — | 172 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 42.2K | — | 384 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 33K | — | 978 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 22.1K | — | 2,910 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 13.7K | — | 8,997 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.5K | — | 9,491 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 202.1 | 212.4 | 47 ms | 4.9 ms | — | 1 | 6,062 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 358.7 | 197.9 | 62 ms | 5.3 ms | — | 2 | 10,762 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 835.6 | 207.6 | 82 ms | 4.8 ms | — | 4 | 25,065 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,476.5 | 183.5 | 107 ms | 5.5 ms | — | 8 | 44,291 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,001.1 | 136.2 | 186 ms | 7.4 ms | — | 16 | 60,029 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,427.3 | 76.9 | 332 ms | 13.0 ms | — | 0 | 72,815 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,734.5 | 42.6 | 658 ms | 23.5 ms | — | 0 | 82,048 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 240.4 | 241.9 | 66 ms | 4.1 ms | — | 2 | 7,211 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 104.1 | 52.5 | 94 ms | 19.1 ms | — | 0 | 3,122 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 506.4 | 116.2 | 134 ms | 8.6 ms | — | 0 | 15,192 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,199.4 | 152.7 | 160 ms | 6.5 ms | — | 8 | 35,981 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,103.3 | 69.5 | 339 ms | 14.4 ms | — | 0 | 33,087 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,650.4 | 51.5 | 649 ms | 19.4 ms | — | 0 | 49,503 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,814.4 | 28.8 | 1,254 ms | 34.7 ms | — | 0 | 54,398 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 211.4 | 212.5 | 90 ms | 4.7 ms | — | 1 | 6,343 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 48.9 | 24.5 | 163 ms | 40.8 ms | — | 0 | 1,466 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 118.1 | 28.7 | 229 ms | 34.8 ms | — | 0 | 3,543 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 211.7 | 26.6 | 386 ms | 37.6 ms | — | 0 | 6,344 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 663.3 | 39.3 | 708 ms | 25.5 ms | — | 0 | 19,888 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 732.3 | 23.3 | 1,331 ms | 42.9 ms | — | 0 | 21,968 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 191.4 | 191.0 | 148 ms | 5.2 ms | — | 1 | 5,743 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 45.0 | 22.5 | 252 ms | 44.4 ms | — | 0 | 1,348 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 89.2 | 22.3 | 416 ms | 44.8 ms | — | 0 | 2,676 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 177.5 | 21.9 | 745 ms | 45.7 ms | — | 0 | 5,320 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 253.6 | 16.1 | 1,458 ms | 62.0 ms | — | 0 | 7,600 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 144.8 | 148.6 | 256 ms | 6.7 ms | — | 1 | 4,342 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 32.0 | 15.8 | 479 ms | 63.3 ms | — | 0 | 960 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 63.7 | 16.0 | 780 ms | 62.6 ms | — | 0 | 1,908 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 100.8 | 12.8 | 1,515 ms | 78.1 ms | — | 0 | 3,024 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.