Speed test matrix
granite-4.2-3b on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev220+gfa7784d14 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev220+gfa7784d14
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
fa7784d140ee
Run
- Served model
- granite-4.2-3b
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-ibm-granite-sglang-fa7784d140ee/bin/sglang" serve --model-path ibm-granite/granite-4.2-3b --served-model-name granite-4.2-3b --tp 1 --reasoning-parser auto --tool-call-parser auto --context-length 131072 --mem-fraction-static 0.78 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --attention-backend flashinfer --bf16-gemm-backend torch
Prefill
prompt tok/s by context length8K context37K222 ms TTFT
16K context31.8K509 ms TTFT
32K context24.8K1,303 ms TTFT
64K context16.8K3,831 ms TTFT
123K context10.3K11,956 ms TTFT
128K context10.2K12,646 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 153.531 ms TTFT | 277.443 ms TTFT | 523.563 ms TTFT | 963.593 ms TTFT | 1,502.3168 ms TTFT | 2,138.9318 ms TTFT | 3,350.4657 ms TTFT | Over capacity |
| 16K | 143.439 ms TTFT | 258.758 ms TTFT | 481.678 ms TTFT | 857.4160 ms TTFT | 1,224.2331 ms TTFT | 1,572.0643 ms TTFT | Over capacity | Over capacity |
| 32K | 127.362 ms TTFT | 230.297 ms TTFT | 423.1171 ms TTFT | 689.6355 ms TTFT | 878.2704 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 104.2101 ms TTFT | Looped | 343.0342 ms TTFT | 489.8693 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 78.3194 ms TTFT | 143.6363 ms TTFT | 252.2715 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 37K | — | 222 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.8K | — | 509 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 24.8K | — | 1,303 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16.8K | — | 3,831 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 10.3K | — | 11,956 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.2K | — | 12,646 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 153.5 | 154.4 | 31 ms | 6.5 ms | 1.0 | 1 | 4,604 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 277.4 | 140.7 | 43 ms | 7.1 ms | 2.0 | 2 | 8,322 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 523.5 | 134.3 | 63 ms | 7.4 ms | 4.0 | 4 | 15,704 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 963.5 | 126.4 | 93 ms | 7.9 ms | 7.9 | 8 | 28,902 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,502.3 | 94.9 | 168 ms | 10.5 ms | 16.0 | 0 | 45,073 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,138.9 | 68.0 | 318 ms | 14.7 ms | 32.0 | 0 | 64,159 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,350.4 | 53.4 | 657 ms | 18.7 ms | 64.0 | 0 | 100,480 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 143.4 | 144.6 | 39 ms | 6.9 ms | 1.0 | 1 | 4,303 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 258.7 | 130.3 | 58 ms | 7.7 ms | 2.0 | 2 | 7,760 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 481.6 | 122.7 | 78 ms | 8.1 ms | 4.0 | 4 | 14,444 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 857.4 | 107.9 | 160 ms | 9.3 ms | 8.0 | 0 | 25,720 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,224.2 | 77.2 | 331 ms | 13.0 ms | 16.0 | 0 | 36,720 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,572.0 | 49.6 | 643 ms | 20.1 ms | 32.0 | 0 | 47,136 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 127.3 | 128.8 | 62 ms | 7.8 ms | 1.0 | 1 | 3,819 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 230.2 | 115.5 | 97 ms | 8.7 ms | 2.0 | 0 | 6,906 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 423.1 | 106.4 | 171 ms | 9.4 ms | 4.0 | 0 | 12,693 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 689.6 | 86.8 | 355 ms | 11.5 ms | 8.0 | 0 | 20,680 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 878.2 | 55.5 | 704 ms | 18.0 ms | 16.0 | 0 | 26,337 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 104.2 | 104.3 | 101 ms | 9.6 ms | 1.0 | 0 | 3,124 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Valid | 343.0 | 86.2 | 342 ms | 11.6 ms | 4.0 | 0 | 10,291 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 489.8 | 61.8 | 693 ms | 16.2 ms | 8.0 | 0 | 14,688 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 78.3 | 78.4 | 194 ms | 12.8 ms | 1.0 | 0 | 2,349 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 143.6 | 72.0 | 363 ms | 13.9 ms | 2.0 | 0 | 4,306 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 252.2 | 63.6 | 715 ms | 15.7 ms | 4.0 | 0 | 7,564 | 30.0 s |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.