Speed test matrix
gemma-4-E2B-it on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev236+g1093c501d · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev236+g1093c501d
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1093c501dfef
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-google-sglang-1093c501dfef/bin/sglang" serve --model-path google/gemma-4-E2B-it --served-model-name gemma --trust-remote-code --tp 1 --reasoning-parser gemma4 --tool-call-parser gemma4 --context-length 131072 --mem-fraction-static 0.78 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend triton --bf16-gemm-backend torch
Prefill
prompt tok/s by context length8K context33.5K245 ms TTFT
16K context28.1K577 ms TTFT
32K context21.1K1,536 ms TTFT
64K context13.1K4,911 ms TTFT
123K context6,71418,446 ms TTFT
128K context6,48119,887 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 153.3127 ms TTFT | 289.649 ms TTFT | 571.3701 ms TTFT | 1,164.361 ms TTFT | 2,268.385 ms TTFT | 3,599.7109 ms TTFT | 5,739.4156 ms TTFT | 7,851.5227 ms TTFT |
| 16K | 128.945 ms TTFT | 247.169 ms TTFT | 490.781 ms TTFT | 1,000.186 ms TTFT | 1,957.5109 ms TTFT | 2,938.9120 ms TTFT | 4,540.9216 ms TTFT | Over capacity |
| 32K | 99.170 ms TTFT | 194.1108 ms TTFT | 388.6133 ms TTFT | 789.9136 ms TTFT | 1,560.6162 ms TTFT | 2,131.71,456 ms TTFT | Over capacity | Over capacity |
| 64K | 67.6121 ms TTFT | 135.3183 ms TTFT | 272.9235 ms TTFT | 554.7243 ms TTFT | 1,102.4270 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 42.4216 ms TTFT | 86.5325 ms TTFT | 173.5430 ms TTFT | 355.9451 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 33.5K | — | 245 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 28.1K | — | 577 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.1K | — | 1,536 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.1K | — | 4,911 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 6,714.0 | — | 18,446 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,481.0 | — | 19,887 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 153.3 | 155.6 | 127 ms | 6.4 ms | 1.0 | 1 | 4,598 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 289.6 | 147.3 | 49 ms | 6.8 ms | 2.0 | 2 | 8,688 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 571.3 | 145.7 | 701 ms | 6.9 ms | 4.0 | 4 | 17,136 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,164.3 | 148.5 | 61 ms | 6.7 ms | 8.0 | 8 | 34,928 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,268.3 | 145.2 | 85 ms | 6.9 ms | 16.0 | 16 | 68,048 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,599.7 | 113.8 | 109 ms | 8.8 ms | 32.0 | 0 | 107,968 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 5,739.4 | 91.0 | 156 ms | 11.0 ms | 64.0 | 0 | 172,160 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,851.5 | 62.3 | 227 ms | 16.1 ms | 128.0 | 0 | 235,520 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 128.9 | 131.8 | 45 ms | 7.6 ms | 1.0 | 1 | 3,865 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 247.1 | 125.9 | 69 ms | 7.9 ms | 2.0 | 2 | 7,414 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 490.7 | 125.1 | 81 ms | 8.0 ms | 4.0 | 4 | 14,720 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,000.1 | 127.8 | 86 ms | 7.8 ms | 8.0 | 8 | 30,000 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,957.5 | 125.3 | 109 ms | 8.0 ms | 16.0 | 16 | 58,720 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,938.9 | 92.5 | 120 ms | 10.8 ms | 32.0 | 0 | 88,160 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,540.9 | 71.6 | 216 ms | 14.0 ms | 64.0 | 0 | 136,192 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 99.1 | 99.3 | 70 ms | 10.1 ms | 1.0 | 0 | 2,973 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 194.1 | 97.3 | 108 ms | 10.3 ms | 2.0 | 0 | 5,820 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 388.6 | 97.5 | 133 ms | 10.3 ms | 4.0 | 0 | 11,654 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 789.9 | 99.1 | 136 ms | 10.1 ms | 8.0 | 0 | 23,695 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,560.6 | 97.9 | 162 ms | 10.2 ms | 16.0 | 0 | 46,816 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,131.7 | 66.9 | 1,456 ms | 15.0 ms | 32.0 | 0 | 63,936 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 67.6 | 67.6 | 121 ms | 14.8 ms | 1.0 | 0 | 2,026 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 135.3 | 67.7 | 183 ms | 14.8 ms | 2.0 | 0 | 4,058 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 272.9 | 68.3 | 235 ms | 14.6 ms | 4.0 | 0 | 8,183 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 554.7 | 69.4 | 243 ms | 14.4 ms | 8.0 | 0 | 16,640 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,102.4 | 68.8 | 270 ms | 14.5 ms | 16.0 | 0 | 33,072 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 42.4 | 42.4 | 216 ms | 23.6 ms | 1.0 | 0 | 1,272 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 86.5 | 43.1 | 325 ms | 23.2 ms | 2.0 | 0 | 2,592 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 173.5 | 43.4 | 430 ms | 23.1 ms | 4.0 | 0 | 5,206 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 355.9 | 44.2 | 451 ms | 22.6 ms | 8.0 | 0 | 10,672 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.