Speed test matrix
gemma-4-E2B-it on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/gemma-4-E2B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --async-scheduling
Prefill
prompt tok/s by context length8K context40.4K203 ms TTFT
16K context34.8K467 ms TTFT
32K context25.2K1,282 ms TTFT
64K context16K4,038 ms TTFT
123K context9,53413,001 ms TTFT
128K context9,18814,039 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 167.736 ms TTFT | 312.850 ms TTFT | 617.381 ms TTFT | 1,237.5128 ms TTFT | 2,329.9226 ms TTFT | 3,842.2393 ms TTFT | 6,270.6604 ms TTFT | 9,005.71,147 ms TTFT |
| 16K | 155.366 ms TTFT | 290.892 ms TTFT | 575.3112 ms TTFT | 1,142.0184 ms TTFT | 2,147.8329 ms TTFT | 3,513.7449 ms TTFT | 5,702.0860 ms TTFT | 7,936.11,918 ms TTFT |
| 32K | 138.7101 ms TTFT | 261.0153 ms TTFT | 515.8214 ms TTFT | 1,004.0341 ms TTFT | 1,904.8542 ms TTFT | 2,976.7940 ms TTFT | 4,758.91,836 ms TTFT | 6,138.73,738 ms TTFT |
| 64K | 114.6188 ms TTFT | 218.3304 ms TTFT | 432.6352 ms TTFT | 839.6558 ms TTFT | 1,599.91,148 ms TTFT | 2,267.41,798 ms TTFT | 3,439.83,888 ms TTFT | 4,138.57,605 ms TTFT |
| 123K | 86.7404 ms TTFT | 166.7564 ms TTFT | 331.41,016 ms TTFT | 640.71,604 ms TTFT | 1,229.52,967 ms TTFT | 1,562.55,767 ms TTFT | 2,268.010,414 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 40.4K | — | 203 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.8K | — | 467 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 25.2K | — | 1,282 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16K | — | 4,038 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 9,534.0 | — | 13,001 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,188.0 | — | 14,039 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 167.7 | 169.7 | 36 ms | 5.9 ms | 1.0 | 1 | 5,031 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 312.8 | 160.5 | 50 ms | 6.2 ms | 2.0 | 2 | 9,383 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 617.3 | 158.9 | 81 ms | 6.3 ms | 4.0 | 4 | 18,520 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,237.5 | 160.1 | 128 ms | 6.2 ms | 8.0 | 8 | 37,123 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,329.9 | 153.4 | 226 ms | 6.5 ms | 15.4 | 16 | 69,892 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,842.2 | 133.8 | 393 ms | 7.5 ms | 30.7 | 32 | 115,266 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,270.6 | 101.5 | 604 ms | 9.9 ms | 64.0 | 0 | 203,607 | 32.5 s |
| Sustained decode | 8K | 128 | Valid | 9,005.7 | 74.1 | 1,147 ms | 13.5 ms | 128.0 | 0 | 300,672 | 33.4 s |
| Sustained decode | 16K | 1 | Valid | 155.3 | 156.4 | 66 ms | 6.4 ms | 1.0 | 1 | 4,659 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 290.8 | 146.7 | 92 ms | 6.8 ms | 2.0 | 2 | 8,725 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 575.3 | 145.7 | 112 ms | 6.9 ms | 4.0 | 4 | 17,260 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,142.0 | 145.1 | 184 ms | 6.9 ms | 8.0 | 8 | 34,258 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,147.8 | 138.8 | 329 ms | 7.2 ms | 15.4 | 16 | 64,434 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,513.7 | 111.8 | 449 ms | 8.9 ms | 32.0 | 0 | 105,403 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,702.0 | 91.2 | 860 ms | 11.0 ms | 64.0 | 0 | 188,032 | 33.0 s |
| Sustained decode | 16K | 128 | Valid | 7,936.1 | 64.5 | 1,918 ms | 15.5 ms | 128.0 | 0 | 252,672 | 31.8 s |
| Sustained decode | 32K | 1 | Valid | 138.7 | 140.1 | 101 ms | 7.1 ms | 1.0 | 1 | 4,160 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 261.0 | 132.2 | 153 ms | 7.6 ms | 2.0 | 2 | 7,830 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 515.8 | 131.1 | 214 ms | 7.6 ms | 3.9 | 4 | 15,475 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,004.0 | 128.2 | 341 ms | 7.8 ms | 7.9 | 8 | 30,122 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,904.8 | 121.4 | 542 ms | 8.2 ms | 15.8 | 16 | 57,117 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,976.7 | 94.0 | 940 ms | 10.6 ms | 32.0 | 0 | 89,296 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,758.9 | 75.2 | 1,836 ms | 13.3 ms | 64.0 | 0 | 150,976 | 31.7 s |
| Sustained decode | 32K | 128 | Valid | 6,138.7 | 49.6 | 3,738 ms | 20.1 ms | 128.0 | 0 | 196,352 | 32.0 s |
| Sustained decode | 64K | 1 | Valid | 114.6 | 114.8 | 188 ms | 8.7 ms | 1.0 | 0 | 3,439 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 218.3 | 109.3 | 304 ms | 9.1 ms | 2.0 | 0 | 6,548 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 432.6 | 108.3 | 352 ms | 9.2 ms | 4.0 | 0 | 12,974 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 839.6 | 105.2 | 558 ms | 9.5 ms | 8.0 | 0 | 25,182 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,599.9 | 100.4 | 1,148 ms | 10.0 ms | 16.0 | 0 | 48,240 | 30.2 s |
| Sustained decode | 64K | 32 | Valid | 2,267.4 | 71.3 | 1,798 ms | 14.0 ms | 32.0 | 0 | 69,056 | 30.5 s |
| Sustained decode | 64K | 64 | Valid | 3,439.8 | 54.2 | 3,888 ms | 18.5 ms | 64.0 | 0 | 104,320 | 30.3 s |
| Sustained decode | 64K | 128 | Valid | 4,138.5 | 33.6 | 7,605 ms | 29.8 ms | 128.0 | 0 | 131,584 | 31.8 s |
| Sustained decode | 123K | 1 | Valid | 86.7 | 86.9 | 404 ms | 11.5 ms | 1.0 | 0 | 2,601 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 166.7 | 83.5 | 564 ms | 12.0 ms | 2.0 | 0 | 5,002 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 331.4 | 82.8 | 1,016 ms | 12.1 ms | 4.0 | 0 | 9,942 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 640.7 | 79.9 | 1,604 ms | 12.5 ms | 8.0 | 0 | 19,224 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,229.5 | 76.3 | 2,967 ms | 13.1 ms | 16.0 | 0 | 36,882 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,562.5 | 49.5 | 5,767 ms | 20.2 ms | 32.0 | 0 | 47,488 | 30.4 s |
| Sustained decode | 123K | 64 | Valid | 2,268.0 | 35.8 | 10,414 ms | 27.9 ms | 64.0 | 0 | 69,312 | 30.6 s |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.