Speed test matrix
gemma-4-E2B-it on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-google-vllm-73a5831127a9/bin/vllm" serve google/gemma-4-E2B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context39.8K206 ms TTFT
16K context34.8K467 ms TTFT
32K context25.2K1,282 ms TTFT
64K context16K4,041 ms TTFT
123K context9,49613,054 ms TTFT
128K context9,19514,027 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 167.736 ms TTFT | 312.749 ms TTFT | 616.980 ms TTFT | 1,237.9125 ms TTFT | 2,330.1218 ms TTFT | 3,831.5464 ms TTFT | 6,271.7620 ms TTFT | 9,035.11,257 ms TTFT |
| 16K | 155.458 ms TTFT | 290.793 ms TTFT | 574.6125 ms TTFT | 1,139.9187 ms TTFT | 2,147.2334 ms TTFT | 3,509.8460 ms TTFT | 5,723.21,001 ms TTFT | 7,919.82,122 ms TTFT |
| 32K | 138.6101 ms TTFT | 261.1156 ms TTFT | 515.8213 ms TTFT | 1,002.8351 ms TTFT | 1,903.3613 ms TTFT | 2,972.61,042 ms TTFT | 4,743.01,996 ms TTFT | 6,008.73,815 ms TTFT |
| 64K | 114.6195 ms TTFT | 218.2338 ms TTFT | 432.4370 ms TTFT | 838.8598 ms TTFT | 1,601.11,030 ms TTFT | 2,260.72,012 ms TTFT | 3,441.13,941 ms TTFT | 4,142.18,234 ms TTFT |
| 123K | 86.7432 ms TTFT | 166.7637 ms TTFT | 331.4879 ms TTFT | 640.61,488 ms TTFT | 1,228.62,689 ms TTFT | 1,567.24,942 ms TTFT | 2,267.610,028 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 39.8K | — | 206 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 34.8K | — | 467 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 25.2K | — | 1,282 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 16K | — | 4,041 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 9,496.0 | — | 13,054 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 9,195.0 | — | 14,027 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 167.7 | 169.6 | 36 ms | 5.9 ms | 1.0 | 1 | 5,031 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 312.7 | 158.5 | 49 ms | 6.3 ms | 2.0 | 2 | 9,382 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 616.9 | 158.8 | 80 ms | 6.3 ms | 4.0 | 4 | 18,505 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,237.9 | 160.3 | 125 ms | 6.2 ms | 7.7 | 8 | 37,145 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,330.1 | 153.1 | 218 ms | 6.5 ms | 15.4 | 16 | 69,904 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,831.5 | 132.1 | 464 ms | 7.6 ms | 30.8 | 32 | 114,991 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,271.7 | 101.3 | 620 ms | 9.9 ms | 64.0 | 0 | 207,316 | 33.1 s |
| Sustained decode | 8K | 128 | Valid | 9,035.1 | 74.4 | 1,257 ms | 13.4 ms | 128.0 | 0 | 295,424 | 32.7 s |
| Sustained decode | 16K | 1 | Valid | 155.4 | 156.5 | 58 ms | 6.4 ms | 1.0 | 1 | 4,662 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 290.7 | 146.5 | 93 ms | 6.8 ms | 2.0 | 2 | 8,724 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 574.6 | 145.0 | 125 ms | 6.9 ms | 3.9 | 4 | 17,237 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,139.9 | 145.6 | 187 ms | 6.9 ms | 7.7 | 8 | 34,198 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,147.2 | 138.5 | 334 ms | 7.2 ms | 15.4 | 16 | 64,588 | 30.1 s |
| Sustained decode | 16K | 32 | Valid | 3,509.8 | 111.7 | 460 ms | 9.0 ms | 32.0 | 0 | 106,325 | 30.3 s |
| Sustained decode | 16K | 64 | Valid | 5,723.2 | 91.5 | 1,001 ms | 10.9 ms | 64.0 | 0 | 178,048 | 31.1 s |
| Sustained decode | 16K | 128 | Valid | 7,919.8 | 64.4 | 2,122 ms | 15.5 ms | 128.0 | 0 | 253,440 | 32.0 s |
| Sustained decode | 32K | 1 | Valid | 138.6 | 140.1 | 101 ms | 7.1 ms | 1.0 | 1 | 4,157 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 261.1 | 132.1 | 156 ms | 7.6 ms | 2.0 | 2 | 7,831 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 515.8 | 131.1 | 213 ms | 7.6 ms | 3.9 | 4 | 15,473 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,002.8 | 127.7 | 351 ms | 7.8 ms | 8.0 | 8 | 30,085 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,903.3 | 121.6 | 613 ms | 8.2 ms | 15.8 | 16 | 57,100 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,972.6 | 93.8 | 1,042 ms | 10.7 ms | 32.0 | 0 | 90,496 | 30.4 s |
| Sustained decode | 32K | 64 | Valid | 4,743.0 | 75.0 | 1,996 ms | 13.3 ms | 64.0 | 0 | 152,576 | 32.2 s |
| Sustained decode | 32K | 128 | Valid | 6,008.7 | 48.9 | 3,815 ms | 20.4 ms | 128.0 | 0 | 209,280 | 34.8 s |
| Sustained decode | 64K | 1 | Valid | 114.6 | 114.8 | 195 ms | 8.7 ms | 1.0 | 0 | 3,439 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 218.2 | 109.3 | 338 ms | 9.1 ms | 2.0 | 0 | 6,546 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 432.4 | 108.3 | 370 ms | 9.2 ms | 4.0 | 0 | 12,971 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 838.8 | 105.1 | 598 ms | 9.5 ms | 8.0 | 0 | 25,163 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,601.1 | 100.3 | 1,030 ms | 10.0 ms | 16.0 | 0 | 48,031 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,260.7 | 71.2 | 2,012 ms | 14.1 ms | 32.0 | 0 | 68,896 | 30.5 s |
| Sustained decode | 64K | 64 | Valid | 3,441.1 | 54.2 | 3,941 ms | 18.4 ms | 64.0 | 0 | 104,128 | 30.3 s |
| Sustained decode | 64K | 128 | Valid | 4,142.1 | 33.7 | 8,234 ms | 29.7 ms | 128.0 | 0 | 133,632 | 32.3 s |
| Sustained decode | 123K | 1 | Valid | 86.7 | 86.9 | 432 ms | 11.5 ms | 1.0 | 0 | 2,599 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 166.7 | 83.5 | 637 ms | 12.0 ms | 2.0 | 0 | 5,002 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 331.4 | 82.9 | 879 ms | 12.1 ms | 4.0 | 0 | 9,942 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 640.6 | 80.0 | 1,488 ms | 12.5 ms | 8.0 | 0 | 19,216 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,228.6 | 76.5 | 2,689 ms | 13.1 ms | 16.0 | 0 | 36,856 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,567.2 | 49.4 | 4,942 ms | 20.2 ms | 32.0 | 0 | 47,616 | 30.4 s |
| Sustained decode | 123K | 64 | Valid | 2,267.6 | 35.9 | 10,028 ms | 27.8 ms | 64.0 | 0 | 69,440 | 30.6 s |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.