Speed test matrix
gemma-4-E4B-it on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-google-vllm-73a5831127a9/bin/vllm" serve google/gemma-4-E4B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context27.1K303 ms TTFT
16K context24.8K657 ms TTFT
32K context19.3K1,677 ms TTFT
64K context13.3K4,854 ms TTFT
123K context8,40314,751 ms TTFT
128K context8,14615,833 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 100.841 ms TTFT | 186.455 ms TTFT | 388.276 ms TTFT | 751.0109 ms TTFT | Looped | Looped | Looped | Looped |
| 16K | 96.062 ms TTFT | 178.578 ms TTFT | 370.7113 ms TTFT | 715.2175 ms TTFT | Looped | Looped | Looped | Looped |
| 32K | 89.3107 ms TTFT | 167.1144 ms TTFT | 345.2208 ms TTFT | Looped | Looped | 1,847.8944 ms TTFT | 2,697.51,949 ms TTFT | Over capacity |
| 64K | 78.4196 ms TTFT | 147.6252 ms TTFT | 304.7357 ms TTFT | Looped | 871.2996 ms TTFT | Looped | Over capacity | Over capacity |
| 123K | 64.2439 ms TTFT | 122.4597 ms TTFT | 250.7935 ms TTFT | Looped | 664.72,579 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.1K | — | 303 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 24.8K | — | 657 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.3K | — | 1,677 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.3K | — | 4,854 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 8,403.0 | — | 14,751 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,146.0 | — | 15,833 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 100.8 | 101.2 | 41 ms | 9.9 ms | 1.0 | 0 | 3,025 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 186.4 | 93.6 | 55 ms | 10.7 ms | 2.0 | 0 | 5,592 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 388.2 | 97.5 | 76 ms | 10.3 ms | 4.0 | 0 | 11,648 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 751.0 | 94.4 | 109 ms | 10.6 ms | 8.0 | 0 | 22,681 | 30.2 s |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 96.0 | 96.1 | 62 ms | 10.4 ms | 1.0 | 0 | 2,879 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 178.5 | 89.4 | 78 ms | 11.2 ms | 2.0 | 0 | 5,355 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 370.7 | 92.8 | 113 ms | 10.8 ms | 4.0 | 0 | 11,121 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 715.2 | 89.6 | 175 ms | 11.2 ms | 8.0 | 0 | 21,456 | 30.0 s |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 89.3 | 89.4 | 107 ms | 11.2 ms | 1.0 | 0 | 2,679 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 167.1 | 83.6 | 144 ms | 12.0 ms | 2.0 | 0 | 5,012 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 345.2 | 86.4 | 208 ms | 11.6 ms | 4.0 | 0 | 10,356 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 1,847.8 | 60.0 | 944 ms | 16.7 ms | 32.0 | 0 | 55,599 | 30.1 s |
| Sustained decode | 32K | 64 | Valid | 2,697.5 | 44.2 | 1,949 ms | 22.6 ms | 64.0 | 0 | 81,109 | 30.1 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 78.4 | 78.5 | 196 ms | 12.7 ms | 1.0 | 0 | 2,352 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 147.6 | 73.9 | 252 ms | 13.5 ms | 2.0 | 0 | 4,428 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 304.7 | 76.2 | 357 ms | 13.1 ms | 4.0 | 0 | 9,141 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 871.2 | 56.5 | 996 ms | 17.7 ms | 16.0 | 0 | 26,136 | 30.0 s |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 64.2 | 64.3 | 439 ms | 15.6 ms | 1.0 | 0 | 1,923 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 122.4 | 61.3 | 597 ms | 16.3 ms | 2.0 | 0 | 3,671 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 250.7 | 62.6 | 935 ms | 16.0 ms | 4.0 | 0 | 7,520 | 30.0 s |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Valid | 664.7 | 41.6 | 2,579 ms | 24.0 ms | 16.0 | 0 | 20,049 | 30.2 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.