Speed test matrix
gemma-4-E4B-it on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/gemma-4-E4B-it --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --async-scheduling
Prefill
prompt tok/s by context length8K context27.1K303 ms TTFT
16K context24.7K657 ms TTFT
32K context19.3K1,677 ms TTFT
64K context13.2K4,881 ms TTFT
123K context8,37614,800 ms TTFT
128K context8,10715,910 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 101.039 ms TTFT | 186.953 ms TTFT | 388.773 ms TTFT | Looped | Looped | Looped | Looped | Looped |
| 16K | 96.262 ms TTFT | 178.977 ms TTFT | 371.5122 ms TTFT | 714.7174 ms TTFT | 1,275.4300 ms TTFT | Looped | Looped | Looped |
| 32K | 89.6106 ms TTFT | 167.2135 ms TTFT | 346.1201 ms TTFT | 661.0320 ms TTFT | Looped | 1,899.2993 ms TTFT | Looped | Over capacity |
| 64K | 78.6197 ms TTFT | 148.3251 ms TTFT | Looped | Looped | Looped | Looped | Over capacity | Over capacity |
| 123K | 64.3425 ms TTFT | 122.6580 ms TTFT | 251.0825 ms TTFT | 460.91,437 ms TTFT | 665.02,683 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.1K | — | 303 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 24.7K | — | 657 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.3K | — | 1,677 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.2K | — | 4,881 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 8,376.0 | — | 14,800 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,107.0 | — | 15,910 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 101.0 | 101.3 | 39 ms | 9.9 ms | 1.0 | 0 | 3,028 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 186.9 | 93.8 | 53 ms | 10.7 ms | 2.0 | 0 | 5,604 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 388.7 | 97.6 | 73 ms | 10.2 ms | 4.0 | 0 | 11,662 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 96.2 | 96.3 | 62 ms | 10.4 ms | 1.0 | 0 | 2,887 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 178.9 | 89.6 | 77 ms | 11.2 ms | 2.0 | 0 | 5,368 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 371.5 | 93.0 | 122 ms | 10.8 ms | 4.0 | 0 | 11,146 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 714.7 | 89.5 | 174 ms | 11.2 ms | 8.0 | 0 | 21,442 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,275.4 | 80.1 | 300 ms | 12.5 ms | 16.0 | 0 | 38,260 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 89.6 | 89.7 | 106 ms | 11.1 ms | 1.0 | 0 | 2,689 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 167.2 | 83.7 | 135 ms | 11.9 ms | 2.0 | 0 | 5,017 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 346.1 | 86.6 | 201 ms | 11.5 ms | 4.0 | 0 | 10,382 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 661.0 | 82.7 | 320 ms | 12.1 ms | 8.0 | 0 | 19,827 | 30.0 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 1,899.2 | 60.1 | 993 ms | 16.6 ms | 32.0 | 0 | 57,596 | 30.3 s |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 78.6 | 78.7 | 197 ms | 12.7 ms | 1.0 | 0 | 2,357 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 148.3 | 74.2 | 251 ms | 13.5 ms | 2.0 | 0 | 4,448 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 64.3 | 64.4 | 425 ms | 15.5 ms | 1.0 | 0 | 1,927 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 122.6 | 61.4 | 580 ms | 16.3 ms | 2.0 | 0 | 3,678 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 251.0 | 62.7 | 825 ms | 15.9 ms | 4.0 | 0 | 7,531 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 460.9 | 57.5 | 1,437 ms | 17.4 ms | 8.0 | 0 | 13,828 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 665.0 | 41.7 | 2,683 ms | 24.0 ms | 16.0 | 0 | 20,032 | 30.1 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.