Speed test matrix
gemma-4-26B-A4B-it-FP8-dynamic on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8
Prefill
prompt tok/s by context length8K context34.1K240 ms TTFT
16K context32.7K498 ms TTFT
32K context26.2K1,237 ms TTFT
64K context18.5K3,481 ms TTFT
128K context11.6K11,117 ms TTFT
251K context6,73437,541 ms TTFT
256K context6,62238,935 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 168.238 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 16K | 154.855 ms TTFT | 261.876 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped |
| 32K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Over capacity |
| 64K | 132.2183 ms TTFT | Looped | 384.8343 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity |
| 128K | 114.7419 ms TTFT | Looped | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
| 251K | 90.6981 ms TTFT | Looped | 262.72,098 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 34.1K | — | 240 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32.7K | — | 498 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 26.2K | — | 1,237 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 18.5K | — | 3,481 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.6K | — | 11,117 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 6,734.0 | — | 37,541 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,622.0 | — | 38,935 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 168.2 | 381.1 | 38 ms | 3.8 ms | 1.0 | 1 | 5,045 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 154.8 | 156.2 | 55 ms | 6.4 ms | 1.0 | 1 | 4,642 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 261.8 | 132.4 | 76 ms | 7.6 ms | 2.0 | 2 | 7,853 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 132.2 | 134.1 | 183 ms | 7.5 ms | 1.0 | 1 | 3,965 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Valid | 384.8 | 96.2 | 343 ms | 10.4 ms | 4.0 | 0 | 11,544 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 114.7 | 114.9 | 419 ms | 8.7 ms | 1.0 | 0 | 3,441 | 30.0 s |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 90.6 | 90.9 | 981 ms | 11.0 ms | 1.0 | 0 | 2,718 | 30.0 s |
| Sustained decode | 251K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 4 | Valid | 262.7 | 65.8 | 2,098 ms | 15.2 ms | 4.0 | 0 | 7,880 | 30.0 s |
| Sustained decode | 251K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.