Speed test matrix
gemma-4-12B-it-FP8-Dynamic on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/gemma-4-12B-it-FP8-Dynamic --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --async-scheduling
Prefill
prompt tok/s by context length8K context19K432 ms TTFT
16K context17.7K916 ms TTFT
32K context14.7K2,209 ms TTFT
64K context10.7K6,026 ms TTFT
128K context6,93518,600 ms TTFT
251K context4,12361,314 ms TTFT
256K context4,05763,557 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 80.445 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 16K | 75.863 ms TTFT | 151.4104 ms TTFT | 295.3120 ms TTFT | Looped | Looped | Looped | Looped | Looped |
| 32K | 71.3106 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped | Over capacity |
| 64K | 66.8197 ms TTFT | 133.7275 ms TTFT | Looped | Looped | Looped | 714.81,990 ms TTFT | Over capacity | Over capacity |
| 128K | Looped | Looped | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
| 251K | 48.71,052 ms TTFT | 97.51,431 ms TTFT | 191.72,177 ms TTFT | 335.53,646 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19K | — | 432 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 17.7K | — | 916 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.7K | — | 2,209 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.7K | — | 6,026 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,935.0 | — | 18,600 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 4,123.0 | — | 61,314 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 4,057.0 | — | 63,557 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 80.4 | 80.6 | 45 ms | 12.4 ms | 1.0 | 0 | 2,413 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 75.8 | 75.9 | 63 ms | 13.2 ms | 1.0 | 0 | 2,274 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 151.4 | 75.9 | 104 ms | 13.2 ms | 2.0 | 0 | 4,543 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 295.3 | 74.0 | 120 ms | 13.5 ms | 4.0 | 0 | 8,858 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 71.3 | 71.4 | 106 ms | 14.0 ms | 1.0 | 0 | 2,140 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 66.8 | 66.9 | 197 ms | 15.0 ms | 1.0 | 0 | 2,002 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 133.7 | 66.9 | 275 ms | 14.9 ms | 2.0 | 0 | 4,011 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Valid | 714.8 | 22.4 | 1,990 ms | 44.6 ms | 32.0 | 0 | 21,536 | 30.1 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 48.7 | 48.9 | 1,052 ms | 20.5 ms | 1.0 | 0 | 1,461 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 97.5 | 48.8 | 1,431 ms | 20.5 ms | 2.0 | 0 | 2,923 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 191.7 | 47.8 | 2,177 ms | 20.9 ms | 4.0 | 0 | 5,748 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 335.5 | 41.6 | 3,646 ms | 24.0 ms | 8.0 | 0 | 10,066 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.