Speed test matrix
gemma-4-31B-it-FP8-dynamic on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/gemma-4-31B-it-FP8-dynamic --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8
Prefill
prompt tok/s by context length8K context7,8221,048 ms TTFT
16K context7,1772,265 ms TTFT
32K context5,9005,486 ms TTFT
64K context4,31514,963 ms TTFT
128K context2,79046,229 ms TTFT
251K context1,650153,234 ms TTFT
256K context1,623158,889 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 37.258 ms TTFT | 73.476 ms TTFT | 144.6128 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity |
| 16K | 35.980 ms TTFT | 70.7116 ms TTFT | Looped | 268.2218 ms TTFT | 413.4329 ms TTFT | 710.3590 ms TTFT | Over capacity | Over capacity |
| 32K | 34.5125 ms TTFT | 67.6167 ms TTFT | 132.9246 ms TTFT | Looped | 333.4562 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 33.1221 ms TTFT | 64.2302 ms TTFT | Looped | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 30.7475 ms TTFT | 58.4632 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 26.91,096 ms TTFT | 49.31,493 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,822.0 | — | 1,048 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,177.0 | — | 2,265 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,900.0 | — | 5,486 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,315.0 | — | 14,963 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,790.0 | — | 46,229 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,650.0 | — | 153,234 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,623.0 | — | 158,889 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 37.2 | 37.1 | 58 ms | 27.0 ms | 1.0 | 0 | 1,112 | 29.9 s |
| Sustained decode | 8K | 2 | Valid | 73.4 | 36.7 | 76 ms | 27.3 ms | 2.0 | 0 | 2,200 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 144.6 | 36.2 | 128 ms | 27.6 ms | 4.0 | 0 | 4,340 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 35.9 | 35.8 | 80 ms | 27.9 ms | 1.0 | 0 | 1,076 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 70.7 | 35.3 | 116 ms | 28.3 ms | 2.0 | 0 | 2,120 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 268.2 | 33.5 | 218 ms | 29.9 ms | 8.0 | 0 | 8,046 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 413.4 | 25.9 | 329 ms | 38.6 ms | 16.0 | 0 | 12,402 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 710.3 | 22.4 | 590 ms | 44.7 ms | 32.0 | 0 | 21,308 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 34.5 | 34.5 | 125 ms | 29.0 ms | 1.0 | 0 | 1,033 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 67.6 | 33.8 | 167 ms | 29.6 ms | 2.0 | 0 | 2,022 | 29.9 s |
| Sustained decode | 32K | 4 | Valid | 132.9 | 33.2 | 246 ms | 30.1 ms | 4.0 | 0 | 3,987 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 333.4 | 20.9 | 562 ms | 47.9 ms | 16.0 | 0 | 10,001 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 33.1 | 33.1 | 221 ms | 30.2 ms | 1.0 | 0 | 992 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 64.2 | 32.1 | 302 ms | 31.1 ms | 2.0 | 0 | 1,925 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 30.7 | 30.7 | 475 ms | 32.6 ms | 1.0 | 0 | 920 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 58.4 | 29.2 | 632 ms | 34.2 ms | 2.0 | 0 | 1,750 | 30.0 s |
| Sustained decode | 128K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 26.9 | 26.9 | 1,096 ms | 37.1 ms | 1.0 | 0 | 805 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 49.3 | 24.7 | 1,493 ms | 40.5 ms | 2.0 | 0 | 1,480 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.