Speed test matrix
Gemma-4-31B-IT-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- gemma
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-vllm-73a5831127a9/bin/vllm" serve nvidia/Gemma-4-31B-IT-NVFP4 --served-model-name gemma --trust-remote-code --tensor-parallel-size 1 --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --max-model-len 262144 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8 --attention-backend TRITON_ATTN --linear-backend cutlass
Prefill
prompt tok/s by context length8K context8,242995 ms TTFT
16K context7,5912,142 ms TTFT
32K context6,1755,242 ms TTFT
64K context4,44014,540 ms TTFT
128K context2,83845,448 ms TTFT
251K context1,666151,738 ms TTFT
256K context1,638157,390 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 32.363 ms TTFT | 65.587 ms TTFT | 129.7126 ms TTFT | 250.4173 ms TTFT | 430.7233 ms TTFT | 763.2322 ms TTFT | Over capacity | Over capacity |
| 16K | Looped | 63.4128 ms TTFT | 125.3172 ms TTFT | 241.4238 ms TTFT | 381.9355 ms TTFT | 670.1541 ms TTFT | Over capacity | Over capacity |
| 32K | 30.4131 ms TTFT | 60.8174 ms TTFT | 119.9244 ms TTFT | 227.9364 ms TTFT | 313.4606 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 29.3301 ms TTFT | 58.1331 ms TTFT | 114.4468 ms TTFT | 212.1679 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 27.4540 ms TTFT | 53.3681 ms TTFT | 104.7994 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 24.41,169 ms TTFT | 45.71,602 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,242.0 | — | 995 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,591.0 | — | 2,142 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,175.0 | — | 5,242 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,440.0 | — | 14,540 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,838.0 | — | 45,448 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 1,666.0 | — | 151,738 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 1,638.0 | — | 157,390 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 32.3 | 32.3 | 63 ms | 30.9 ms | 1.0 | 0 | 970 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 65.5 | 32.8 | 87 ms | 30.5 ms | 2.0 | 0 | 1,966 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 129.7 | 32.4 | 126 ms | 30.8 ms | 4.0 | 0 | 3,891 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 250.4 | 31.4 | 173 ms | 31.9 ms | 8.0 | 0 | 7,511 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 430.7 | 27.0 | 233 ms | 37.0 ms | 16.0 | 0 | 12,921 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 763.2 | 24.1 | 322 ms | 41.6 ms | 32.0 | 0 | 22,896 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 63.4 | 31.6 | 128 ms | 31.6 ms | 2.0 | 0 | 1,901 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 125.3 | 31.3 | 172 ms | 32.0 ms | 4.0 | 0 | 3,758 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 241.4 | 30.1 | 238 ms | 33.2 ms | 8.0 | 0 | 7,241 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 381.9 | 24.0 | 355 ms | 41.8 ms | 16.0 | 0 | 11,457 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 670.1 | 21.1 | 541 ms | 47.4 ms | 32.0 | 0 | 20,102 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 30.4 | 30.3 | 131 ms | 32.9 ms | 1.0 | 0 | 910 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 60.8 | 30.4 | 174 ms | 32.9 ms | 2.0 | 0 | 1,824 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 119.9 | 30.0 | 244 ms | 33.4 ms | 4.0 | 0 | 3,596 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 227.9 | 28.5 | 364 ms | 35.1 ms | 8.0 | 0 | 6,836 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 313.4 | 19.6 | 606 ms | 51.0 ms | 16.0 | 0 | 9,402 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 29.3 | 29.3 | 301 ms | 34.1 ms | 1.0 | 0 | 878 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 58.1 | 29.1 | 331 ms | 34.4 ms | 2.0 | 0 | 1,742 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 114.4 | 28.6 | 468 ms | 34.9 ms | 4.0 | 0 | 3,431 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 212.1 | 26.5 | 679 ms | 37.8 ms | 8.0 | 0 | 6,364 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 27.4 | 27.4 | 540 ms | 36.5 ms | 1.0 | 0 | 820 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 53.3 | 26.7 | 681 ms | 37.5 ms | 2.0 | 0 | 1,598 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 104.7 | 26.1 | 994 ms | 38.3 ms | 4.0 | 0 | 3,140 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 24.4 | 24.4 | 1,169 ms | 41.0 ms | 1.0 | 0 | 732 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 45.7 | 22.9 | 1,602 ms | 43.7 ms | 2.0 | 0 | 1,370 | 30.0 s |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.