Speed test matrix
granite-4.2-3b-nvfp4 on RTX 3090 24GB
vllm · 0.32.0 · NVFP45 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve ibm-granite/granite-4.2-3b-nvfp4 --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context4,7221,745 ms TTFT
16K context3,0855,288 ms TTFT
32K context1,76218,433 ms TTFT
64K context954.067,925 ms TTFT
120K context530.0229,076 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 150.421 ms TTFT | 289.216 ms TTFT | 505.720 ms TTFT | 862.632 ms TTFT | 1,359.656 ms TTFT | Looped | 3,409.488 ms TTFT | Over capacity |
| 16K | 95.868 ms TTFT | 164.7101 ms TTFT | 231.9151 ms TTFT | 293.6194 ms TTFT | 356.8286 ms TTFT | 489.2435 ms TTFT | Over capacity | Over capacity |
| 32K | 69.3119 ms TTFT | 113.4178 ms TTFT | 147.1246 ms TTFT | 171.2293 ms TTFT | 197.9470 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 44.4231 ms TTFT | 68.9311 ms TTFT | 83.8437 ms TTFT | 92.4643 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 120K | 27.1Requests queued447 ms TTFT | 40.6597 ms TTFT | 47.8920 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,722.0 | — | 1,745 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 3,085.0 | — | 5,288 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,762.0 | — | 18,433 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 954.0 | — | 67,925 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 530.0 | — | 229,076 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 150.4 | 153.3 | 21 ms | 6.5 ms | 1.0 | 0 | 4,513 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 289.2 | 148.0 | 16 ms | 6.8 ms | 2.0 | 0 | 8,674 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 505.7 | 131.4 | 20 ms | 7.6 ms | 4.0 | 0 | 15,168 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 862.6 | 113.9 | 32 ms | 8.8 ms | 8.0 | 0 | 25,872 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,359.6 | 91.7 | 56 ms | 10.9 ms | 16.0 | 0 | 40,784 | 30.0 s |
| Sustained decode | 0 | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 0 | 64 | Valid | 3,409.4 | 58.4 | 88 ms | 17.1 ms | 64.0 | 0 | 102,208 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 95.8 | 96.3 | 68 ms | 10.4 ms | 1.0 | 0 | 2,875 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 164.7 | 82.9 | 101 ms | 12.1 ms | 2.0 | 0 | 4,938 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 231.9 | 58.4 | 151 ms | 17.1 ms | 4.0 | 0 | 6,956 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 293.6 | 37.0 | 194 ms | 27.1 ms | 8.0 | 0 | 8,800 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 356.8 | 22.4 | 286 ms | 44.7 ms | 16.0 | 0 | 10,704 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 489.2 | 15.4 | 435 ms | 65.1 ms | 32.0 | 0 | 14,656 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 69.3 | 69.5 | 119 ms | 14.4 ms | 1.0 | 0 | 2,077 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 113.4 | 56.9 | 178 ms | 17.6 ms | 2.0 | 0 | 3,400 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 147.1 | 36.9 | 246 ms | 27.1 ms | 4.0 | 0 | 4,412 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 171.2 | 21.4 | 293 ms | 46.6 ms | 8.0 | 0 | 5,128 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 197.9 | 12.4 | 470 ms | 80.8 ms | 16.0 | 0 | 5,936 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 44.4 | 44.5 | 231 ms | 22.5 ms | 1.0 | 0 | 1,331 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 68.9 | 34.5 | 311 ms | 28.9 ms | 2.0 | 0 | 2,066 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 83.8 | 21.0 | 437 ms | 47.6 ms | 4.0 | 0 | 2,512 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 92.4 | 11.6 | 643 ms | 86.4 ms | 8.0 | 0 | 2,768 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Requests queued | 27.1 | 27.1 | 447 ms | 36.9 ms | 1.0 | 0 | 812 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 40.6 | 20.3 | 597 ms | 49.3 ms | 2.0 | 0 | 1,218 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 47.8 | 11.9 | 920 ms | 84.1 ms | 4.0 | 0 | 1,432 | 30.0 s |
| Sustained decode | 120K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.