Speed test matrix
NVIDIA-Nemotron-Nano-9B-v2-quantized.w4a16 on RTX 3090 24GB
vllm · 0.32.0 · W4A165 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- W4A16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-quantized.w4a16 --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context4,5271,853 ms TTFT
16K context4,0644,088 ms TTFT
32K context3,3449,891 ms TTFT
64K context2,47726,637 ms TTFT
120K context1,72471,666 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 126.430 ms TTFT | 239.8188 ms TTFT | 434.386 ms TTFT | 734.8151 ms TTFT | 1,115.1222 ms TTFT | 1,475.3337 ms TTFT | 1,679.8944 ms TTFT | 1,567.7Requests queued1,060 ms TTFT |
| 16K | 116.8572 ms TTFT | 209.1859 ms TTFT | 359.11,427 ms TTFT | 565.52,571 ms TTFT | 786.44,867 ms TTFT | 970.19,515 ms TTFT | 1,055.319,148 ms TTFT | Over capacity |
| 32K | 108.81,461 ms TTFT | 185.82,171 ms TTFT | 305.93,623 ms TTFT | 459.36,474 ms TTFT | 606.912,227 ms TTFT | 723.423,833 ms TTFT | Over capacity | Over capacity |
| 64K | 95.91,460 ms TTFT | 152.82,166 ms TTFT | 237.43,527 ms TTFT | 335.36,328 ms TTFT | 418.711,969 ms TTFT | Over capacity | Over capacity | Over capacity |
| 120K | 79.51,317 ms TTFT | 116.41,903 ms TTFT | 172.13,036 ms TTFT | 229.05,407 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,527.0 | — | 1,853 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,064.0 | — | 4,088 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,344.0 | — | 9,891 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 2,477.0 | — | 26,637 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 1,724.0 | — | 71,666 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 126.4 | 126.7 | 30 ms | 7.9 ms | 1.0 | 0 | 3,791 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 239.8 | 120.2 | 188 ms | 8.3 ms | 2.0 | 0 | 7,192 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 434.3 | 109.0 | 86 ms | 9.2 ms | 4.0 | 0 | 13,027 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 734.8 | 92.3 | 151 ms | 10.8 ms | 8.0 | 0 | 22,040 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,115.1 | 70.0 | 222 ms | 14.3 ms | 16.0 | 0 | 33,440 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,475.3 | 46.0 | 337 ms | 21.7 ms | 32.0 | 0 | 44,256 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,679.8 | 26.2 | 944 ms | 38.2 ms | 64.0 | 0 | 50,368 | 30.0 s |
| Sustained decode | 0 | 128 | Requests queued | 1,567.7 | 17.6 | 1,060 ms | 56.8 ms | 90.0 | 0 | 46,980 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 116.8 | 116.9 | 572 ms | 8.6 ms | 1.0 | 0 | 3,502 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 209.1 | 103.9 | 859 ms | 9.6 ms | 2.0 | 0 | 6,272 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 359.1 | 87.8 | 1,427 ms | 11.4 ms | 4.0 | 0 | 10,772 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 565.5 | 67.1 | 2,571 ms | 14.9 ms | 8.0 | 0 | 16,960 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 786.4 | 43.9 | 4,867 ms | 22.8 ms | 16.0 | 0 | 23,584 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 970.1 | 24.5 | 9,515 ms | 40.8 ms | 32.0 | 0 | 29,088 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,055.3 | 11.5 | 19,148 ms | 87.2 ms | 64.0 | 0 | 31,616 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 108.8 | 108.9 | 1,461 ms | 9.2 ms | 1.0 | 0 | 3,265 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 185.8 | 91.2 | 2,171 ms | 11.0 ms | 2.0 | 0 | 5,574 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 305.9 | 72.1 | 3,623 ms | 13.9 ms | 4.0 | 0 | 9,177 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 459.3 | 50.0 | 6,474 ms | 20.0 ms | 8.0 | 0 | 13,776 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 606.9 | 29.0 | 12,227 ms | 34.5 ms | 16.0 | 0 | 18,208 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 723.4 | 13.9 | 23,833 ms | 71.8 ms | 32.0 | 0 | 21,696 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 95.9 | 96.0 | 1,460 ms | 10.4 ms | 1.0 | 0 | 2,878 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 152.8 | 75.0 | 2,166 ms | 13.3 ms | 2.0 | 0 | 4,584 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 237.4 | 56.1 | 3,527 ms | 17.8 ms | 4.0 | 0 | 7,120 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 335.3 | 36.7 | 6,328 ms | 27.3 ms | 8.0 | 0 | 10,056 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 418.7 | 20.3 | 11,969 ms | 49.3 ms | 16.0 | 0 | 12,560 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 79.5 | 79.6 | 1,317 ms | 12.6 ms | 1.0 | 0 | 2,384 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 116.4 | 57.3 | 1,903 ms | 17.5 ms | 2.0 | 0 | 3,492 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 172.1 | 41.1 | 3,036 ms | 24.3 ms | 4.0 | 0 | 5,164 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 229.0 | 25.6 | 5,407 ms | 39.0 ms | 8.0 | 0 | 6,864 | 30.0 s |
| Sustained decode | 120K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.