Speed test matrix
NVIDIA-Nemotron-Nano-12B-v2-AWQ-4bit on RTX 3090 24GB
vllm · 0.32.0 · AWQ 4-bit5 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve cyankiwi/NVIDIA-Nemotron-Nano-12B-v2-AWQ-4bit --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context3,5872,338 ms TTFT
16K context3,1745,234 ms TTFT
32K context2,52513,099 ms TTFT
64K context1,82436,167 ms TTFT
120K context1,228100,611 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 94.238 ms TTFT | 180.563 ms TTFT | 333.4105 ms TTFT | 580.9183 ms TTFT | 916.3270 ms TTFT | 1,286.5560 ms TTFT | 1,469.41,137 ms TTFT | 1,214.7Requests queued1,024 ms TTFT |
| 16K | 86.2753 ms TTFT | 154.81,130 ms TTFT | 268.41,887 ms TTFT | 427.53,384 ms TTFT | 603.16,413 ms TTFT | 766.012,548 ms TTFT | Over capacity | Over capacity |
| 32K | 79.82,000 ms TTFT | 135.72,970 ms TTFT | 224.84,933 ms TTFT | 338.68,816 ms TTFT | 451.416,760 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 69.32,045 ms TTFT | 109.93,017 ms TTFT | 170.64,964 ms TTFT | 240.38,954 ms TTFT | 300.516,777 ms TTFT | Over capacity | Over capacity | Over capacity |
| 120K | 56.71,768 ms TTFT | 82.22,603 ms TTFT | 120.94,267 ms TTFT | 160.27,670 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,587.0 | — | 2,338 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 3,174.0 | — | 5,234 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 2,525.0 | — | 13,099 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,824.0 | — | 36,167 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 1,228.0 | — | 100,611 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 94.2 | 94.3 | 38 ms | 10.6 ms | 1.0 | 0 | 2,824 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 180.5 | 90.4 | 63 ms | 11.1 ms | 2.0 | 0 | 5,414 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 333.4 | 83.6 | 105 ms | 12.0 ms | 4.0 | 0 | 10,000 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 580.9 | 72.9 | 183 ms | 13.7 ms | 8.0 | 0 | 17,424 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 916.3 | 57.5 | 270 ms | 17.4 ms | 16.0 | 0 | 27,491 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,286.5 | 40.3 | 560 ms | 24.8 ms | 32.0 | 0 | 38,592 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,469.4 | 22.9 | 1,137 ms | 43.7 ms | 64.0 | 0 | 44,032 | 30.0 s |
| Sustained decode | 0 | 128 | Requests queued | 1,214.7 | 18.4 | 1,024 ms | 54.3 ms | 67.0 | 0 | 36,381 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 86.2 | 86.2 | 753 ms | 11.6 ms | 1.0 | 0 | 2,584 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 154.8 | 76.7 | 1,130 ms | 13.0 ms | 2.0 | 0 | 4,642 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 268.4 | 65.1 | 1,887 ms | 15.4 ms | 4.0 | 0 | 8,052 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 427.5 | 49.8 | 3,384 ms | 20.1 ms | 8.0 | 0 | 12,824 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 603.1 | 32.6 | 6,413 ms | 30.7 ms | 16.0 | 0 | 18,080 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 766.0 | 18.2 | 12,548 ms | 54.8 ms | 32.0 | 0 | 22,976 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 79.8 | 79.8 | 2,000 ms | 12.5 ms | 1.0 | 0 | 2,392 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 135.7 | 66.1 | 2,970 ms | 15.1 ms | 2.0 | 0 | 4,070 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 224.8 | 51.8 | 4,933 ms | 19.3 ms | 4.0 | 0 | 6,744 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 338.6 | 35.4 | 8,816 ms | 28.3 ms | 8.0 | 0 | 10,152 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 451.4 | 19.8 | 16,760 ms | 50.6 ms | 16.0 | 0 | 13,536 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 69.3 | 69.4 | 2,045 ms | 14.4 ms | 1.0 | 0 | 2,080 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 109.9 | 53.5 | 3,017 ms | 18.7 ms | 2.0 | 0 | 3,294 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 170.6 | 39.4 | 4,964 ms | 25.4 ms | 4.0 | 0 | 5,116 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 240.3 | 25.1 | 8,954 ms | 39.9 ms | 8.0 | 0 | 7,208 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 300.5 | 13.3 | 16,777 ms | 75.0 ms | 16.0 | 0 | 9,008 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 56.7 | 56.8 | 1,768 ms | 17.6 ms | 1.0 | 0 | 1,701 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 82.2 | 40.2 | 2,603 ms | 24.9 ms | 2.0 | 0 | 2,466 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 120.9 | 28.3 | 4,267 ms | 35.4 ms | 4.0 | 0 | 3,624 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 160.2 | 17.2 | 7,670 ms | 58.1 ms | 8.0 | 0 | 4,800 | 30.0 s |
| Sustained decode | 120K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.