Speed test matrix
Nemotron-Cascade-2-30B-A3B-AWQ-4bit on RTX 3090 24GB
vllm · 0.32.0 · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: fp8 # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve cyankiwi/Nemotron-Cascade-2-30B-A3B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 88 --enable-prefix-caching --trust-remote-code --kv-cache-dtype fp8
Prefill
prompt tok/s by context length8K context9,394895 ms TTFT
16K context9,2611,797 ms TTFT
32K context8,5793,858 ms TTFT
64K context7,3998,920 ms TTFT
128K context5,81322,681 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 151.433 ms TTFT | 269.254 ms TTFT | 439.073 ms TTFT | 684.6113 ms TTFT | 1,020.5158 ms TTFT | 953.4Requests queued243 ms TTFT | Over capacity | Over capacity |
| 16K | 149.3486 ms TTFT | 264.9812 ms TTFT | 430.21,381 ms TTFT | 667.92,292 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 147.8540 ms TTFT | 262.4784 ms TTFT | 424.11,286 ms TTFT | 652.72,289 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 145.2652 ms TTFT | 258.9947 ms TTFT | 412.71,518 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 140.5750 ms TTFT | 249.61,043 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 132.2641 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,394.0 | — | 895 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,261.0 | — | 1,797 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,579.0 | — | 3,858 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,399.0 | — | 8,920 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,813.0 | — | 22,681 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 151.4 | 151.5 | 33 ms | 6.6 ms | 1.0 | 0 | 4,543 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 269.2 | 134.7 | 54 ms | 7.4 ms | 2.0 | 0 | 8,074 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 439.0 | 110.0 | 73 ms | 9.1 ms | 4.0 | 0 | 13,168 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 684.6 | 85.9 | 113 ms | 11.6 ms | 8.0 | 0 | 20,536 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,020.5 | 64.2 | 158 ms | 15.6 ms | 16.0 | 0 | 30,608 | 30.0 s |
| Sustained decode | 0 | 32 | Requests queued | 953.4 | 53.6 | 243 ms | 18.7 ms | 18.0 | 0 | 28,591 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 149.3 | 149.4 | 486 ms | 6.7 ms | 1.0 | 0 | 4,479 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 264.9 | 132.0 | 812 ms | 7.6 ms | 2.0 | 0 | 7,946 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 430.2 | 106.3 | 1,381 ms | 9.4 ms | 4.0 | 0 | 12,907 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 667.9 | 80.5 | 2,292 ms | 12.4 ms | 8.0 | 0 | 20,032 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 147.8 | 147.8 | 540 ms | 6.8 ms | 1.0 | 0 | 4,433 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 262.4 | 130.4 | 784 ms | 7.7 ms | 2.0 | 0 | 7,870 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 424.1 | 103.9 | 1,286 ms | 9.6 ms | 4.0 | 0 | 12,724 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 652.7 | 77.9 | 2,289 ms | 12.8 ms | 8.0 | 0 | 19,576 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 145.2 | 145.3 | 652 ms | 6.9 ms | 1.0 | 0 | 4,357 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 258.9 | 128.4 | 947 ms | 7.8 ms | 2.0 | 0 | 7,766 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 412.7 | 100.8 | 1,518 ms | 9.9 ms | 4.0 | 0 | 12,379 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 140.5 | 140.6 | 750 ms | 7.1 ms | 1.0 | 0 | 4,213 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 249.6 | 123.8 | 1,043 ms | 8.1 ms | 2.0 | 0 | 7,488 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 132.2 | 132.4 | 641 ms | 7.6 ms | 1.0 | 0 | 3,966 | 30.0 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.