Speed test matrix
Nemotron-Cascade-2-30B-A3B-AWQ-4bit on RTX 3090 24GB
vllm · 0.32.0 · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- Intel Core i7-9700K (8C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04 vllm serve cyankiwi/Nemotron-Cascade-2-30B-A3B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 104 --enable-prefix-caching --trust-remote-code --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context8,943941 ms TTFT
16K context7,4082,246 ms TTFT
32K context5,7745,733 ms TTFT
64K context3,99316,529 ms TTFT
128K context2,43754,085 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 145.735 ms TTFT | 259.343 ms TTFT | 425.8105 ms TTFT | 672.5115 ms TTFT | 1,001.4210 ms TTFT | 1,095.8Requests queued233 ms TTFT | 1,086.4Requests queued315 ms TTFT | Over capacity |
| 16K | Looped | 232.8329 ms TTFT | 405.3540 ms TTFT | 661.9980 ms TTFT | 925.01,749 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 120.9494 ms TTFT | 215.0736 ms TTFT | 353.51,146 ms TTFT | 555.72,013 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 103.61,439 ms TTFT | 184.12,081 ms TTFT | 304.83,421 ms TTFT | 486.66,008 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 80.24,650 ms TTFT | 142.56,867 ms TTFT | 238.611,263 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 46.2Requests queued5,871 ms TTFT | 65.5Requests queued8,408 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,943.0 | — | 941 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,408.0 | — | 2,246 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,774.0 | — | 5,733 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,993.0 | — | 16,529 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,437.0 | — | 54,085 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 145.7 | 146.1 | 35 ms | 6.8 ms | 1.0 | 0 | 4,371 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 259.3 | 130.1 | 43 ms | 7.7 ms | 2.0 | 0 | 7,779 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 425.8 | 106.7 | 105 ms | 9.4 ms | 4.0 | 0 | 12,772 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 672.5 | 84.4 | 115 ms | 11.8 ms | 8.0 | 0 | 20,168 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,001.4 | 63.1 | 210 ms | 15.8 ms | 16.0 | 0 | 30,032 | 30.0 s |
| Sustained decode | 0 | 32 | Requests queued | 1,095.8 | 53.1 | 233 ms | 18.8 ms | 21.0 | 0 | 32,865 | 30.0 s |
| Sustained decode | 0 | 64 | Requests queued | 1,086.4 | 53.0 | 315 ms | 18.9 ms | 21.0 | 0 | 32,593 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 232.8 | 116.3 | 329 ms | 8.6 ms | 2.0 | 0 | 6,984 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 405.3 | 100.7 | 540 ms | 9.9 ms | 4.0 | 0 | 12,159 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 661.9 | 81.3 | 980 ms | 12.3 ms | 8.0 | 0 | 19,857 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 925.0 | 55.7 | 1,749 ms | 18.0 ms | 16.0 | 0 | 27,760 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 120.9 | 121.0 | 494 ms | 8.3 ms | 1.0 | 0 | 3,626 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 215.0 | 107.1 | 736 ms | 9.3 ms | 2.0 | 0 | 6,450 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 353.5 | 87.0 | 1,146 ms | 11.5 ms | 4.0 | 0 | 10,604 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 555.7 | 66.7 | 2,013 ms | 15.0 ms | 8.0 | 0 | 16,664 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 103.6 | 103.8 | 1,439 ms | 9.6 ms | 1.0 | 0 | 3,107 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 184.1 | 90.5 | 2,081 ms | 11.1 ms | 2.0 | 0 | 5,522 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 304.8 | 72.3 | 3,421 ms | 13.8 ms | 4.0 | 0 | 9,144 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 486.6 | 53.8 | 6,008 ms | 18.6 ms | 8.0 | 0 | 14,592 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 80.2 | 80.4 | 4,650 ms | 12.4 ms | 1.0 | 0 | 2,406 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 142.5 | 67.3 | 6,867 ms | 14.9 ms | 2.0 | 0 | 4,274 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 238.6 | 50.3 | 11,263 ms | 19.9 ms | 4.0 | 0 | 7,156 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 46.2 | 56.2 | 5,871 ms | 17.8 ms | 0.9 | 0 | 1,386 | 30.0 s |
| Sustained decode | 248K | 2 | Requests queued | 65.5 | 45.0 | 8,408 ms | 22.5 ms | 1.7 | 0 | 1,964 | 30.0 s |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.