Speed test matrix

NVIDIA-Nemotron-Nano-9B-v2-quantized.w4a16 on RTX 3090 24GB

vllm · 0.32.0 · W4A165 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 29, 2026, 08:46 UTCEdited Oct 29, 2026, 12:41 UTC

Hardware

Type: Discrete GPU
GPU: RTX 3090
VRAM: 24 GB
CPU: Intel Core i7-9700K (8C)
RAM: 32 GB
OS: Ubuntu 24.04.5

Engine

Engine: vllm
Version: 0.32.0
Quantization: W4A16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: vllm/vllm-openai:v0.32.0-x86_64

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none
# 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04
vllm serve RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-quantized.w4a16 --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code

Prefill

prompt tok/s by context length

8K context5,6461,485 ms TTFT

16K context5,4563,044 ms TTFT

32K context5,2726,273 ms TTFT

64K context4,90713,446 ms TTFT

120K context4,38428,188 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	132.133 ms TTFT	249.847 ms TTFT	455.0307 ms TTFT	770.7148 ms TTFT	1,173.9281 ms TTFT	1,564.8437 ms TTFT	1,780.0853 ms TTFT	Over capacity
16K	126.3208 ms TTFT	233.2309 ms TTFT	418.3510 ms TTFT	659.1909 ms TTFT	936.91,655 ms TTFT	Over capacity	Over capacity	Over capacity
32K	122.1150 ms TTFT	218.4175 ms TTFT	392.5256 ms TTFT	589.2430 ms TTFT	768.2803 ms TTFT	Over capacity	Over capacity	Over capacity
64K	113.2241 ms TTFT	195.8292 ms TTFT	345.9405 ms TTFT	482.2650 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
120K	101.4355 ms TTFT	163.8452 ms TTFT	284.7688 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	5,646.0	—	1,485 ms	—	—	1	—	—
Prefill	16K	1	Valid	5,456.0	—	3,044 ms	—	—	1	—	—
Prefill	32K	1	Valid	5,272.0	—	6,273 ms	—	—	1	—	—
Prefill	64K	1	Valid	4,907.0	—	13,446 ms	—	—	1	—	—
Prefill	120K	1	Valid	4,384.0	—	28,188 ms	—	—	1	—	—
Sustained decode	0	1	Valid	132.1	132.3	33 ms	7.6 ms	1.0	0	3,964	30.0 s
Sustained decode	0	2	Valid	249.8	125.2	47 ms	8.0 ms	2.0	0	7,494	30.0 s
Sustained decode	0	4	Valid	455.0	114.1	307 ms	8.8 ms	4.0	0	13,651	30.0 s
Sustained decode	0	8	Valid	770.7	96.7	148 ms	10.3 ms	8.0	0	23,120	30.0 s
Sustained decode	0	16	Valid	1,173.9	73.7	281 ms	13.6 ms	16.0	0	35,216	30.0 s
Sustained decode	0	32	Valid	1,564.8	49.0	437 ms	20.4 ms	32.0	0	46,944	30.0 s
Sustained decode	0	64	Valid	1,780.0	27.8	853 ms	36.0 ms	64.0	0	53,376	30.0 s
Sustained decode	0	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	126.3	126.4	208 ms	7.9 ms	1.0	0	3,790	30.0 s
Sustained decode	16K	2	Valid	233.2	116.4	309 ms	8.6 ms	2.0	0	6,994	30.0 s
Sustained decode	16K	4	Valid	418.3	104.0	510 ms	9.6 ms	4.0	0	12,548	30.0 s
Sustained decode	16K	8	Valid	659.1	81.1	909 ms	12.3 ms	8.0	0	19,768	30.0 s
Sustained decode	16K	16	Valid	936.9	56.3	1,655 ms	17.8 ms	16.0	0	28,096	30.0 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	122.1	122.3	150 ms	8.2 ms	1.0	0	3,663	30.0 s
Sustained decode	32K	2	Valid	218.4	109.2	175 ms	9.2 ms	2.0	0	6,550	30.0 s
Sustained decode	32K	4	Valid	392.5	98.0	256 ms	10.2 ms	4.0	0	11,776	30.0 s
Sustained decode	32K	8	Valid	589.2	73.5	430 ms	13.6 ms	8.0	0	17,672	30.0 s
Sustained decode	32K	16	Valid	768.2	47.9	803 ms	20.9 ms	16.0	0	23,040	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	113.2	113.3	241 ms	8.8 ms	1.0	0	3,396	30.0 s
Sustained decode	64K	2	Valid	195.8	97.9	292 ms	10.2 ms	2.0	0	5,874	30.0 s
Sustained decode	64K	4	Valid	345.9	86.2	405 ms	11.6 ms	4.0	0	10,376	30.0 s
Sustained decode	64K	8	Valid	482.2	59.4	650 ms	16.8 ms	8.0	0	14,464	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	1	Valid	101.4	101.6	355 ms	9.8 ms	1.0	0	3,043	30.0 s
Sustained decode	120K	2	Valid	163.8	82.0	452 ms	12.2 ms	2.0	0	4,914	30.0 s
Sustained decode	120K	4	Valid	284.7	71.0	688 ms	14.1 ms	4.0	0	8,540	30.0 s
Sustained decode	120K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	120K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.