Speed test matrix

NVIDIA-Nemotron-3-Nano-4B-FP8 on RTX 3060 12GB

vllm · 0.32.0 · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 31, 2026, 02:20 UTC

Hardware

Type: Discrete GPU
GPU: RTX 3060
VRAM: 12 GB
CPU: Intel Xeon E5-2680 v4 (14C)
RAM: 32 GB
OS: Ubuntu 24.04.5

Engine

Engine: vllm
Version: 0.32.0
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: vllm/vllm-openai:v0.32.0-x86_64

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: none
# 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04
vllm serve nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code  --max-cudagraph-capture-size 16

Prefill

prompt tok/s by context length

8K context3,0102,775 ms TTFT

16K context2,8485,798 ms TTFT

32K context2,64712,417 ms TTFT

64K context2,31628,327 ms TTFT

128K context1,84371,106 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	67.8117 ms TTFT	127.6140 ms TTFT	237.4643 ms TTFT	416.6343 ms TTFT	672.5589 ms TTFT	952.3906 ms TTFT	857.8Requests queued992 ms TTFT	Over capacity
16K	66.1479 ms TTFT	122.6666 ms TTFT	212.91,053 ms TTFT	360.31,811 ms TTFT	555.83,366 ms TTFT	Over capacity	Over capacity	Over capacity
32K	64.4860 ms TTFT	117.81,263 ms TTFT	193.12,004 ms TTFT	316.53,474 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
64K	61.2975 ms TTFT	109.21,334 ms TTFT	162.32,045 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
128K	55.8743 ms TTFT	94.9976 ms TTFT	123.31,569 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
248K	43.8Requests queued3,313 ms TTFT	63.9Requests queued4,610 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	3,010.0	—	2,775 ms	—	—	1	—	—
Prefill	16K	1	Valid	2,848.0	—	5,798 ms	—	—	1	—	—
Prefill	32K	1	Valid	2,647.0	—	12,417 ms	—	—	1	—	—
Prefill	64K	1	Valid	2,316.0	—	28,327 ms	—	—	1	—	—
Prefill	128K	1	Valid	1,843.0	—	71,106 ms	—	—	1	—	—
Sustained decode	0	1	Valid	67.8	67.9	117 ms	14.7 ms	1.0	0	2,034	30.0 s
Sustained decode	0	2	Valid	127.6	63.9	140 ms	15.7 ms	2.0	0	3,828	30.0 s
Sustained decode	0	4	Valid	237.4	59.4	643 ms	16.8 ms	4.0	0	7,120	30.0 s
Sustained decode	0	8	Valid	416.6	52.2	343 ms	19.2 ms	8.0	0	12,496	30.0 s
Sustained decode	0	16	Valid	672.5	42.1	589 ms	23.7 ms	16.0	0	20,175	30.0 s
Sustained decode	0	32	Valid	952.3	29.6	906 ms	33.8 ms	32.0	0	28,573	30.0 s
Sustained decode	0	64	Requests queued	857.8	21.1	992 ms	47.3 ms	42.7	0	25,302	29.5 s
Sustained decode	0	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	66.1	66.1	479 ms	15.1 ms	1.0	0	1,981	30.0 s
Sustained decode	16K	2	Valid	122.6	61.1	666 ms	16.4 ms	2.0	0	3,677	30.0 s
Sustained decode	16K	4	Valid	212.9	52.6	1,053 ms	19.0 ms	4.0	0	6,384	30.0 s
Sustained decode	16K	8	Valid	360.3	43.6	1,811 ms	22.9 ms	8.0	0	10,809	30.0 s
Sustained decode	16K	16	Valid	555.8	32.4	3,366 ms	30.9 ms	16.0	0	16,672	30.0 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	64.4	64.5	860 ms	15.5 ms	1.0	0	1,931	30.0 s
Sustained decode	32K	2	Valid	117.8	58.4	1,263 ms	17.1 ms	2.0	0	3,534	30.0 s
Sustained decode	32K	4	Valid	193.1	47.0	2,004 ms	21.3 ms	4.0	0	5,792	30.0 s
Sustained decode	32K	8	Valid	316.5	37.0	3,474 ms	27.0 ms	8.0	0	9,488	30.0 s
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	61.2	61.4	975 ms	16.3 ms	1.0	0	1,837	30.0 s
Sustained decode	64K	2	Valid	109.2	54.2	1,334 ms	18.4 ms	2.0	0	3,274	30.0 s
Sustained decode	64K	4	Valid	162.3	39.6	2,045 ms	25.3 ms	4.0	0	4,868	30.0 s
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	55.8	56.0	743 ms	17.8 ms	1.0	0	1,674	30.0 s
Sustained decode	128K	2	Valid	94.9	47.5	976 ms	21.0 ms	2.0	0	2,846	30.0 s
Sustained decode	128K	4	Valid	123.3	30.8	1,569 ms	32.5 ms	4.0	0	3,696	30.0 s
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Requests queued	43.8	48.3	3,313 ms	20.7 ms	0.9	0	1,315	30.0 s
Sustained decode	248K	2	Requests queued	63.9	37.1	4,610 ms	27.0 ms	1.8	0	1,916	30.0 s
Sustained decode	248K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.