Speed test matrix

Qwen3.5-4B-FP8-dynamic on RTX 3090 24GB

vllm · 0.32.0 · FP8 dynamic6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K

by @Sandbergllm-inference-bench v0.8.0Measured Oct 30, 2026, 07:40 UTC

Hardware

Type: Discrete GPU
GPU: RTX 3090
VRAM: 24 GB
CPU: Intel Core i7-9700K (8C)
RAM: 32 GB
OS: Ubuntu 24.04.5

Engine

Engine: vllm
Version: 0.32.0
Quantization: FP8 dynamic
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: vllm/vllm-openai:v0.32.0-x86_64

Run

Served model: bench
Revision: main
Benchmark tool: llm-inference-bench v0.8.0
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 8,192 tokens

Launch command

# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04
vllm serve RedHatAI/Qwen3.5-4B-FP8-dynamic --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}

Prefill

prompt tok/s by context length

8K context11.1K735 ms TTFT

16K context10.2K1,585 ms TTFT

32K context9,1293,537 ms TTFT

64K context7,4868,605 ms TTFT

128K context5,39723,844 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
0	152.440 ms TTFT	303.547 ms TTFT	569.183 ms TTFT	953.9108 ms TTFT	1,528.7185 ms TTFT	2,189.3236 ms TTFT	Over capacity	Over capacity
16K	108.5148 ms TTFT	213.6213 ms TTFT	414.3393 ms TTFT	568.4613 ms TTFT	727.81,080 ms TTFT	Over capacity	Over capacity	Over capacity
32K	83.3183 ms TTFT	157.7276 ms TTFT	310.1442 ms TTFT	398.6714 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
64K	53.3301 ms TTFT	108.8429 ms TTFT	203.7635 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
128K	33.0538 ms TTFT	65.2710 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
248K	18.4883 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	11.1K	—	735 ms	—	—	1	—	—
Prefill	16K	1	Valid	10.2K	—	1,585 ms	—	—	1	—	—
Prefill	32K	1	Valid	9,129.0	—	3,537 ms	—	—	1	—	—
Prefill	64K	1	Valid	7,486.0	—	8,605 ms	—	—	1	—	—
Prefill	128K	1	Valid	5,397.0	—	23,844 ms	—	—	1	—	—
Sustained decode	0	1	Valid	152.4	155.4	40 ms	6.4 ms	1.0	0	4,569	30.0 s
Sustained decode	0	2	Valid	303.5	153.7	47 ms	6.5 ms	2.0	0	9,103	30.0 s
Sustained decode	0	4	Valid	569.1	146.0	83 ms	6.8 ms	4.0	0	17,065	30.0 s
Sustained decode	0	8	Valid	953.9	121.7	108 ms	8.2 ms	8.0	0	28,619	30.0 s
Sustained decode	0	16	Valid	1,528.7	99.6	185 ms	10.0 ms	16.0	0	45,858	30.0 s
Sustained decode	0	32	Valid	2,189.3	70.9	236 ms	14.1 ms	32.0	0	65,628	30.0 s
Sustained decode	0	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	0	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	108.5	110.4	148 ms	9.1 ms	1.0	0	3,253	30.0 s
Sustained decode	16K	2	Valid	213.6	108.2	213 ms	9.2 ms	2.0	0	6,409	30.0 s
Sustained decode	16K	4	Valid	414.3	103.3	393 ms	9.7 ms	4.0	0	12,429	30.0 s
Sustained decode	16K	8	Valid	568.4	72.2	613 ms	13.9 ms	8.0	0	17,043	30.0 s
Sustained decode	16K	16	Valid	727.8	45.8	1,080 ms	21.9 ms	16.0	0	21,827	30.0 s
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	83.3	84.7	183 ms	11.8 ms	1.0	0	2,498	30.0 s
Sustained decode	32K	2	Valid	157.7	79.6	276 ms	12.6 ms	2.0	0	4,731	30.0 s
Sustained decode	32K	4	Valid	310.1	78.5	442 ms	12.7 ms	4.0	0	9,294	30.0 s
Sustained decode	32K	8	Valid	398.6	49.9	714 ms	20.1 ms	8.0	0	11,953	30.0 s
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	53.3	54.5	301 ms	18.4 ms	1.0	0	1,598	30.0 s
Sustained decode	64K	2	Valid	108.8	55.1	429 ms	18.1 ms	2.0	0	3,260	30.0 s
Sustained decode	64K	4	Valid	203.7	51.7	635 ms	19.3 ms	4.0	0	6,110	30.0 s
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	33.0	34.0	538 ms	29.4 ms	1.0	0	991	30.0 s
Sustained decode	128K	2	Valid	65.2	33.2	710 ms	30.1 ms	2.0	0	1,952	30.0 s
Sustained decode	128K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	1	Valid	18.4	19.1	883 ms	52.3 ms	1.0	0	551	30.0 s
Sustained decode	248K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	248K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.