Speed test matrix
Qwen3.5-4B-FP8-dynamic on RTX 3060 12GB
vllm · 0.32.0 · FP8 dynamic6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3060
- VRAM
- 12 GB
- CPU
- Intel Xeon E5-2680 v4 (14C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- FP8 dynamic
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead. KV cache: int4_per_token_head
# 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04
vllm serve RedHatAI/Qwen3.5-4B-FP8-dynamic --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2} --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTNPrefill
prompt tok/s by context length8K context2,1863,727 ms TTFT
16K context1,7799,064 ms TTFT
32K context1,32524,215 ms TTFT
64K context875.073,169 ms TTFT
128K context523.0244,675 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 63.596 ms TTFT | 125.0181 ms TTFT | 232.5247 ms TTFT | 428.8401 ms TTFT | 536.8Requests queued602 ms TTFT | 545.9Requests queued653 ms TTFT | Over capacity | Over capacity |
| 16K | 44.92,662 ms TTFT | 86.04,875 ms TTFT | 165.87,472 ms TTFT | 288.913,298 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 31.63,592 ms TTFT | 65.06,652 ms TTFT | 125.411,367 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 21.74,344 ms TTFT | 43.38,312 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 8.3Requests queued13,783 ms TTFT | 2.3Requests queued22,260 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,186.0 | — | 3,727 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 1,779.0 | — | 9,064 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,325.0 | — | 24,215 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 875.0 | — | 73,169 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 523.0 | — | 244,675 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 63.5 | 65.0 | 96 ms | 15.4 ms | 1.0 | 0 | 1,903 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 125.0 | 63.2 | 181 ms | 15.8 ms | 2.0 | 0 | 3,750 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 232.5 | 59.6 | 247 ms | 16.8 ms | 4.0 | 0 | 6,973 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 428.8 | 54.6 | 401 ms | 18.3 ms | 8.0 | 0 | 12,853 | 30.0 s |
| Sustained decode | 0 | 16 | Requests queued | 536.8 | 40.7 | 602 ms | 24.6 ms | 13.0 | 0 | 16,106 | 30.0 s |
| Sustained decode | 0 | 32 | Requests queued | 545.9 | 39.6 | 653 ms | 25.2 ms | 13.0 | 0 | 16,374 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 44.9 | 45.6 | 2,662 ms | 21.9 ms | 1.0 | 0 | 1,348 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 86.0 | 43.5 | 4,875 ms | 23.0 ms | 2.0 | 0 | 2,580 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 165.8 | 39.3 | 7,472 ms | 25.5 ms | 4.0 | 0 | 4,975 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 288.9 | 30.2 | 13,298 ms | 33.2 ms | 8.0 | 0 | 8,662 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 31.6 | 32.3 | 3,592 ms | 30.9 ms | 1.0 | 0 | 947 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 65.0 | 32.8 | 6,652 ms | 30.5 ms | 2.0 | 0 | 1,948 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 125.4 | 29.9 | 11,367 ms | 33.5 ms | 4.0 | 0 | 3,760 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 21.7 | 22.5 | 4,344 ms | 44.4 ms | 1.0 | 0 | 651 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 43.3 | 22.2 | 8,312 ms | 45.1 ms | 2.0 | 0 | 1,297 | 30.0 s |
| Sustained decode | 64K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Requests queued | 8.3 | 14.9 | 13,783 ms | 67.1 ms | 0.8 | 0 | 248 | 29.9 s |
| Sustained decode | 128K | 2 | Requests queued | 2.3 | 7.9 | 22,260 ms | 224.0 ms | 1.5 | 0 | 70 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Over capacity | — | — | — | — | 1.0 | 0 | 0 | 30.0 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.