Speed test matrix
Qwen3.5-9B-AWQ-4bit on RTX 3090 24GB
vllm · 0.32.0 · W4A16 g326 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- Intel Core i7-9700K (8C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- W4A16 g32
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04 vllm serve cyankiwi/Qwen3.5-9B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context6,1111,340 ms TTFT
16K context5,2993,061 ms TTFT
32K context4,1867,715 ms TTFT
64K context2,95421,804 ms TTFT
128K context1,86069,190 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 112.039 ms TTFT | 209.344 ms TTFT | 404.583 ms TTFT | 774.8131 ms TTFT | 1,325.6284 ms TTFT | 2,089.1386 ms TTFT | 2,795.8747 ms TTFT | 3,300.3975 ms TTFT |
| 16K | 97.9487 ms TTFT | 180.2714 ms TTFT | 308.71,168 ms TTFT | 531.32,082 ms TTFT | 796.93,934 ms TTFT | 1,049.87,673 ms TTFT | 1,201.415,380 ms TTFT | Over capacity |
| 32K | 86.8614 ms TTFT | 158.2891 ms TTFT | 247.21,439 ms TTFT | 401.32,542 ms TTFT | 564.84,757 ms TTFT | 698.99,293 ms TTFT | Over capacity | Over capacity |
| 64K | 70.8688 ms TTFT | 126.9941 ms TTFT | 179.61,484 ms TTFT | 271.02,551 ms TTFT | 358.24,740 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 51.51,933 ms TTFT | 90.92,744 ms TTFT | 115.84,437 ms TTFT | 163.37,774 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 31.3Requests queued3,006 ms TTFT | 59.44,216 ms TTFT | 69.36,668 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,111.0 | — | 1,340 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,299.0 | — | 3,061 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,186.0 | — | 7,715 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 2,954.0 | — | 21,804 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,860.0 | — | 69,190 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 112.0 | 112.4 | 39 ms | 8.9 ms | 1.0 | 0 | 3,360 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 209.3 | 105.0 | 44 ms | 9.5 ms | 2.0 | 0 | 6,278 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 404.5 | 101.6 | 83 ms | 9.8 ms | 4.0 | 0 | 12,135 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 774.8 | 97.6 | 131 ms | 10.2 ms | 8.0 | 0 | 23,240 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,325.6 | 83.8 | 284 ms | 11.9 ms | 16.0 | 0 | 39,774 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,089.1 | 66.1 | 386 ms | 15.1 ms | 32.0 | 0 | 62,656 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,795.8 | 44.1 | 747 ms | 22.7 ms | 64.0 | 0 | 83,886 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,300.3 | 25.5 | 975 ms | 39.2 ms | 128.0 | 0 | 99,027 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 97.9 | 98.1 | 487 ms | 10.2 ms | 1.0 | 0 | 2,938 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 180.2 | 89.7 | 714 ms | 11.2 ms | 2.0 | 0 | 5,406 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 308.7 | 75.9 | 1,168 ms | 13.2 ms | 4.0 | 0 | 9,260 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 531.3 | 63.8 | 2,082 ms | 15.7 ms | 8.0 | 0 | 15,936 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 796.9 | 45.6 | 3,934 ms | 21.9 ms | 16.0 | 0 | 23,904 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,049.8 | 27.7 | 7,673 ms | 36.1 ms | 32.0 | 0 | 31,496 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,201.4 | 13.9 | 15,380 ms | 71.8 ms | 64.0 | 0 | 36,032 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 86.8 | 86.9 | 614 ms | 11.5 ms | 1.0 | 0 | 2,603 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 158.2 | 78.6 | 891 ms | 12.7 ms | 2.0 | 0 | 4,744 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 247.2 | 60.5 | 1,439 ms | 16.5 ms | 4.0 | 0 | 7,416 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 401.3 | 47.7 | 2,542 ms | 21.0 ms | 8.0 | 0 | 12,032 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 564.8 | 31.8 | 4,757 ms | 31.4 ms | 16.0 | 0 | 16,944 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 698.9 | 17.8 | 9,293 ms | 56.0 ms | 32.0 | 0 | 20,960 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 70.8 | 70.9 | 688 ms | 14.1 ms | 1.0 | 0 | 2,123 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 126.9 | 63.1 | 941 ms | 15.9 ms | 2.0 | 0 | 3,808 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 179.6 | 44.0 | 1,484 ms | 22.7 ms | 4.0 | 0 | 5,387 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 271.0 | 32.3 | 2,551 ms | 30.9 ms | 8.0 | 0 | 8,128 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 358.2 | 20.3 | 4,740 ms | 49.2 ms | 16.0 | 0 | 10,736 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 51.5 | 51.6 | 1,933 ms | 19.4 ms | 1.0 | 0 | 1,543 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 90.9 | 44.5 | 2,744 ms | 22.5 ms | 2.0 | 0 | 2,728 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 115.8 | 27.1 | 4,437 ms | 36.9 ms | 4.0 | 0 | 3,472 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 163.3 | 17.6 | 7,774 ms | 57.0 ms | 8.0 | 0 | 4,896 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 31.3 | 34.1 | 3,006 ms | 29.3 ms | 0.9 | 0 | 939 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 59.4 | 28.8 | 4,216 ms | 34.7 ms | 2.0 | 0 | 1,782 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 69.3 | 15.7 | 6,668 ms | 63.6 ms | 4.0 | 0 | 2,076 | 29.9 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.