Speed test matrix
Ornith-1.5-9B-NVFP4 on RTX 3090 24GB
vllm · 0.32.0 · NVFP46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- Intel Core i7-9700K (8C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04 vllm serve ornith-ai/Ornith-1.5-9B-NVFP4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context6,0701,350 ms TTFT
16K context5,1683,140 ms TTFT
32K context4,1477,788 ms TTFT
64K context2,93121,977 ms TTFT
128K context1,85269,476 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 117.839 ms TTFT | 219.243 ms TTFT | 422.984 ms TTFT | 807.5132 ms TTFT | 1,385.4259 ms TTFT | 2,166.5373 ms TTFT | 2,865.6751 ms TTFT | 3,274.31,161 ms TTFT |
| 16K | 102.4506 ms TTFT | 187.5728 ms TTFT | 319.21,191 ms TTFT | 548.12,105 ms TTFT | 820.43,983 ms TTFT | 1,071.27,749 ms TTFT | Over capacity | Over capacity |
| 32K | 90.3667 ms TTFT | 163.8898 ms TTFT | 254.11,462 ms TTFT | 410.52,573 ms TTFT | 576.64,801 ms TTFT | 709.49,351 ms TTFT | Over capacity | Over capacity |
| 64K | 73.0685 ms TTFT | 130.8945 ms TTFT | 183.11,491 ms TTFT | 275.52,569 ms TTFT | 362.84,755 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 52.71,953 ms TTFT | 92.82,775 ms TTFT | 117.04,458 ms TTFT | 164.97,823 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 31.9Requests queued3,008 ms TTFT | 60.14,281 ms TTFT | 70.06,733 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,070.0 | — | 1,350 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,168.0 | — | 3,140 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,147.0 | — | 7,788 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 2,931.0 | — | 21,977 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,852.0 | — | 69,476 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 117.8 | 118.2 | 39 ms | 8.5 ms | 1.0 | 0 | 3,535 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 219.2 | 110.0 | 43 ms | 9.1 ms | 2.0 | 0 | 6,574 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 422.9 | 106.2 | 84 ms | 9.4 ms | 4.0 | 0 | 12,684 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 807.5 | 101.6 | 132 ms | 9.8 ms | 8.0 | 0 | 24,224 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,385.4 | 87.6 | 259 ms | 11.4 ms | 16.0 | 0 | 41,552 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,166.5 | 68.6 | 373 ms | 14.6 ms | 32.0 | 0 | 64,992 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,865.6 | 45.2 | 751 ms | 22.1 ms | 64.0 | 0 | 85,993 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,274.3 | 25.4 | 1,161 ms | 39.4 ms | 128.0 | 0 | 98,247 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 102.4 | 102.6 | 506 ms | 9.7 ms | 1.0 | 0 | 3,073 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 187.5 | 93.3 | 728 ms | 10.7 ms | 2.0 | 0 | 5,624 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 319.2 | 78.5 | 1,191 ms | 12.7 ms | 4.0 | 0 | 9,576 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 548.1 | 65.8 | 2,105 ms | 15.2 ms | 8.0 | 0 | 16,440 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 820.4 | 46.9 | 3,983 ms | 21.3 ms | 16.0 | 0 | 24,608 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,071.2 | 28.1 | 7,749 ms | 35.5 ms | 32.0 | 0 | 32,128 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 90.3 | 90.5 | 667 ms | 11.1 ms | 1.0 | 0 | 2,710 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 163.8 | 81.4 | 898 ms | 12.3 ms | 2.0 | 0 | 4,914 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 254.1 | 62.2 | 1,462 ms | 16.1 ms | 4.0 | 0 | 7,620 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 410.5 | 48.7 | 2,573 ms | 20.5 ms | 8.0 | 0 | 12,315 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 576.6 | 32.5 | 4,801 ms | 30.8 ms | 16.0 | 0 | 17,297 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 709.4 | 18.0 | 9,351 ms | 55.5 ms | 32.0 | 0 | 21,280 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 73.0 | 73.1 | 685 ms | 13.7 ms | 1.0 | 0 | 2,189 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 130.8 | 65.0 | 945 ms | 15.4 ms | 2.0 | 0 | 3,924 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 183.1 | 44.9 | 1,491 ms | 22.3 ms | 4.0 | 0 | 5,492 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 275.5 | 32.8 | 2,569 ms | 30.5 ms | 8.0 | 0 | 8,264 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 362.8 | 20.5 | 4,755 ms | 48.7 ms | 16.0 | 0 | 10,880 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 52.7 | 52.7 | 1,953 ms | 19.0 ms | 1.0 | 0 | 1,579 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 92.8 | 45.4 | 2,775 ms | 22.0 ms | 2.0 | 0 | 2,782 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 117.0 | 27.4 | 4,458 ms | 36.6 ms | 4.0 | 0 | 3,508 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 164.9 | 17.7 | 7,823 ms | 56.5 ms | 8.0 | 0 | 4,944 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 31.9 | 34.7 | 3,008 ms | 28.8 ms | 0.9 | 0 | 955 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 60.1 | 29.1 | 4,281 ms | 34.4 ms | 2.0 | 0 | 1,804 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 70.0 | 15.9 | 6,733 ms | 63.0 ms | 4.0 | 0 | 2,100 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.