Speed test matrix
Ornith-1.5-9B-NVFP4 on RTX 3090 24GB
vllm · 0.32.0 · NVFP46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02
vllm serve ornith-ai/Ornith-1.5-9B-NVFP4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}Prefill
prompt tok/s by context length8K context5,7391,427 ms TTFT
16K context5,4202,993 ms TTFT
32K context4,8916,603 ms TTFT
64K context4,04815,914 ms TTFT
128K context3,06841,942 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 143.645 ms TTFT | 265.471 ms TTFT | 512.097 ms TTFT | 890.8166 ms TTFT | 1,448.9257 ms TTFT | 1,704.7480 ms TTFT | 1,488.6Requests queued940 ms TTFT | Over capacity |
| 16K | 130.5273 ms TTFT | 244.8512 ms TTFT | 492.6887 ms TTFT | 845.21,390 ms TTFT | 1,206.22,420 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 130.8338 ms TTFT | 265.8616 ms TTFT | 441.71,047 ms TTFT | 756.51,635 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 120.0466 ms TTFT | 215.7827 ms TTFT | 408.41,380 ms TTFT | 658.32,147 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 112.5746 ms TTFT | 207.61,279 ms TTFT | 323.02,099 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 82.81,516 ms TTFT | 163.52,668 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,739.0 | — | 1,427 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,420.0 | — | 2,993 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,891.0 | — | 6,603 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,048.0 | — | 15,914 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,068.0 | — | 41,942 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 143.6 | 147.3 | 45 ms | 6.8 ms | 1.0 | 0 | 4,308 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 265.4 | 137.3 | 71 ms | 7.3 ms | 2.0 | 0 | 7,959 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 512.0 | 132.0 | 97 ms | 7.6 ms | 4.0 | 0 | 15,358 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 890.8 | 117.2 | 166 ms | 8.5 ms | 8.0 | 0 | 26,721 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,448.9 | 93.2 | 257 ms | 10.7 ms | 16.0 | 0 | 43,448 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,704.7 | 53.8 | 480 ms | 18.6 ms | 32.0 | 0 | 51,102 | 30.0 s |
| Sustained decode | 0 | 64 | Requests queued | 1,488.6 | 31.3 | 940 ms | 32.0 ms | 52.7 | 0 | 44,658 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 130.5 | 134.2 | 273 ms | 7.5 ms | 1.0 | 0 | 3,912 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 244.8 | 126.9 | 512 ms | 7.9 ms | 2.0 | 0 | 7,342 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 492.6 | 125.6 | 887 ms | 8.0 ms | 4.0 | 0 | 14,779 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 845.2 | 105.9 | 1,390 ms | 9.5 ms | 8.0 | 0 | 25,351 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,206.2 | 73.9 | 2,420 ms | 13.5 ms | 16.0 | 0 | 36,176 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 130.8 | 134.5 | 338 ms | 7.4 ms | 1.0 | 0 | 3,924 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 265.8 | 135.4 | 616 ms | 7.4 ms | 2.0 | 0 | 7,972 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 441.7 | 113.9 | 1,047 ms | 8.8 ms | 4.0 | 0 | 13,249 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 756.5 | 92.6 | 1,635 ms | 10.8 ms | 8.0 | 0 | 22,694 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 120.0 | 122.7 | 466 ms | 8.1 ms | 1.0 | 0 | 3,601 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 215.7 | 111.7 | 827 ms | 9.0 ms | 2.0 | 0 | 6,470 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 408.4 | 101.1 | 1,380 ms | 9.9 ms | 4.0 | 0 | 12,245 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 658.3 | 77.6 | 2,147 ms | 12.9 ms | 8.0 | 0 | 19,750 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 112.5 | 115.9 | 746 ms | 8.6 ms | 1.0 | 0 | 3,373 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 207.6 | 106.0 | 1,279 ms | 9.4 ms | 2.0 | 0 | 6,225 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 323.0 | 81.9 | 2,099 ms | 12.2 ms | 4.0 | 0 | 9,691 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 82.8 | 86.3 | 1,516 ms | 11.6 ms | 1.0 | 0 | 2,484 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 163.5 | 84.5 | 2,668 ms | 11.8 ms | 2.0 | 0 | 4,904 | 30.0 s |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.