Speed test matrix
Ornith-1.5-9B-NVFP4 on RTX 3090 24GB
vllm · 0.32.0 · NVFP46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve ornith-ai/Ornith-1.5-9B-NVFP4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only
Prefill
prompt tok/s by context length8K context5,9611,374 ms TTFT
16K context5,6612,866 ms TTFT
32K context5,1386,285 ms TTFT
64K context4,28415,038 ms TTFT
128K context3,29339,086 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 121.231 ms TTFT | 224.041 ms TTFT | 432.874 ms TTFT | 843.2136 ms TTFT | 1,457.4225 ms TTFT | 2,280.2361 ms TTFT | 2,813.1725 ms TTFT | Over capacity |
| 16K | 116.2108 ms TTFT | 213.5154 ms TTFT | 397.6255 ms TTFT | 720.8455 ms TTFT | 1,063.0859 ms TTFT | 1,428.71,720 ms TTFT | Over capacity | Over capacity |
| 32K | 112.3185 ms TTFT | 204.2256 ms TTFT | 365.8411 ms TTFT | 621.7713 ms TTFT | 830.31,341 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 105.2113 ms TTFT | 187.6143 ms TTFT | 314.8239 ms TTFT | 485.9386 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 93.2568 ms TTFT | 159.9762 ms TTFT | 244.61,179 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 77.01,195 ms TTFT | 123.01,587 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,961.0 | — | 1,374 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,661.0 | — | 2,866 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,138.0 | — | 6,285 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,284.0 | — | 15,038 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,293.0 | — | 39,086 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 121.2 | 121.4 | 31 ms | 8.2 ms | 1.0 | 0 | 3,636 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 224.0 | 112.3 | 41 ms | 8.9 ms | 2.0 | 0 | 6,718 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 432.8 | 108.6 | 74 ms | 9.2 ms | 4.0 | 0 | 12,984 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 843.2 | 105.9 | 136 ms | 9.4 ms | 8.0 | 0 | 25,288 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,457.4 | 91.8 | 225 ms | 10.9 ms | 16.0 | 0 | 43,712 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,280.2 | 71.8 | 361 ms | 13.9 ms | 32.0 | 0 | 68,384 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,813.1 | 44.1 | 725 ms | 22.7 ms | 64.0 | 0 | 84,398 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 116.2 | 116.3 | 108 ms | 8.6 ms | 1.0 | 0 | 3,486 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 213.5 | 106.8 | 154 ms | 9.4 ms | 2.0 | 0 | 6,406 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 397.6 | 99.3 | 255 ms | 10.1 ms | 4.0 | 0 | 11,926 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 720.8 | 89.6 | 455 ms | 11.2 ms | 8.0 | 0 | 21,624 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,063.0 | 65.6 | 859 ms | 15.3 ms | 16.0 | 0 | 31,888 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,428.7 | 43.4 | 1,720 ms | 23.0 ms | 32.0 | 0 | 42,848 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 112.3 | 112.3 | 185 ms | 8.9 ms | 1.0 | 0 | 3,367 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 204.2 | 102.0 | 256 ms | 9.8 ms | 2.0 | 0 | 6,125 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 365.8 | 91.1 | 411 ms | 11.0 ms | 4.0 | 0 | 10,972 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 621.7 | 76.8 | 713 ms | 13.0 ms | 8.0 | 0 | 18,648 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 830.3 | 50.6 | 1,341 ms | 19.7 ms | 16.0 | 0 | 24,896 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 105.2 | 105.3 | 113 ms | 9.5 ms | 1.0 | 0 | 3,155 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 187.6 | 93.9 | 143 ms | 10.7 ms | 2.0 | 0 | 5,626 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 314.8 | 78.9 | 239 ms | 12.7 ms | 4.0 | 0 | 9,444 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 485.9 | 60.9 | 386 ms | 16.4 ms | 8.0 | 0 | 14,576 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 93.2 | 93.3 | 568 ms | 10.7 ms | 1.0 | 0 | 2,796 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 159.9 | 79.6 | 762 ms | 12.6 ms | 2.0 | 0 | 4,798 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 244.6 | 60.3 | 1,179 ms | 16.6 ms | 4.0 | 0 | 7,336 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 77.0 | 77.1 | 1,195 ms | 13.0 ms | 1.0 | 0 | 2,310 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 123.0 | 60.9 | 1,587 ms | 16.4 ms | 2.0 | 0 | 3,688 | 30.0 s |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.