Speed test matrix
Ornith-1.5-9B-NVFP4 on RTX 3090 24GB
vllm · 0.32.0 · NVFP46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve ornith-ai/Ornith-1.5-9B-NVFP4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context5,3901,520 ms TTFT
16K context4,7203,437 ms TTFT
32K context3,7368,644 ms TTFT
64K context2,63324,467 ms TTFT
128K context1,66977,094 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 115.531 ms TTFT | 214.649 ms TTFT | 412.776 ms TTFT | 789.1144 ms TTFT | 1,346.2227 ms TTFT | 2,078.3386 ms TTFT | 2,711.1767 ms TTFT | 3,075.41,149 ms TTFT |
| 16K | Looped | 182.2772 ms TTFT | 309.01,284 ms TTFT | 525.52,310 ms TTFT | 775.54,360 ms TTFT | 999.18,512 ms TTFT | Over capacity | Over capacity |
| 32K | 87.7644 ms TTFT | 158.7950 ms TTFT | 245.11,561 ms TTFT | 389.72,786 ms TTFT | 539.65,235 ms TTFT | 650.410,241 ms TTFT | Over capacity | Over capacity |
| 64K | 71.0662 ms TTFT | 126.0953 ms TTFT | 175.71,536 ms TTFT | 258.72,716 ms TTFT | 336.65,093 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 51.41,970 ms TTFT | 89.12,906 ms TTFT | 111.44,781 ms TTFT | 154.38,517 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 31.1Requests queued3,051 ms TTFT | 57.54,421 ms TTFT | 66.77,086 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,390.0 | — | 1,520 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,720.0 | — | 3,437 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,736.0 | — | 8,644 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 2,633.0 | — | 24,467 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,669.0 | — | 77,094 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 115.5 | 115.9 | 31 ms | 8.6 ms | 1.0 | 0 | 3,465 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 214.6 | 107.7 | 49 ms | 9.3 ms | 2.0 | 0 | 6,437 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 412.7 | 103.7 | 76 ms | 9.6 ms | 4.0 | 0 | 12,380 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 789.1 | 99.6 | 144 ms | 10.0 ms | 8.0 | 0 | 23,673 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,346.2 | 85.2 | 227 ms | 11.7 ms | 16.0 | 0 | 40,384 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,078.3 | 65.9 | 386 ms | 15.2 ms | 32.0 | 0 | 62,336 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,711.1 | 42.7 | 767 ms | 23.4 ms | 64.0 | 0 | 81,280 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,075.4 | 23.8 | 1,149 ms | 42.0 ms | 128.0 | 0 | 92,160 | 30.0 s |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 182.2 | 90.6 | 772 ms | 11.0 ms | 2.0 | 0 | 5,464 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 309.0 | 75.8 | 1,284 ms | 13.2 ms | 4.0 | 0 | 9,268 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 525.5 | 62.7 | 2,310 ms | 15.9 ms | 8.0 | 0 | 15,767 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 775.5 | 43.9 | 4,360 ms | 22.8 ms | 16.0 | 0 | 23,264 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 999.1 | 25.8 | 8,512 ms | 38.8 ms | 32.0 | 0 | 29,952 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 87.7 | 87.9 | 644 ms | 11.4 ms | 1.0 | 0 | 2,631 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 158.7 | 78.8 | 950 ms | 12.7 ms | 2.0 | 0 | 4,760 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 245.1 | 59.8 | 1,561 ms | 16.7 ms | 4.0 | 0 | 7,353 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 389.7 | 46.1 | 2,786 ms | 21.7 ms | 8.0 | 0 | 11,688 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 539.6 | 30.0 | 5,235 ms | 33.3 ms | 16.0 | 0 | 16,176 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 650.4 | 16.2 | 10,241 ms | 61.7 ms | 32.0 | 0 | 19,488 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 71.0 | 71.0 | 662 ms | 14.1 ms | 1.0 | 0 | 2,129 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 126.0 | 62.5 | 953 ms | 16.0 ms | 2.0 | 0 | 3,778 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 175.7 | 43.0 | 1,536 ms | 23.3 ms | 4.0 | 0 | 5,268 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 258.7 | 30.7 | 2,716 ms | 32.6 ms | 8.0 | 0 | 7,760 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 336.6 | 18.8 | 5,093 ms | 53.1 ms | 16.0 | 0 | 10,096 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 51.4 | 51.4 | 1,970 ms | 19.4 ms | 1.0 | 0 | 1,541 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 89.1 | 43.5 | 2,906 ms | 23.0 ms | 2.0 | 0 | 2,670 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 111.4 | 25.8 | 4,781 ms | 38.8 ms | 4.0 | 0 | 3,340 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 154.3 | 16.3 | 8,517 ms | 61.4 ms | 8.0 | 0 | 4,624 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 31.1 | 33.9 | 3,051 ms | 29.5 ms | 0.9 | 0 | 932 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 57.5 | 27.8 | 4,421 ms | 36.1 ms | 2.0 | 0 | 1,726 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 66.7 | 15.0 | 7,086 ms | 66.9 ms | 4.0 | 0 | 2,000 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.