Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on RTX 3090 24GB
vllm · 0.32.0 · NVFP48 contexts (0–896K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --max-model-len 925696 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context8,758959 ms TTFT
16K context8,6441,923 ms TTFT
32K context7,9924,139 ms TTFT
64K context6,8809,591 ms TTFT
128K context5,43924,235 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 216.536 ms TTFT | 344.5192 ms TTFT | 528.170 ms TTFT | 781.5105 ms TTFT | 1,106.6147 ms TTFT | 1,571.0259 ms TTFT | 1,791.8Requests queued303 ms TTFT | Over capacity |
| 16K | Looped | 336.7765 ms TTFT | 513.51,369 ms TTFT | 759.72,503 ms TTFT | 1,070.14,553 ms TTFT | 1,464.88,660 ms TTFT | Over capacity | Over capacity |
| 32K | 209.3561 ms TTFT | 331.5834 ms TTFT | 508.91,363 ms TTFT | Looped | 1,027.24,655 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 204.5654 ms TTFT | Looped | 496.91,538 ms TTFT | 698.12,768 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | Looped | 310.41,062 ms TTFT | 456.21,657 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 178.2557 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 149.2Requests queued1,245 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 896K | 106.9Requests queued5,082 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,758.0 | — | 959 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,644.0 | — | 1,923 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,992.0 | — | 4,139 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,880.0 | — | 9,591 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,439.0 | — | 24,235 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 216.5 | 216.7 | 36 ms | 4.6 ms | 1.0 | 0 | 6,493 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 344.5 | 172.5 | 192 ms | 5.8 ms | 2.0 | 0 | 10,334 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 528.1 | 132.8 | 70 ms | 7.5 ms | 4.0 | 0 | 15,840 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 781.5 | 98.8 | 105 ms | 10.1 ms | 8.0 | 0 | 23,440 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,106.6 | 70.5 | 147 ms | 14.2 ms | 16.0 | 0 | 33,198 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,571.0 | 50.5 | 259 ms | 19.8 ms | 32.0 | 0 | 47,104 | 30.0 s |
| Sustained decode | 0 | 64 | Requests queued | 1,791.8 | 39.2 | 303 ms | 25.5 ms | 47.0 | 0 | 53,721 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 336.7 | 167.2 | 765 ms | 6.0 ms | 2.0 | 0 | 10,100 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 513.5 | 126.5 | 1,369 ms | 7.9 ms | 4.0 | 0 | 15,404 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 759.7 | 91.3 | 2,503 ms | 11.0 ms | 8.0 | 0 | 22,784 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,070.1 | 61.1 | 4,553 ms | 16.4 ms | 16.0 | 0 | 32,096 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,464.8 | 37.9 | 8,660 ms | 26.4 ms | 32.0 | 0 | 43,936 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 209.3 | 209.5 | 561 ms | 4.8 ms | 1.0 | 0 | 6,280 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 331.5 | 164.7 | 834 ms | 6.1 ms | 2.0 | 0 | 9,944 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 508.9 | 124.6 | 1,363 ms | 8.0 ms | 4.0 | 0 | 15,264 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 1,027.2 | 57.8 | 4,655 ms | 17.3 ms | 16.0 | 0 | 30,816 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 204.5 | 204.7 | 654 ms | 4.9 ms | 1.0 | 0 | 6,135 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Valid | 496.9 | 121.2 | 1,538 ms | 8.3 ms | 4.0 | 0 | 14,907 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 698.1 | 82.7 | 2,768 ms | 12.1 ms | 8.0 | 0 | 20,944 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Valid | 310.4 | 154.1 | 1,062 ms | 6.5 ms | 2.0 | 0 | 9,310 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 456.2 | 111.6 | 1,657 ms | 9.0 ms | 4.0 | 0 | 13,684 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 178.2 | 178.5 | 557 ms | 5.6 ms | 1.0 | 0 | 5,345 | 30.0 s |
| Sustained decode | 256K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Requests queued | 149.2 | 152.9 | 1,245 ms | 6.5 ms | 1.0 | 0 | 4,474 | 30.0 s |
| Sustained decode | 512K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 1 | Requests queued | 106.9 | 126.1 | 5,082 ms | 7.9 ms | 0.9 | 0 | 3,206 | 30.0 s |
| Sustained decode | 896K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.