Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on RTX 3090 24GB
vllm · 0.32.0 · NVFP48 contexts (0–896K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- Intel Core i7-9700K (8C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --max-model-len 925696 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context9,619873 ms TTFT
16K context9,3031,787 ms TTFT
32K context8,5603,865 ms TTFT
64K context7,6638,612 ms TTFT
128K context6,17221,357 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 214.940 ms TTFT | 346.646 ms TTFT | 533.4289 ms TTFT | 781.4107 ms TTFT | 1,119.8183 ms TTFT | 1,604.3238 ms TTFT | Looped | Over capacity |
| 16K | 212.1482 ms TTFT | 336.6705 ms TTFT | 527.31,266 ms TTFT | 754.12,276 ms TTFT | 1,093.34,090 ms TTFT | 1,508.67,723 ms TTFT | Over capacity | Over capacity |
| 32K | 207.7541 ms TTFT | 335.4790 ms TTFT | 512.61,272 ms TTFT | 741.02,242 ms TTFT | 1,055.54,183 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 202.9703 ms TTFT | 327.0924 ms TTFT | Looped | 709.22,522 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 193.2811 ms TTFT | Looped | 472.31,618 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | Looped | 295.3964 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 146.8Requests queued1,529 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 896K | 106.4Requests queued5,265 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,619.0 | — | 873 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,303.0 | — | 1,787 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,560.0 | — | 3,865 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,663.0 | — | 8,612 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,172.0 | — | 21,357 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 214.9 | 214.7 | 40 ms | 4.7 ms | 1.0 | 0 | 6,445 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 346.6 | 173.8 | 46 ms | 5.8 ms | 2.0 | 0 | 10,396 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 533.4 | 134.2 | 289 ms | 7.4 ms | 4.0 | 0 | 16,000 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 781.4 | 98.9 | 107 ms | 10.1 ms | 8.0 | 0 | 23,440 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,119.8 | 71.5 | 183 ms | 14.0 ms | 16.0 | 0 | 33,584 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,604.3 | 51.5 | 238 ms | 19.4 ms | 32.0 | 0 | 48,128 | 30.0 s |
| Sustained decode | 0 | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 212.1 | 212.2 | 482 ms | 4.7 ms | 1.0 | 0 | 6,363 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 336.6 | 167.6 | 705 ms | 6.0 ms | 2.0 | 0 | 10,096 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 527.3 | 129.8 | 1,266 ms | 7.7 ms | 4.0 | 0 | 15,816 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 754.1 | 91.2 | 2,276 ms | 11.0 ms | 8.0 | 0 | 22,624 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,093.3 | 63.0 | 4,090 ms | 15.9 ms | 16.0 | 0 | 32,800 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,508.6 | 40.1 | 7,723 ms | 25.0 ms | 32.0 | 0 | 45,248 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 207.7 | 208.0 | 541 ms | 4.8 ms | 1.0 | 0 | 6,232 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 335.4 | 166.7 | 790 ms | 6.0 ms | 2.0 | 0 | 10,061 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 512.6 | 125.9 | 1,272 ms | 7.9 ms | 4.0 | 0 | 15,376 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 741.0 | 88.8 | 2,242 ms | 11.3 ms | 8.0 | 0 | 22,224 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,055.5 | 60.3 | 4,183 ms | 16.6 ms | 16.0 | 0 | 31,664 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 202.9 | 203.1 | 703 ms | 4.9 ms | 1.0 | 0 | 6,087 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 327.0 | 162.6 | 924 ms | 6.1 ms | 2.0 | 0 | 9,809 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 709.2 | 84.7 | 2,522 ms | 11.8 ms | 8.0 | 0 | 21,272 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 193.2 | 193.7 | 811 ms | 5.2 ms | 1.0 | 0 | 5,795 | 30.0 s |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Valid | 472.3 | 115.7 | 1,618 ms | 8.6 ms | 4.0 | 0 | 14,169 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 256K | 2 | Valid | 295.3 | 147.8 | 964 ms | 6.8 ms | 2.0 | 0 | 8,858 | 30.0 s |
| Sustained decode | 256K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Requests queued | 146.8 | 152.0 | 1,529 ms | 6.6 ms | 1.0 | 0 | 4,405 | 30.0 s |
| Sustained decode | 512K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 1 | Requests queued | 106.4 | 126.4 | 5,265 ms | 7.9 ms | 0.9 | 0 | 3,191 | 30.0 s |
| Sustained decode | 896K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 896K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.