Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on RTX 3090 24GB
vllm · 0.32.0 · NVFP48 contexts (0–1016K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 7900X (12C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 350 W limit (VBIOS max), PCIe Gen4 x16 | AMD Ryzen 9 7900X 12C | 32 GB DDR5 | Ubuntu 24.04.5, NVIDIA driver 595.99.02 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --max-model-len 1048576 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context7,5241,116 ms TTFT
16K context6,4962,559 ms TTFT
32K context4,9486,685 ms TTFT
64K context3,36319,622 ms TTFT
128K context2,05864,042 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 201.037 ms TTFT | 324.3201 ms TTFT | 502.271 ms TTFT | 752.3107 ms TTFT | 1,070.6153 ms TTFT | 1,529.8286 ms TTFT | 1,724.8Requests queued424 ms TTFT | 1,736.7Requests queued393 ms TTFT |
| 16K | 175.9210 ms TTFT | 284.4305 ms TTFT | 447.1503 ms TTFT | Looped | Looped | Looped | Looped | Over capacity |
| 32K | 156.5514 ms TTFT | 256.7773 ms TTFT | 406.51,242 ms TTFT | 612.02,224 ms TTFT | 836.84,206 ms TTFT | 1,098.78,274 ms TTFT | Over capacity | Over capacity |
| 64K | 128.31,546 ms TTFT | 214.52,293 ms TTFT | 340.03,722 ms TTFT | 519.26,693 ms TTFT | 676.412,667 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 93.75,259 ms TTFT | 158.77,763 ms TTFT | 252.212,894 ms TTFT | 387.323,191 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 47.5Requests queued7,074 ms TTFT | 59.2Requests queued11,868 ms TTFT | 170.019,842 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1016K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,524.0 | — | 1,116 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,496.0 | — | 2,559 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,948.0 | — | 6,685 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,363.0 | — | 19,622 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,058.0 | — | 64,042 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 201.0 | 202.1 | 37 ms | 4.9 ms | 1.0 | 0 | 6,030 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 324.3 | 163.0 | 201 ms | 6.1 ms | 2.0 | 0 | 9,728 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 502.2 | 126.5 | 71 ms | 7.9 ms | 4.0 | 0 | 15,064 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 752.3 | 95.1 | 107 ms | 10.5 ms | 8.0 | 0 | 22,568 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,070.6 | 68.3 | 153 ms | 14.6 ms | 16.0 | 0 | 32,119 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,529.8 | 49.1 | 286 ms | 20.4 ms | 32.0 | 0 | 45,888 | 30.0 s |
| Sustained decode | 0 | 64 | Requests queued | 1,724.8 | 35.9 | 424 ms | 27.9 ms | 50.0 | 0 | 51,700 | 30.0 s |
| Sustained decode | 0 | 128 | Requests queued | 1,736.7 | 36.1 | 393 ms | 27.7 ms | 50.0 | 0 | 52,100 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 175.9 | 176.4 | 210 ms | 5.7 ms | 1.0 | 0 | 5,277 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 284.4 | 142.2 | 305 ms | 7.0 ms | 2.0 | 0 | 8,532 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 447.1 | 111.3 | 503 ms | 9.0 ms | 4.0 | 0 | 13,412 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 156.5 | 157.0 | 514 ms | 6.4 ms | 1.0 | 0 | 4,696 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 256.7 | 127.8 | 773 ms | 7.8 ms | 2.0 | 0 | 7,702 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 406.5 | 100.0 | 1,242 ms | 10.0 ms | 4.0 | 0 | 12,197 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 612.0 | 73.2 | 2,224 ms | 13.7 ms | 8.0 | 0 | 18,352 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 836.8 | 47.6 | 4,206 ms | 21.0 ms | 16.0 | 0 | 25,104 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,098.7 | 28.5 | 8,274 ms | 35.1 ms | 32.0 | 0 | 32,960 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 128.3 | 128.5 | 1,546 ms | 7.8 ms | 1.0 | 0 | 3,848 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 214.5 | 105.1 | 2,293 ms | 9.5 ms | 2.0 | 0 | 6,436 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 340.0 | 80.1 | 3,722 ms | 12.5 ms | 4.0 | 0 | 10,196 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 519.2 | 56.6 | 6,693 ms | 17.7 ms | 8.0 | 0 | 15,576 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 676.4 | 32.2 | 12,667 ms | 31.1 ms | 16.0 | 0 | 20,288 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 93.7 | 93.8 | 5,259 ms | 10.7 ms | 1.0 | 0 | 2,811 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 158.7 | 74.3 | 7,763 ms | 13.5 ms | 2.0 | 0 | 4,760 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 252.2 | 51.7 | 12,894 ms | 19.4 ms | 4.0 | 0 | 7,564 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 387.3 | 32.1 | 23,191 ms | 31.2 ms | 8.0 | 0 | 11,616 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Requests queued | 47.5 | 60.8 | 7,074 ms | 16.4 ms | 0.9 | 0 | 1,425 | 30.0 s |
| Sustained decode | 256K | 2 | Requests queued | 59.2 | 48.0 | 11,868 ms | 21.0 ms | 1.7 | 0 | 1,774 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 170.0 | 35.4 | 19,842 ms | 28.4 ms | 4.0 | 0 | 5,100 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Over capacity | — | — | — | — | 0.8 | 0 | 0 | 30.0 s |
| Sustained decode | 512K | 2 | Over capacity | — | — | — | — | 1.5 | 0 | 0 | 30.0 s |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 1 | Over capacity | — | — | — | — | 0.8 | 0 | 0 | 30.0 s |
| Sustained decode | 1016K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.