Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on RTX 3090 24GB
vllm · 0.26.0 · NVFP43 contexts (0–16K) × 3 concurrency levels (1–16) · prefill at 4K, 8K, 16K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- AMD Ryzen 9 9950X 16-Core Processor
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.0
- Quantization
- NVFP4
Run
- Served model
- nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.7
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 15 s
- Max output
- 8,192 tokens
Launch command
/home/lotto/vllm/bin/vllm serve /home/lotto/models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --served-model-name nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --host 127.0.0.1 --port 36865 --max-model-len 24576 --gpu-memory-utilization 0.9 --max-num-seqs 64
Prefill
prompt tok/s by context length4K context6,659629 ms TTFT
8K context6,4771,265 ms TTFT
16K context6,4082,533 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 4 | 16 |
|---|---|---|---|
| 0 | 208.833 ms TTFT | 511.6351 ms TTFT | 1,092.4245 ms TTFT |
| 4K | 207.8634 ms TTFT | 507.91,809 ms TTFT | 1,087.35,318 ms TTFT |
| 16K | 204.72,516 ms TTFT | 494.16,238 ms TTFT | 1,021.921,292 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 4K | 1 | Valid | 6,659.0 | — | 629 ms | — | — | 1 | — | — |
| Prefill | 8K | 1 | Valid | 6,477.0 | — | 1,265 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,408.0 | — | 2,533 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 208.8 | 208.8 | 33 ms | 4.8 ms | 1.0 | 0 | 3,132 | 15.0 s |
| Sustained decode | 0 | 4 | Valid | 511.6 | 129.2 | 351 ms | 7.7 ms | 4.0 | 0 | 7,672 | 15.0 s |
| Sustained decode | 0 | 16 | Valid | 1,092.4 | 70.7 | 245 ms | 14.2 ms | 16.0 | 0 | 16,384 | 15.0 s |
| Sustained decode | 4K | 1 | Valid | 207.8 | 208.0 | 634 ms | 4.8 ms | 1.0 | 0 | 3,117 | 15.0 s |
| Sustained decode | 4K | 4 | Valid | 507.9 | 123.4 | 1,809 ms | 8.1 ms | 4.0 | 0 | 7,616 | 15.0 s |
| Sustained decode | 4K | 16 | Valid | 1,087.3 | 56.2 | 5,318 ms | 17.8 ms | 16.0 | 0 | 16,304 | 15.0 s |
| Sustained decode | 16K | 1 | Valid | 204.7 | 204.9 | 2,516 ms | 4.9 ms | 1.0 | 0 | 3,070 | 15.0 s |
| Sustained decode | 16K | 4 | Valid | 494.1 | 104.7 | 6,238 ms | 9.6 ms | 4.0 | 0 | 7,408 | 15.0 s |
| Sustained decode | 16K | 16 | Valid | 1,021.9 | 33.6 | 21,292 ms | 29.8 ms | 16.0 | 0 | 15,328 | 15.0 s |