Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on RTX 3090 24GB
vllm · 0.32.0 · NVFP48 contexts (0–1016K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX 3090
- VRAM
- 24 GB
- CPU
- Intel Core i7-9700K (8C)
- RAM
- 32 GB
- OS
- Ubuntu 24.04.5
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- vllm/vllm-openai:v0.32.0-x86_64
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. KV cache: int4_per_token_head # 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04 vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --max-model-len 1048576 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --kv-cache-dtype int4_per_token_head --attention-backend TRITON_ATTN
Prefill
prompt tok/s by context length8K context8,3471,006 ms TTFT
16K context7,1252,333 ms TTFT
32K context5,6105,897 ms TTFT
64K context3,88816,972 ms TTFT
128K context2,40454,842 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 199.740 ms TTFT | 328.4313 ms TTFT | 513.074 ms TTFT | 761.4110 ms TTFT | 1,041.2Requests queued196 ms TTFT | 1,568.6273 ms TTFT | 1,784.8Requests queued368 ms TTFT | Looped |
| 16K | 175.7211 ms TTFT | 288.9307 ms TTFT | 458.8519 ms TTFT | 681.3938 ms TTFT | 937.81,731 ms TTFT | Looped | Looped | Over capacity |
| 32K | 157.1518 ms TTFT | 261.3746 ms TTFT | 417.11,178 ms TTFT | 637.12,067 ms TTFT | 865.93,880 ms TTFT | 1,144.77,497 ms TTFT | Over capacity | Over capacity |
| 64K | 129.01,437 ms TTFT | 215.32,107 ms TTFT | 348.63,414 ms TTFT | 535.36,052 ms TTFT | 707.011,334 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 94.74,687 ms TTFT | 161.16,930 ms TTFT | 263.611,388 ms TTFT | 412.220,301 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 49.6Requests queued6,417 ms TTFT | 65.7Requests queued10,640 ms TTFT | 177.917,528 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1016K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,347.0 | — | 1,006 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,125.0 | — | 2,333 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,610.0 | — | 5,897 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,888.0 | — | 16,972 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,404.0 | — | 54,842 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 199.7 | 200.5 | 40 ms | 5.0 ms | 1.0 | 0 | 5,992 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 328.4 | 164.8 | 313 ms | 6.1 ms | 2.0 | 0 | 9,852 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 513.0 | 129.2 | 74 ms | 7.7 ms | 4.0 | 0 | 15,388 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 761.4 | 96.3 | 110 ms | 10.4 ms | 8.0 | 0 | 22,840 | 30.0 s |
| Sustained decode | 0 | 16 | Requests queued | 1,041.2 | 67.2 | 196 ms | 14.9 ms | 16.0 | 0 | 31,232 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,568.6 | 50.3 | 273 ms | 19.9 ms | 32.0 | 0 | 47,040 | 30.0 s |
| Sustained decode | 0 | 64 | Requests queued | 1,784.8 | 37.1 | 368 ms | 26.9 ms | 50.0 | 0 | 53,550 | 30.0 s |
| Sustained decode | 0 | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 175.7 | 176.2 | 211 ms | 5.7 ms | 1.0 | 0 | 5,270 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 288.9 | 144.4 | 307 ms | 6.9 ms | 2.0 | 0 | 8,668 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 458.8 | 114.2 | 519 ms | 8.8 ms | 4.0 | 0 | 13,764 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 681.3 | 83.8 | 938 ms | 11.9 ms | 8.0 | 0 | 20,441 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 937.8 | 56.7 | 1,731 ms | 17.6 ms | 16.0 | 0 | 28,145 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 157.1 | 157.5 | 518 ms | 6.4 ms | 1.0 | 0 | 4,713 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 261.3 | 130.2 | 746 ms | 7.7 ms | 2.0 | 0 | 7,838 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 417.1 | 102.7 | 1,178 ms | 9.7 ms | 4.0 | 0 | 12,511 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 637.1 | 76.7 | 2,067 ms | 13.0 ms | 8.0 | 0 | 19,113 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 865.9 | 49.8 | 3,880 ms | 20.1 ms | 16.0 | 0 | 25,968 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,144.7 | 30.3 | 7,497 ms | 33.0 ms | 32.0 | 0 | 34,336 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 129.0 | 129.2 | 1,437 ms | 7.7 ms | 1.0 | 0 | 3,868 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 215.3 | 105.9 | 2,107 ms | 9.4 ms | 2.0 | 0 | 6,458 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 348.6 | 82.7 | 3,414 ms | 12.1 ms | 4.0 | 0 | 10,456 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 535.3 | 59.4 | 6,052 ms | 16.8 ms | 8.0 | 0 | 16,056 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 707.0 | 34.6 | 11,334 ms | 28.9 ms | 16.0 | 0 | 21,210 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 94.7 | 95.0 | 4,687 ms | 10.5 ms | 1.0 | 0 | 2,842 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 161.1 | 76.1 | 6,930 ms | 13.2 ms | 2.0 | 0 | 4,832 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 263.6 | 55.5 | 11,388 ms | 18.1 ms | 4.0 | 0 | 7,908 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 412.2 | 35.5 | 20,301 ms | 28.2 ms | 8.0 | 0 | 12,360 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Requests queued | 49.6 | 61.8 | 6,417 ms | 16.2 ms | 0.9 | 0 | 1,488 | 30.0 s |
| Sustained decode | 256K | 2 | Requests queued | 65.7 | 49.9 | 10,640 ms | 20.2 ms | 1.7 | 0 | 1,971 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 177.9 | 37.8 | 17,528 ms | 26.6 ms | 4.0 | 0 | 5,336 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Over capacity | — | — | — | — | 0.8 | 0 | 0 | 30.0 s |
| Sustained decode | 512K | 2 | Over capacity | — | — | — | — | 1.5 | 0 | 0 | 30.0 s |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 1 | Over capacity | — | — | — | — | 0.8 | 0 | 0 | 30.0 s |
| Sustained decode | 1016K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.