Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · NVFP48 contexts (0–1016K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 3 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 --max-model-len 1048576 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --speculative-config {"method":"mtp","num_speculative_tokens":3}Prefill
prompt tok/s by context length8K context7,5701,103 ms TTFT
16K context7,4552,215 ms TTFT
32K context5,8205,647 ms TTFT
64K context6,35310,323 ms TTFT
128K context5,14525,458 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 87.9142 ms TTFT | 176.22,716 ms TTFT | 347.72,520 ms TTFT | 644.52,674 ms TTFT | 979.62,658 ms TTFT | 1,413.63,069 ms TTFT | 1,940.91,866 ms TTFT | 2,385.63,516 ms TTFT |
| 16K | 86.51,182 ms TTFT | 179.71,908 ms TTFT | Looped | 657.65,080 ms TTFT | 960.79,348 ms TTFT | 1,345.318,009 ms TTFT | 1,120.2Requests queued36,414 ms TTFT | 552.5Requests queued42,001 ms TTFT |
| 32K | 80.01,322 ms TTFT | 168.92,151 ms TTFT | Looped | 616.35,626 ms TTFT | 901.710,284 ms TTFT | 1,242.119,495 ms TTFT | 819.7Requests queued38,934 ms TTFT | 457.0Requests queued43,090 ms TTFT |
| 64K | 84.11,515 ms TTFT | 166.92,518 ms TTFT | 336.84,009 ms TTFT | 580.16,337 ms TTFT | 829.510,957 ms TTFT | 1,134.521,012 ms TTFT | 1,428.541,984 ms TTFT | 513.1Requests queued68,198 ms TTFT |
| 128K | 88.02,723 ms TTFT | 192.34,446 ms TTFT | 324.56,613 ms TTFT | 585.811,010 ms TTFT | 735.219,819 ms TTFT | 907.037,605 ms TTFT | 1,136.075,394 ms TTFT | Over capacity |
| 256K | 85.54,335 ms TTFT | 184.46,915 ms TTFT | 331.710,761 ms TTFT | 450.516,835 ms TTFT | 573.529,947 ms TTFT | 686.556,288 ms TTFT | Over capacity | Over capacity |
| 512K | 88.1Requests queued7,252 ms TTFT | 107.0Requests queued10,964 ms TTFT | 267.816,799 ms TTFT | 359.026,048 ms TTFT | 403.644,515 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1016K | Over capacity | Over capacity | Over capacity | 1.0Requests queued27,864 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,570.0 | — | 1,103 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,455.0 | — | 2,215 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,820.0 | — | 5,647 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,353.0 | — | 10,323 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,145.0 | — | 25,458 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 87.9 | 88.2 | 142 ms | 11.3 ms | 1.0 | 0 | 2,635 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 176.2 | 89.0 | 2,716 ms | 11.3 ms | 2.0 | 0 | 5,285 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 347.7 | 87.7 | 2,520 ms | 11.4 ms | 4.0 | 0 | 10,424 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 644.5 | 81.5 | 2,674 ms | 12.3 ms | 8.0 | 0 | 19,330 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 979.6 | 62.7 | 2,658 ms | 15.9 ms | 16.0 | 0 | 29,359 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,413.6 | 45.8 | 3,069 ms | 21.9 ms | 32.0 | 0 | 42,380 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,940.9 | 31.7 | 1,866 ms | 31.5 ms | 64.0 | 0 | 58,278 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 2,385.6 | 19.3 | 3,516 ms | 51.8 ms | 128.0 | 0 | 71,498 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 86.5 | 87.5 | 1,182 ms | 11.4 ms | 1.0 | 0 | 2,594 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 179.7 | 90.4 | 1,908 ms | 11.1 ms | 2.0 | 0 | 5,387 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 657.6 | 74.3 | 5,080 ms | 13.5 ms | 8.0 | 0 | 19,729 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 960.7 | 52.3 | 9,348 ms | 19.1 ms | 16.0 | 0 | 28,798 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,345.3 | 31.2 | 18,009 ms | 32.0 ms | 32.0 | 0 | 40,367 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,120.2 | 14.5 | 36,414 ms | 68.8 ms | 61.0 | 0 | 33,617 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 552.5 | 9.4 | 42,001 ms | 105.9 ms | 63.7 | 0 | 16,517 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 80.0 | 82.6 | 1,322 ms | 12.1 ms | 1.0 | 0 | 2,398 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 168.9 | 85.6 | 2,151 ms | 11.7 ms | 2.0 | 0 | 5,064 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 616.3 | 72.1 | 5,626 ms | 13.9 ms | 8.0 | 0 | 18,485 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 901.7 | 48.9 | 10,284 ms | 20.5 ms | 16.0 | 0 | 27,019 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,242.1 | 29.6 | 19,495 ms | 33.8 ms | 32.0 | 0 | 37,204 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 819.7 | 11.4 | 38,934 ms | 87.4 ms | 59.2 | 0 | 24,563 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 457.0 | 8.3 | 43,090 ms | 120.9 ms | 60.4 | 0 | 13,712 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 84.1 | 85.9 | 1,515 ms | 11.6 ms | 1.0 | 0 | 2,521 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 166.9 | 84.6 | 2,518 ms | 11.8 ms | 2.0 | 0 | 5,007 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 336.8 | 82.9 | 4,009 ms | 12.1 ms | 4.0 | 0 | 10,098 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 580.1 | 66.7 | 6,337 ms | 15.0 ms | 8.0 | 0 | 17,401 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 829.5 | 42.7 | 10,957 ms | 23.4 ms | 16.0 | 0 | 24,854 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,134.5 | 26.4 | 21,012 ms | 37.9 ms | 32.0 | 0 | 34,010 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,428.5 | 13.7 | 41,984 ms | 73.0 ms | 64.0 | 0 | 42,840 | 30.0 s |
| Sustained decode | 64K | 128 | Requests queued | 513.1 | 6.1 | 68,198 ms | 162.8 ms | 95.4 | 0 | 15,342 | 29.9 s |
| Sustained decode | 128K | 1 | Valid | 88.0 | 90.0 | 2,723 ms | 11.1 ms | 1.0 | 0 | 2,640 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 192.3 | 95.3 | 4,446 ms | 10.5 ms | 2.0 | 0 | 5,766 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 324.5 | 76.5 | 6,613 ms | 13.1 ms | 4.0 | 0 | 9,732 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 585.8 | 63.7 | 11,010 ms | 15.7 ms | 8.0 | 0 | 17,556 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 735.2 | 33.6 | 19,819 ms | 29.7 ms | 16.0 | 0 | 22,038 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 907.0 | 16.7 | 37,605 ms | 59.7 ms | 32.0 | 0 | 27,209 | 30.0 s |
| Sustained decode | 128K | 64 | Valid | 1,136.0 | 8.6 | 75,394 ms | 116.0 ms | 64.0 | 0 | 34,074 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 85.5 | 88.0 | 4,335 ms | 11.4 ms | 1.0 | 0 | 2,562 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 184.4 | 91.5 | 6,915 ms | 10.9 ms | 2.0 | 0 | 5,529 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 331.7 | 79.8 | 10,761 ms | 12.5 ms | 4.0 | 0 | 9,951 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 450.5 | 45.6 | 16,835 ms | 22.0 ms | 8.0 | 0 | 13,499 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 573.5 | 23.1 | 29,947 ms | 43.2 ms | 16.0 | 0 | 17,193 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 686.5 | 10.7 | 56,288 ms | 93.4 ms | 32.0 | 0 | 20,544 | 29.9 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Requests queued | 88.1 | 113.7 | 7,252 ms | 8.8 ms | 0.8 | 0 | 2,643 | 30.0 s |
| Sustained decode | 512K | 2 | Requests queued | 107.0 | 82.2 | 10,964 ms | 12.2 ms | 1.6 | 0 | 3,210 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 267.8 | 62.8 | 16,799 ms | 16.0 ms | 4.0 | 0 | 8,029 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 359.0 | 34.1 | 26,048 ms | 29.4 ms | 8.0 | 0 | 10,762 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 403.6 | 14.0 | 44,515 ms | 71.2 ms | 16.0 | 0 | 12,086 | 29.9 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 1 | Over capacity | — | — | — | — | 0.8 | 0 | 0 | 30.0 s |
| Sustained decode | 1016K | 2 | Over capacity | — | — | — | — | 1.6 | 0 | 0 | 30.0 s |
| Sustained decode | 1016K | 4 | Over capacity | — | — | — | — | 1.5 | 0 | 0 | 30.0 s |
| Sustained decode | 1016K | 8 | Requests queued | 1.0 | 1.4 | 27,864 ms | 710.4 ms | 2.3 | 0 | 29 | 29.4 s |
| Sustained decode | 1016K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1016K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.