Speed test matrix
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 on CMP 170HX 64GB ×2
vllm · 0.32.0rc1.dev249+g18305948a · NVFP4 mixed5 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- GPU count
- 2
- VRAM
- 64 GB (2 × 32 GB)
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- NVFP4 mixed
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none. Layout: TP2 (stock vLLM flag) # 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --tensor-parallel-size 2
Prefill
prompt tok/s by context length8K context1,0338,084 ms TTFT
16K context1,03815,910 ms TTFT
32K context1,02432,086 ms TTFT
64K context1,02963,757 ms TTFT
120K context1,012121,378 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 86.7143 ms TTFT | 140.0398 ms TTFT | 225.1434 ms TTFT | 324.3738 ms TTFT | 440.31,730 ms TTFT | 584.62,593 ms TTFT | 695.24,334 ms TTFT | 778.06,755 ms TTFT |
| 16K | Looped | Looped | 210.89,383 ms TTFT | 305.416,838 ms TTFT | 380.5Requests queued31,738 ms TTFT | 10.4Requests queued42,966 ms TTFT | 10.4Requests queued43,018 ms TTFT | 10.4Requests queued43,175 ms TTFT |
| 32K | 85.73,471 ms TTFT | Looped | Looped | 302.414,908 ms TTFT | 414.528,002 ms TTFT | 13.4Requests queued44,478 ms TTFT | 13.4Requests queued44,576 ms TTFT | 13.3Requests queued44,722 ms TTFT |
| 64K | 85.22,524 ms TTFT | Looped | Looped | 297.510,336 ms TTFT | 403.319,254 ms TTFT | 535.637,174 ms TTFT | 64.9Requests queued73,465 ms TTFT | Over capacity |
| 120K | 83.2Requests queued1,029 ms TTFT | Looped | 202.11,527 ms TTFT | 288.12,324 ms TTFT | 386.53,806 ms TTFT | 503.87,001 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 1,033.0 | — | 8,084 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 1,038.0 | — | 15,910 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,024.0 | — | 32,086 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,029.0 | — | 63,757 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 1,012.0 | — | 121,378 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 86.7 | 86.7 | 143 ms | 11.5 ms | 1.0 | 0 | 2,600 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 140.0 | 69.6 | 398 ms | 14.4 ms | 2.0 | 0 | 4,200 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 225.1 | 56.5 | 434 ms | 17.7 ms | 4.0 | 0 | 6,752 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 324.3 | 40.9 | 738 ms | 24.5 ms | 8.0 | 0 | 9,728 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 440.3 | 27.8 | 1,730 ms | 35.9 ms | 16.0 | 0 | 13,200 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 584.6 | 18.3 | 2,593 ms | 54.6 ms | 32.0 | 0 | 17,536 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 695.2 | 10.6 | 4,334 ms | 94.1 ms | 64.0 | 0 | 20,864 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 778.0 | 5.5 | 6,755 ms | 182.1 ms | 128.0 | 0 | 23,296 | 29.9 s |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Valid | 210.8 | 45.5 | 9,383 ms | 22.0 ms | 4.0 | 0 | 6,320 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 305.4 | 27.7 | 16,838 ms | 36.1 ms | 8.0 | 0 | 9,160 | 30.0 s |
| Sustained decode | 16K | 16 | Requests queued | 380.5 | 13.2 | 31,738 ms | 75.8 ms | 16.0 | 0 | 11,407 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 10.4 | 0.5 | 42,966 ms | 1874.7 ms | 19.4 | 0 | 296 | 28.4 s |
| Sustained decode | 16K | 64 | Requests queued | 10.4 | 0.5 | 43,018 ms | 1874.2 ms | 19.4 | 0 | 296 | 28.4 s |
| Sustained decode | 16K | 128 | Requests queued | 10.4 | 0.5 | 43,175 ms | 1875.5 ms | 19.4 | 0 | 296 | 28.5 s |
| Sustained decode | 32K | 1 | Valid | 85.7 | 85.9 | 3,471 ms | 11.6 ms | 1.0 | 0 | 2,570 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 302.4 | 28.6 | 14,908 ms | 34.9 ms | 8.0 | 0 | 9,064 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 414.5 | 15.2 | 28,002 ms | 65.8 ms | 16.0 | 0 | 12,432 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 13.4 | 0.6 | 44,478 ms | 1650.3 ms | 22.3 | 0 | 387 | 29.0 s |
| Sustained decode | 32K | 64 | Requests queued | 13.4 | 0.6 | 44,576 ms | 1649.6 ms | 22.3 | 0 | 387 | 29.0 s |
| Sustained decode | 32K | 128 | Requests queued | 13.3 | 0.6 | 44,722 ms | 1650.2 ms | 22.2 | 0 | 387 | 29.1 s |
| Sustained decode | 64K | 1 | Valid | 85.2 | 85.4 | 2,524 ms | 11.7 ms | 1.0 | 0 | 2,554 | 30.0 s |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 297.5 | 30.7 | 10,336 ms | 32.6 ms | 8.0 | 0 | 8,920 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 403.3 | 17.3 | 19,254 ms | 57.9 ms | 16.0 | 0 | 12,096 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 535.6 | 8.7 | 37,174 ms | 115.5 ms | 32.0 | 0 | 16,064 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 64.9 | 0.9 | 73,465 ms | 1056.1 ms | 58.4 | 0 | 1,943 | 29.9 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Requests queued | 83.2 | 84.6 | 1,029 ms | 11.8 ms | 1.0 | 0 | 2,495 | 30.0 s |
| Sustained decode | 120K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 120K | 4 | Valid | 202.1 | 50.3 | 1,527 ms | 19.9 ms | 4.0 | 0 | 6,060 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 288.1 | 35.3 | 2,324 ms | 28.4 ms | 8.0 | 0 | 8,643 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 386.5 | 22.9 | 3,806 ms | 43.7 ms | 16.0 | 0 | 11,584 | 30.0 s |
| Sustained decode | 120K | 32 | Valid | 503.8 | 14.2 | 7,001 ms | 70.6 ms | 32.0 | 0 | 15,104 | 30.0 s |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.