Speed test matrix
NVIDIA-Nemotron-Nano-12B-v2-AWQ-4bit on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · AWQ 4-bit5 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve cyankiwi/NVIDIA-Nemotron-Nano-12B-v2-AWQ-4bit --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context4,1692,000 ms TTFT
16K context3,9854,141 ms TTFT
32K context3,8538,527 ms TTFT
64K context3,60818,170 ms TTFT
120K context3,21238,229 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 113.7109 ms TTFT | 214.1118 ms TTFT | 396.8190 ms TTFT | 703.2277 ms TTFT | 1,076.0464 ms TTFT | 1,445.3731 ms TTFT | 1,718.51,329 ms TTFT | 1,831.91,912 ms TTFT |
| 16K | 109.1318 ms TTFT | 201.2445 ms TTFT | Looped | 592.31,108 ms TTFT | 834.51,950 ms TTFT | 1,042.03,990 ms TTFT | 1,173.68,301 ms TTFT | Over capacity |
| 32K | 104.8260 ms TTFT | 189.5370 ms TTFT | 324.0427 ms TTFT | 508.2875 ms TTFT | 684.31,351 ms TTFT | 817.42,516 ms TTFT | Over capacity | Over capacity |
| 64K | 97.8729 ms TTFT | 168.2967 ms TTFT | 271.11,441 ms TTFT | 395.12,366 ms TTFT | 496.24,248 ms TTFT | Over capacity | Over capacity | Over capacity |
| 120K | 87.61,031 ms TTFT | 140.61,304 ms TTFT | 210.41,639 ms TTFT | 284.32,616 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,169.0 | — | 2,000 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 3,985.0 | — | 4,141 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,853.0 | — | 8,527 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,608.0 | — | 18,170 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 3,212.0 | — | 38,229 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 113.7 | 113.9 | 109 ms | 8.8 ms | 1.0 | 0 | 3,412 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 214.1 | 107.2 | 118 ms | 9.3 ms | 2.0 | 0 | 6,422 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 396.8 | 99.5 | 190 ms | 10.0 ms | 4.0 | 0 | 11,904 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 703.2 | 88.3 | 277 ms | 11.3 ms | 8.0 | 0 | 21,096 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,076.0 | 67.6 | 464 ms | 14.8 ms | 16.0 | 0 | 32,304 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,445.3 | 45.3 | 731 ms | 22.1 ms | 32.0 | 0 | 43,360 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,718.5 | 26.8 | 1,329 ms | 37.3 ms | 64.0 | 0 | 51,520 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 1,831.9 | 13.9 | 1,912 ms | 71.7 ms | 128.0 | 0 | 54,912 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 109.1 | 109.3 | 318 ms | 9.2 ms | 1.0 | 0 | 3,271 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 201.2 | 100.6 | 445 ms | 9.9 ms | 2.0 | 0 | 6,036 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 592.3 | 72.8 | 1,108 ms | 13.7 ms | 8.0 | 0 | 17,768 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 834.5 | 50.2 | 1,950 ms | 19.9 ms | 16.0 | 0 | 25,040 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,042.0 | 29.9 | 3,990 ms | 33.4 ms | 32.0 | 0 | 31,264 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,173.6 | 15.5 | 8,301 ms | 64.3 ms | 64.0 | 0 | 35,200 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 104.8 | 105.1 | 260 ms | 9.5 ms | 1.0 | 0 | 3,143 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 189.5 | 95.0 | 370 ms | 10.5 ms | 2.0 | 0 | 5,684 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 324.0 | 81.1 | 427 ms | 12.3 ms | 4.0 | 0 | 9,716 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 508.2 | 63.5 | 875 ms | 15.7 ms | 8.0 | 0 | 15,240 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 684.3 | 42.6 | 1,351 ms | 23.5 ms | 16.0 | 0 | 20,528 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 817.4 | 25.2 | 2,516 ms | 39.7 ms | 32.0 | 0 | 24,544 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 97.8 | 98.1 | 729 ms | 10.2 ms | 1.0 | 0 | 2,935 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 168.2 | 83.9 | 967 ms | 11.9 ms | 2.0 | 0 | 5,046 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 271.1 | 66.8 | 1,441 ms | 15.0 ms | 4.0 | 0 | 8,130 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 395.1 | 47.6 | 2,366 ms | 21.0 ms | 8.0 | 0 | 11,848 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 496.2 | 28.5 | 4,248 ms | 35.1 ms | 16.0 | 0 | 14,880 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 87.6 | 88.0 | 1,031 ms | 11.4 ms | 1.0 | 0 | 2,628 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 140.6 | 70.3 | 1,304 ms | 14.2 ms | 2.0 | 0 | 4,218 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 210.4 | 52.1 | 1,639 ms | 19.2 ms | 4.0 | 0 | 6,308 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 284.3 | 34.5 | 2,616 ms | 29.0 ms | 8.0 | 0 | 8,530 | 30.0 s |
| Sustained decode | 120K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.