Speed test matrix
Nemotron-Cascade-2-30B-A3B-AWQ-4bit on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve cyankiwi/Nemotron-Cascade-2-30B-A3B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context9,370893 ms TTFT
16K context8,8991,858 ms TTFT
32K context8,4383,897 ms TTFT
64K context7,6708,552 ms TTFT
128K context6,23621,008 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 175.9103 ms TTFT | 271.0557 ms TTFT | 457.4229 ms TTFT | 770.4272 ms TTFT | 1,156.7364 ms TTFT | 1,720.3511 ms TTFT | 2,471.9629 ms TTFT | 3,233.21,103 ms TTFT |
| 16K | 172.8333 ms TTFT | 257.8449 ms TTFT | 431.5671 ms TTFT | 714.61,135 ms TTFT | 1,055.01,994 ms TTFT | 1,544.13,988 ms TTFT | 2,111.37,899 ms TTFT | 2,617.617,241 ms TTFT |
| 32K | 169.6398 ms TTFT | 252.9526 ms TTFT | 419.6749 ms TTFT | 686.71,199 ms TTFT | 986.62,106 ms TTFT | 1,402.43,999 ms TTFT | 1,861.88,178 ms TTFT | 2,232.818,454 ms TTFT |
| 64K | 164.8463 ms TTFT | 244.4601 ms TTFT | 396.7900 ms TTFT | 624.81,481 ms TTFT | 871.32,468 ms TTFT | 1,187.94,099 ms TTFT | 1,489.38,115 ms TTFT | Over capacity |
| 128K | 155.91,006 ms TTFT | 228.61,229 ms TTFT | 355.41,802 ms TTFT | 533.32,846 ms TTFT | 706.95,108 ms TTFT | 908.78,837 ms TTFT | Over capacity | Over capacity |
| 248K | 141.71,456 ms TTFT | 199.91,923 ms TTFT | 296.32,909 ms TTFT | 415.44,704 ms TTFT | 521.17,913 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,370.0 | — | 893 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,899.0 | — | 1,858 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,438.0 | — | 3,897 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,670.0 | — | 8,552 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,236.0 | — | 21,008 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 175.9 | 176.1 | 103 ms | 5.7 ms | 1.0 | 0 | 5,276 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 271.0 | 135.6 | 557 ms | 7.4 ms | 2.0 | 0 | 8,130 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 457.4 | 114.8 | 229 ms | 8.7 ms | 4.0 | 0 | 13,721 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 770.4 | 96.7 | 272 ms | 10.3 ms | 8.0 | 0 | 23,112 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,156.7 | 73.0 | 364 ms | 13.7 ms | 16.0 | 0 | 34,704 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,720.3 | 54.4 | 511 ms | 18.4 ms | 32.0 | 0 | 51,616 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,471.9 | 38.9 | 629 ms | 25.7 ms | 64.0 | 0 | 74,176 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,233.2 | 25.2 | 1,103 ms | 39.7 ms | 128.0 | 0 | 97,024 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 172.8 | 173.1 | 333 ms | 5.8 ms | 1.0 | 0 | 5,184 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 257.8 | 128.7 | 449 ms | 7.8 ms | 2.0 | 0 | 7,734 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 431.5 | 107.2 | 671 ms | 9.3 ms | 4.0 | 0 | 12,944 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 714.6 | 87.7 | 1,135 ms | 11.4 ms | 8.0 | 0 | 21,440 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,055.0 | 63.4 | 1,994 ms | 15.8 ms | 16.0 | 0 | 31,680 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,544.1 | 44.4 | 3,988 ms | 22.5 ms | 32.0 | 0 | 46,336 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,111.3 | 27.6 | 7,899 ms | 36.2 ms | 64.0 | 0 | 63,360 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,617.6 | 14.5 | 17,241 ms | 69.1 ms | 128.0 | 0 | 78,589 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 169.6 | 170.1 | 398 ms | 5.9 ms | 1.0 | 0 | 5,088 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 252.9 | 126.5 | 526 ms | 7.9 ms | 2.0 | 0 | 7,586 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 419.6 | 104.3 | 749 ms | 9.6 ms | 4.0 | 0 | 12,590 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 686.7 | 84.5 | 1,199 ms | 11.8 ms | 8.0 | 0 | 20,608 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 986.6 | 59.4 | 2,106 ms | 16.8 ms | 16.0 | 0 | 29,600 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,402.4 | 40.3 | 3,999 ms | 24.8 ms | 32.0 | 0 | 42,080 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,861.8 | 24.0 | 8,178 ms | 41.7 ms | 64.0 | 0 | 55,872 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 2,232.8 | 11.7 | 18,454 ms | 85.4 ms | 128.0 | 0 | 66,944 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 164.8 | 165.4 | 463 ms | 6.0 ms | 1.0 | 0 | 4,944 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 244.4 | 122.4 | 601 ms | 8.2 ms | 2.0 | 0 | 7,333 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 396.7 | 99.0 | 900 ms | 10.1 ms | 4.0 | 0 | 11,900 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 624.8 | 77.4 | 1,481 ms | 12.9 ms | 8.0 | 0 | 18,760 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 871.3 | 53.1 | 2,468 ms | 18.8 ms | 16.0 | 0 | 26,169 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,187.9 | 34.7 | 4,099 ms | 28.8 ms | 32.0 | 0 | 35,648 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,489.3 | 19.5 | 8,115 ms | 51.4 ms | 64.0 | 0 | 44,672 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 155.9 | 156.7 | 1,006 ms | 6.4 ms | 1.0 | 0 | 4,675 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 228.6 | 114.5 | 1,229 ms | 8.7 ms | 2.0 | 0 | 6,857 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 355.4 | 88.3 | 1,802 ms | 11.3 ms | 4.0 | 0 | 10,660 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 533.3 | 65.1 | 2,846 ms | 15.4 ms | 8.0 | 0 | 16,000 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 706.9 | 41.6 | 5,108 ms | 24.1 ms | 16.0 | 0 | 21,216 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 908.7 | 24.5 | 8,837 ms | 40.9 ms | 32.0 | 0 | 27,264 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 141.7 | 143.0 | 1,456 ms | 7.0 ms | 1.0 | 0 | 4,249 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 199.9 | 101.0 | 1,923 ms | 9.9 ms | 2.0 | 0 | 5,996 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 296.3 | 74.8 | 2,909 ms | 13.4 ms | 4.0 | 0 | 8,888 | 30.0 s |
| Sustained decode | 248K | 8 | Valid | 415.4 | 52.2 | 4,704 ms | 19.1 ms | 8.0 | 0 | 12,456 | 30.0 s |
| Sustained decode | 248K | 16 | Valid | 521.1 | 32.5 | 7,913 ms | 30.8 ms | 16.0 | 0 | 15,632 | 30.0 s |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.