Speed test matrix
NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP8 dynamic5 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8 dynamic
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve RedHatAI/NVIDIA-Nemotron-Nano-9B-v2-FP8-dynamic --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context6,3071,323 ms TTFT
16K context6,0652,721 ms TTFT
32K context5,9135,557 ms TTFT
64K context5,51711,885 ms TTFT
120K context4,90225,053 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 115.485 ms TTFT | 216.9190 ms TTFT | 401.3172 ms TTFT | 706.4251 ms TTFT | 1,148.2379 ms TTFT | 1,449.4553 ms TTFT | 1,858.0990 ms TTFT | 2,101.81,307 ms TTFT |
| 16K | 112.0261 ms TTFT | 207.7327 ms TTFT | 373.1479 ms TTFT | 627.6776 ms TTFT | 957.11,406 ms TTFT | 1,154.52,816 ms TTFT | 1,392.95,724 ms TTFT | Over capacity |
| 32K | 108.6275 ms TTFT | 198.4369 ms TTFT | 346.7546 ms TTFT | 560.5882 ms TTFT | 819.61,478 ms TTFT | 960.82,746 ms TTFT | Over capacity | Over capacity |
| 64K | 103.6616 ms TTFT | 182.2764 ms TTFT | 302.71,110 ms TTFT | 460.51,726 ms TTFT | 633.73,000 ms TTFT | 714.95,831 ms TTFT | Over capacity | Over capacity |
| 120K | 95.7961 ms TTFT | 159.01,132 ms TTFT | 248.11,615 ms TTFT | 351.92,536 ms TTFT | 455.34,592 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,307.0 | — | 1,323 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,065.0 | — | 2,721 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,913.0 | — | 5,557 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,517.0 | — | 11,885 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 4,902.0 | — | 25,053 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 115.4 | 115.6 | 85 ms | 8.7 ms | 1.0 | 0 | 3,461 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 216.9 | 108.7 | 190 ms | 9.2 ms | 2.0 | 0 | 6,508 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 401.3 | 100.6 | 172 ms | 9.9 ms | 4.0 | 0 | 12,036 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 706.4 | 88.6 | 251 ms | 11.3 ms | 8.0 | 0 | 21,192 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,148.2 | 72.1 | 379 ms | 13.9 ms | 16.0 | 0 | 34,448 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,449.4 | 45.4 | 553 ms | 22.0 ms | 32.0 | 0 | 43,488 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,858.0 | 29.0 | 990 ms | 34.5 ms | 64.0 | 0 | 55,744 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 2,101.8 | 16.2 | 1,307 ms | 61.8 ms | 128.0 | 0 | 63,104 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 112.0 | 112.2 | 261 ms | 8.9 ms | 1.0 | 0 | 3,358 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 207.7 | 103.9 | 327 ms | 9.6 ms | 2.0 | 0 | 6,230 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 373.1 | 93.0 | 479 ms | 10.7 ms | 4.0 | 0 | 11,192 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 627.6 | 77.7 | 776 ms | 12.9 ms | 8.0 | 0 | 18,824 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 957.1 | 58.3 | 1,406 ms | 17.1 ms | 16.0 | 0 | 28,736 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,154.5 | 34.1 | 2,816 ms | 29.3 ms | 32.0 | 0 | 34,624 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,392.9 | 19.6 | 5,724 ms | 51.1 ms | 64.0 | 0 | 41,792 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 108.6 | 108.9 | 275 ms | 9.2 ms | 1.0 | 0 | 3,259 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 198.4 | 99.5 | 369 ms | 10.1 ms | 2.0 | 0 | 5,952 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 346.7 | 86.8 | 546 ms | 11.5 ms | 4.0 | 0 | 10,400 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 560.5 | 70.1 | 882 ms | 14.3 ms | 8.0 | 0 | 16,816 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 819.6 | 51.1 | 1,478 ms | 19.6 ms | 16.0 | 0 | 24,592 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 960.8 | 29.9 | 2,746 ms | 33.4 ms | 32.0 | 0 | 28,832 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 103.6 | 103.8 | 616 ms | 9.6 ms | 1.0 | 0 | 3,107 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 182.2 | 91.1 | 764 ms | 11.0 ms | 2.0 | 0 | 5,464 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 302.7 | 75.1 | 1,110 ms | 13.3 ms | 4.0 | 0 | 9,080 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 460.5 | 56.3 | 1,726 ms | 17.8 ms | 8.0 | 0 | 13,808 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 633.7 | 37.4 | 3,000 ms | 26.7 ms | 16.0 | 0 | 19,008 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 714.9 | 19.8 | 5,831 ms | 50.5 ms | 32.0 | 0 | 21,440 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 95.7 | 96.1 | 961 ms | 10.4 ms | 1.0 | 0 | 2,869 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 159.0 | 79.7 | 1,132 ms | 12.6 ms | 2.0 | 0 | 4,768 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 248.1 | 61.8 | 1,615 ms | 16.2 ms | 4.0 | 0 | 7,440 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 351.9 | 43.3 | 2,536 ms | 23.1 ms | 8.0 | 0 | 10,552 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 455.3 | 27.4 | 4,592 ms | 36.5 ms | 16.0 | 0 | 13,648 | 30.0 s |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.