Speed test matrix
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve RedHatAI/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context8,877941 ms TTFT
16K context8,6991,898 ms TTFT
32K context8,2943,963 ms TTFT
64K context7,4478,806 ms TTFT
128K context6,10321,463 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 173.894 ms TTFT | Looped | 436.3199 ms TTFT | 653.8281 ms TTFT | 936.8330 ms TTFT | 1,359.4523 ms TTFT | 1,995.0721 ms TTFT | 2,729.11,109 ms TTFT |
| 16K | Looped | 267.5454 ms TTFT | 407.5683 ms TTFT | 617.41,164 ms TTFT | 877.22,059 ms TTFT | 1,249.04,081 ms TTFT | 1,744.88,215 ms TTFT | 2,272.017,955 ms TTFT |
| 32K | 168.4384 ms TTFT | 262.6487 ms TTFT | 399.5737 ms TTFT | 590.91,223 ms TTFT | 833.42,129 ms TTFT | 1,160.34,105 ms TTFT | 1,561.18,457 ms TTFT | Over capacity |
| 64K | 163.9439 ms TTFT | 252.5631 ms TTFT | 377.7880 ms TTFT | 543.51,472 ms TTFT | 744.82,648 ms TTFT | 1,010.14,431 ms TTFT | Over capacity | Over capacity |
| 128K | Looped | 234.91,217 ms TTFT | 343.61,798 ms TTFT | 475.52,829 ms TTFT | 624.44,576 ms TTFT | Over capacity | Over capacity | Over capacity |
| 248K | 141.31,354 ms TTFT | 205.41,843 ms TTFT | 287.42,738 ms TTFT | 379.04,360 ms TTFT | 472.98,037 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,877.0 | — | 941 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,699.0 | — | 1,898 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,294.0 | — | 3,963 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,447.0 | — | 8,806 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,103.0 | — | 21,463 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 173.8 | 173.9 | 94 ms | 5.8 ms | 1.0 | 0 | 5,213 | 30.0 s |
| Sustained decode | 0 | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 0 | 4 | Valid | 436.3 | 109.7 | 199 ms | 9.1 ms | 4.0 | 0 | 13,088 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 653.8 | 83.1 | 281 ms | 12.0 ms | 8.0 | 0 | 19,616 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 936.8 | 60.2 | 330 ms | 16.6 ms | 16.0 | 0 | 28,096 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,359.4 | 44.0 | 523 ms | 22.7 ms | 32.0 | 0 | 40,768 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,995.0 | 32.2 | 721 ms | 31.1 ms | 64.0 | 0 | 59,840 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 2,729.1 | 21.7 | 1,109 ms | 46.1 ms | 128.0 | 0 | 81,920 | 30.0 s |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Valid | 267.5 | 133.6 | 454 ms | 7.5 ms | 2.0 | 0 | 8,026 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 407.5 | 101.5 | 683 ms | 9.9 ms | 4.0 | 0 | 12,224 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 617.4 | 76.0 | 1,164 ms | 13.2 ms | 8.0 | 0 | 18,520 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 877.2 | 52.9 | 2,059 ms | 18.9 ms | 16.0 | 0 | 26,336 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,249.0 | 36.0 | 4,081 ms | 27.8 ms | 32.0 | 0 | 37,472 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,744.8 | 22.9 | 8,215 ms | 43.6 ms | 64.0 | 0 | 52,352 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,272.0 | 12.8 | 17,955 ms | 78.4 ms | 128.0 | 0 | 68,352 | 30.1 s |
| Sustained decode | 32K | 1 | Valid | 168.4 | 168.7 | 384 ms | 5.9 ms | 1.0 | 0 | 5,053 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 262.6 | 131.4 | 487 ms | 7.6 ms | 2.0 | 0 | 7,878 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 399.5 | 99.5 | 737 ms | 10.0 ms | 4.0 | 0 | 11,984 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 590.9 | 72.9 | 1,223 ms | 13.7 ms | 8.0 | 0 | 17,728 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 833.4 | 50.3 | 2,129 ms | 19.9 ms | 16.0 | 0 | 25,024 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,160.3 | 33.4 | 4,105 ms | 30.0 ms | 32.0 | 0 | 34,816 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,561.1 | 20.3 | 8,457 ms | 49.4 ms | 64.0 | 0 | 46,848 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 163.9 | 164.4 | 439 ms | 6.1 ms | 1.0 | 0 | 4,916 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 252.5 | 126.5 | 631 ms | 7.9 ms | 2.0 | 0 | 7,575 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 377.7 | 94.4 | 880 ms | 10.6 ms | 4.0 | 0 | 11,328 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 543.5 | 67.5 | 1,472 ms | 14.8 ms | 8.0 | 0 | 16,304 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 744.8 | 45.6 | 2,648 ms | 21.9 ms | 16.0 | 0 | 22,352 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,010.1 | 29.6 | 4,431 ms | 33.8 ms | 32.0 | 0 | 30,304 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Valid | 234.9 | 117.7 | 1,217 ms | 8.5 ms | 2.0 | 0 | 7,045 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 343.6 | 85.4 | 1,798 ms | 11.7 ms | 4.0 | 0 | 10,308 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 475.5 | 58.2 | 2,829 ms | 17.2 ms | 8.0 | 0 | 14,264 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 624.4 | 36.7 | 4,576 ms | 27.2 ms | 16.0 | 0 | 18,736 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 141.3 | 142.5 | 1,354 ms | 7.0 ms | 1.0 | 0 | 4,238 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 205.4 | 103.8 | 1,843 ms | 9.6 ms | 2.0 | 0 | 6,160 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 287.4 | 72.6 | 2,738 ms | 13.8 ms | 4.0 | 0 | 8,624 | 30.0 s |
| Sustained decode | 248K | 8 | Valid | 379.0 | 47.6 | 4,360 ms | 21.0 ms | 8.0 | 0 | 11,368 | 30.0 s |
| Sustained decode | 248K | 16 | Valid | 472.9 | 29.4 | 8,037 ms | 34.0 ms | 16.0 | 0 | 14,176 | 30.0 s |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.