Speed test matrix
Trinity-Mini-FP8-Block on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP8 block5 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8 block
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve arcee-ai/Trinity-Mini-FP8-Block --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context10.3K768 ms TTFT
16K context9,7741,599 ms TTFT
32K context8,9553,475 ms TTFT
64K context7,8907,873 ms TTFT
120K context6,26218,579 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 129.662 ms TTFT | 219.7106 ms TTFT | 352.499 ms TTFT | 535.9159 ms TTFT | 808.8198 ms TTFT | 1,195.3311 ms TTFT | 1,727.3488 ms TTFT | 2,422.8735 ms TTFT |
| 16K | 124.0196 ms TTFT | 206.1257 ms TTFT | 319.2349 ms TTFT | 454.7445 ms TTFT | 612.0897 ms TTFT | 777.21,519 ms TTFT | 934.42,645 ms TTFT | Over capacity |
| 32K | 122.1249 ms TTFT | 198.4404 ms TTFT | 303.3566 ms TTFT | 422.7896 ms TTFT | 558.11,578 ms TTFT | 682.62,954 ms TTFT | Over capacity | Over capacity |
| 64K | 115.0437 ms TTFT | 183.6710 ms TTFT | 271.21,052 ms TTFT | 367.71,823 ms TTFT | 468.32,638 ms TTFT | Over capacity | Over capacity | Over capacity |
| 120K | 107.1853 ms TTFT | 162.71,293 ms TTFT | 230.32,041 ms TTFT | 298.03,449 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.3K | — | 768 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,774.0 | — | 1,599 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,955.0 | — | 3,475 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,890.0 | — | 7,873 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 6,262.0 | — | 18,579 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 129.6 | 132.5 | 62 ms | 7.5 ms | 1.0 | 0 | 3,889 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 219.7 | 112.2 | 106 ms | 8.9 ms | 2.0 | 0 | 6,590 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 352.4 | 91.1 | 99 ms | 11.0 ms | 4.0 | 0 | 10,572 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 535.9 | 70.6 | 159 ms | 14.2 ms | 8.0 | 0 | 16,078 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 808.8 | 54.2 | 198 ms | 18.4 ms | 16.0 | 0 | 24,272 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,195.3 | 40.6 | 311 ms | 24.6 ms | 32.0 | 0 | 35,840 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,727.3 | 29.7 | 488 ms | 33.7 ms | 64.0 | 0 | 51,840 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 2,422.8 | 21.2 | 735 ms | 47.2 ms | 128.0 | 0 | 72,704 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 124.0 | 124.3 | 196 ms | 8.0 ms | 1.0 | 0 | 3,721 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 206.1 | 103.3 | 257 ms | 9.7 ms | 2.0 | 0 | 6,182 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 319.2 | 80.0 | 349 ms | 12.5 ms | 4.0 | 0 | 9,576 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 454.7 | 57.1 | 445 ms | 17.5 ms | 8.0 | 0 | 13,648 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 612.0 | 38.4 | 897 ms | 26.1 ms | 16.0 | 0 | 18,368 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 777.2 | 24.3 | 1,519 ms | 41.2 ms | 32.0 | 0 | 23,296 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 934.4 | 14.5 | 2,645 ms | 69.1 ms | 64.0 | 0 | 28,032 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 122.1 | 122.5 | 249 ms | 8.2 ms | 1.0 | 0 | 3,662 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 198.4 | 99.4 | 404 ms | 10.1 ms | 2.0 | 0 | 5,952 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 303.3 | 75.9 | 566 ms | 13.2 ms | 4.0 | 0 | 9,096 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 422.7 | 53.0 | 896 ms | 18.9 ms | 8.0 | 0 | 12,680 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 558.1 | 35.0 | 1,578 ms | 28.6 ms | 16.0 | 0 | 16,752 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 682.6 | 21.3 | 2,954 ms | 46.9 ms | 32.0 | 0 | 20,480 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 115.0 | 115.2 | 437 ms | 8.7 ms | 1.0 | 0 | 3,449 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 183.6 | 92.1 | 710 ms | 10.9 ms | 2.0 | 0 | 5,508 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 271.2 | 68.0 | 1,052 ms | 14.7 ms | 4.0 | 0 | 8,136 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 367.7 | 46.0 | 1,823 ms | 21.7 ms | 8.0 | 0 | 11,032 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 468.3 | 29.2 | 2,638 ms | 34.2 ms | 16.0 | 0 | 14,048 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 107.1 | 107.5 | 853 ms | 9.3 ms | 1.0 | 0 | 3,214 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 162.7 | 81.8 | 1,293 ms | 12.2 ms | 2.0 | 0 | 4,881 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 230.3 | 57.8 | 2,041 ms | 17.3 ms | 4.0 | 0 | 6,908 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 298.0 | 37.4 | 3,449 ms | 26.8 ms | 8.0 | 0 | 8,936 | 30.0 s |
| Sustained decode | 120K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.