Speed test matrix
Tess-4.5-27B-FP8 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve vwdubb/Tess-4.5-27B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only
Prefill
prompt tok/s by context length8K context2,1073,886 ms TTFT
16K context1,9888,128 ms TTFT
32K context1,86617,221 ms TTFT
64K context1,62939,331 ms TTFT
128K context1,30697,939 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 37.5175 ms TTFT | 73.2265 ms TTFT | 141.3367 ms TTFT | 269.1627 ms TTFT | 489.71,043 ms TTFT | 672.71,524 ms TTFT | Over capacity | Over capacity |
| 16K | 36.0409 ms TTFT | 68.5571 ms TTFT | 126.3895 ms TTFT | 219.91,529 ms TTFT | 350.12,913 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 34.9654 ms TTFT | 64.6962 ms TTFT | 113.91,533 ms TTFT | 185.72,692 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 32.8866 ms TTFT | 57.81,152 ms TTFT | 95.11,731 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 29.3948 ms TTFT | 47.51,327 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 22.6Requests queued2,740 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,107.0 | — | 3,886 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 1,988.0 | — | 8,128 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,866.0 | — | 17,221 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,629.0 | — | 39,331 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,306.0 | — | 97,939 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 37.5 | 37.5 | 175 ms | 26.7 ms | 1.0 | 0 | 1,124 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 73.2 | 36.7 | 265 ms | 27.3 ms | 2.0 | 0 | 2,196 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 141.3 | 35.4 | 367 ms | 28.3 ms | 4.0 | 0 | 4,240 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 269.1 | 33.7 | 627 ms | 29.7 ms | 8.0 | 0 | 8,072 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 489.7 | 30.7 | 1,043 ms | 32.6 ms | 16.0 | 0 | 14,688 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 672.7 | 20.8 | 1,524 ms | 48.1 ms | 32.0 | 0 | 20,192 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 36.0 | 36.1 | 409 ms | 27.7 ms | 1.0 | 0 | 1,080 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 68.5 | 34.2 | 571 ms | 29.3 ms | 2.0 | 0 | 2,056 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 126.3 | 31.2 | 895 ms | 32.0 ms | 4.0 | 0 | 3,788 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 219.9 | 26.8 | 1,529 ms | 37.3 ms | 8.0 | 0 | 6,592 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 350.1 | 20.6 | 2,913 ms | 48.5 ms | 16.0 | 0 | 10,496 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 34.9 | 34.9 | 654 ms | 28.6 ms | 1.0 | 0 | 1,045 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 64.6 | 32.1 | 962 ms | 31.2 ms | 2.0 | 0 | 1,936 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 113.9 | 27.9 | 1,533 ms | 35.9 ms | 4.0 | 0 | 3,416 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 185.7 | 22.1 | 2,692 ms | 45.3 ms | 8.0 | 0 | 5,568 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 32.8 | 32.9 | 866 ms | 30.4 ms | 1.0 | 0 | 984 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 57.8 | 28.8 | 1,152 ms | 34.8 ms | 2.0 | 0 | 1,732 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 95.1 | 23.4 | 1,731 ms | 42.8 ms | 4.0 | 0 | 2,852 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 29.3 | 29.4 | 948 ms | 34.1 ms | 1.0 | 0 | 877 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 47.5 | 23.8 | 1,327 ms | 42.1 ms | 2.0 | 0 | 1,426 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 22.6 | 24.3 | 2,740 ms | 41.1 ms | 0.9 | 0 | 677 | 30.0 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.