Speed test matrix
Tess-4-27B-FP8 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve lflare/Tess-4-27B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}Prefill
prompt tok/s by context length8K context2,1613,769 ms TTFT
16K context2,0357,920 ms TTFT
32K context1,90116,874 ms TTFT
64K context1,66138,535 ms TTFT
128K context1,31996,983 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 52.1187 ms TTFT | 112.3364 ms TTFT | 210.2371 ms TTFT | 344.3525 ms TTFT | 627.0870 ms TTFT | 837.81,351 ms TTFT | Over capacity | Over capacity |
| 16K | 47.0639 ms TTFT | 85.71,057 ms TTFT | 139.71,970 ms TTFT | 200.13,194 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 35.1751 ms TTFT | 71.51,160 ms TTFT | 97.92,164 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 24.81,037 ms TTFT | 49.91,597 ms TTFT | 63.22,765 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 17.02,315 ms TTFT | 33.33,621 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 9.7Requests queued3,491 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,161.0 | — | 3,769 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 2,035.0 | — | 7,920 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,901.0 | — | 16,874 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,661.0 | — | 38,535 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,319.0 | — | 96,983 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 52.1 | 54.0 | 187 ms | 18.5 ms | 1.0 | 0 | 1,563 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 112.3 | 57.0 | 364 ms | 17.5 ms | 2.0 | 0 | 3,367 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 210.2 | 54.3 | 371 ms | 18.4 ms | 4.0 | 0 | 6,301 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 344.3 | 44.2 | 525 ms | 22.6 ms | 8.0 | 0 | 10,324 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 627.0 | 40.3 | 870 ms | 24.8 ms | 16.0 | 0 | 18,790 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 837.8 | 27.0 | 1,351 ms | 37.1 ms | 32.0 | 0 | 25,116 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 47.0 | 47.7 | 639 ms | 21.0 ms | 1.0 | 0 | 1,410 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 85.7 | 43.5 | 1,057 ms | 23.0 ms | 2.0 | 0 | 2,572 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 139.7 | 35.2 | 1,970 ms | 28.4 ms | 4.0 | 0 | 4,189 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 200.1 | 25.0 | 3,194 ms | 40.0 ms | 8.0 | 0 | 5,987 | 29.9 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 35.1 | 36.0 | 751 ms | 27.8 ms | 1.0 | 0 | 1,052 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 71.5 | 36.1 | 1,160 ms | 27.7 ms | 2.0 | 0 | 2,143 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 97.9 | 24.7 | 2,164 ms | 40.4 ms | 4.0 | 0 | 2,932 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 24.8 | 25.6 | 1,037 ms | 39.0 ms | 1.0 | 0 | 741 | 29.9 s |
| Sustained decode | 64K | 2 | Valid | 49.9 | 25.5 | 1,597 ms | 39.1 ms | 2.0 | 0 | 1,493 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 63.2 | 16.2 | 2,765 ms | 61.7 ms | 4.0 | 0 | 1,894 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 17.0 | 17.5 | 2,315 ms | 57.1 ms | 1.0 | 0 | 508 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 33.3 | 16.9 | 3,621 ms | 59.1 ms | 2.0 | 0 | 998 | 29.9 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 9.7 | 10.7 | 3,491 ms | 93.1 ms | 0.9 | 0 | 289 | 29.8 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.