Speed test matrix
Fara1.5-27B-AWQ-W4A16 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · AWQ W4A166 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- AWQ W4A16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve Sohailhosseini/Fara1.5-27B-AWQ-W4A16 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only
Prefill
prompt tok/s by context length8K context1,9674,140 ms TTFT
16K context1,8618,661 ms TTFT
32K context1,74618,376 ms TTFT
64K context1,54641,418 ms TTFT
128K context1,253102,096 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 55.9179 ms TTFT | 107.7302 ms TTFT | 205.5305 ms TTFT | 382.9503 ms TTFT | 612.2860 ms TTFT | 936.11,474 ms TTFT | 1,170.92,721 ms TTFT | Over capacity |
| 16K | 52.9381 ms TTFT | 97.5551 ms TTFT | 174.5875 ms TTFT | Looped | 415.02,799 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 50.3686 ms TTFT | 89.2979 ms TTFT | 151.81,529 ms TTFT | 236.82,625 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | Looped | 76.31,103 ms TTFT | 120.51,660 ms TTFT | 171.22,749 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | Looped | 59.11,228 ms TTFT | 85.11,644 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 28.3Requests queued2,718 ms TTFT | 37.3Requests queued3,407 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 1,967.0 | — | 4,140 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 1,861.0 | — | 8,661 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,746.0 | — | 18,376 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,546.0 | — | 41,418 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,253.0 | — | 102,096 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 55.9 | 56.0 | 179 ms | 17.9 ms | 1.0 | 0 | 1,677 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 107.7 | 54.0 | 302 ms | 18.5 ms | 2.0 | 0 | 3,230 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 205.5 | 51.5 | 305 ms | 19.4 ms | 4.0 | 0 | 6,164 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 382.9 | 48.0 | 503 ms | 20.8 ms | 8.0 | 0 | 11,488 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 612.2 | 38.4 | 860 ms | 26.0 ms | 16.0 | 0 | 18,368 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 936.1 | 29.3 | 1,474 ms | 34.1 ms | 32.0 | 0 | 28,114 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,170.9 | 18.3 | 2,721 ms | 54.8 ms | 64.0 | 0 | 35,136 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 52.9 | 53.0 | 381 ms | 18.9 ms | 1.0 | 0 | 1,588 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 97.5 | 48.7 | 551 ms | 20.6 ms | 2.0 | 0 | 2,924 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 174.5 | 43.2 | 875 ms | 23.1 ms | 4.0 | 0 | 5,232 | 30.0 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Valid | 415.0 | 24.5 | 2,799 ms | 40.9 ms | 16.0 | 0 | 12,449 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 50.3 | 50.4 | 686 ms | 19.8 ms | 1.0 | 0 | 1,509 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 89.2 | 44.4 | 979 ms | 22.5 ms | 2.0 | 0 | 2,676 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 151.8 | 37.2 | 1,529 ms | 26.9 ms | 4.0 | 0 | 4,552 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 236.8 | 28.2 | 2,625 ms | 35.5 ms | 8.0 | 0 | 7,104 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 2 | Valid | 76.3 | 38.0 | 1,103 ms | 26.3 ms | 2.0 | 0 | 2,288 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 120.5 | 29.6 | 1,660 ms | 33.8 ms | 4.0 | 0 | 3,611 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 171.2 | 20.4 | 2,749 ms | 49.0 ms | 8.0 | 0 | 5,136 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Valid | 59.1 | 29.6 | 1,228 ms | 33.7 ms | 2.0 | 0 | 1,772 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 85.1 | 21.2 | 1,644 ms | 47.2 ms | 4.0 | 0 | 2,552 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 28.3 | 30.5 | 2,718 ms | 32.8 ms | 0.9 | 0 | 848 | 30.0 s |
| Sustained decode | 248K | 2 | Requests queued | 37.3 | 20.6 | 3,407 ms | 48.5 ms | 1.8 | 0 | 1,118 | 30.0 s |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.