Speed test matrix
Qwen3.5-27B-AWQ-4bit on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · AWQ 4-bit6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- AWQ 4-bit
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve cyankiwi/Qwen3.5-27B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}Prefill
prompt tok/s by context length8K context1,9214,240 ms TTFT
16K context1,8128,897 ms TTFT
32K context1,69618,913 ms TTFT
64K context1,50042,668 ms TTFT
128K context1,213105,432 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 100.3186 ms TTFT | 187.4389 ms TTFT | 320.8371 ms TTFT | 495.5569 ms TTFT | 741.9947 ms TTFT | 849.31,511 ms TTFT | 1,001.02,891 ms TTFT | Over capacity |
| 16K | 69.5664 ms TTFT | 129.91,124 ms TTFT | 178.12,134 ms TTFT | 258.23,629 ms TTFT | 306.65,920 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 51.7765 ms TTFT | 100.21,282 ms TTFT | 123.32,418 ms TTFT | 170.94,072 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 35.11,054 ms TTFT | 65.71,605 ms TTFT | 76.12,973 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 21.02,279 ms TTFT | 39.33,803 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 10.8Requests queued3,594 ms TTFT | 20.7Requests queued5,267 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 1,921.0 | — | 4,240 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 1,812.0 | — | 8,897 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,696.0 | — | 18,913 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,500.0 | — | 42,668 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,213.0 | — | 105,432 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 100.3 | 100.6 | 186 ms | 9.9 ms | 1.0 | 0 | 3,010 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 187.4 | 95.1 | 389 ms | 10.5 ms | 2.0 | 0 | 5,618 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 320.8 | 81.0 | 371 ms | 12.3 ms | 4.0 | 0 | 9,619 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 495.5 | 63.2 | 569 ms | 15.8 ms | 8.0 | 0 | 14,848 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 741.9 | 47.1 | 947 ms | 21.2 ms | 16.0 | 0 | 22,256 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 849.3 | 27.2 | 1,511 ms | 36.8 ms | 32.0 | 0 | 25,432 | 29.9 s |
| Sustained decode | 0 | 64 | Valid | 1,001.0 | 16.0 | 2,891 ms | 62.5 ms | 64.0 | 0 | 30,031 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 69.5 | 70.1 | 664 ms | 14.3 ms | 1.0 | 0 | 2,084 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 129.9 | 65.5 | 1,124 ms | 15.3 ms | 2.0 | 0 | 3,896 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 178.1 | 44.5 | 2,134 ms | 22.4 ms | 4.0 | 0 | 5,339 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 258.2 | 31.6 | 3,629 ms | 31.7 ms | 8.0 | 0 | 7,740 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 306.6 | 18.0 | 5,920 ms | 55.5 ms | 16.0 | 0 | 9,165 | 29.9 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 51.7 | 52.3 | 765 ms | 19.1 ms | 1.0 | 0 | 1,551 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 100.2 | 50.4 | 1,282 ms | 19.8 ms | 2.0 | 0 | 3,004 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 123.3 | 31.2 | 2,418 ms | 32.1 ms | 4.0 | 0 | 3,690 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 170.9 | 21.1 | 4,072 ms | 47.4 ms | 8.0 | 0 | 5,111 | 29.9 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 35.1 | 35.5 | 1,054 ms | 28.2 ms | 1.0 | 0 | 1,051 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 65.7 | 33.3 | 1,605 ms | 30.1 ms | 2.0 | 0 | 1,968 | 29.9 s |
| Sustained decode | 64K | 4 | Valid | 76.1 | 19.1 | 2,973 ms | 52.4 ms | 4.0 | 0 | 2,280 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 21.0 | 21.4 | 2,279 ms | 46.8 ms | 1.0 | 0 | 629 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 39.3 | 19.8 | 3,803 ms | 50.5 ms | 2.0 | 0 | 1,177 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 10.8 | 12.0 | 3,594 ms | 83.6 ms | 0.9 | 0 | 323 | 30.0 s |
| Sustained decode | 248K | 2 | Requests queued | 20.7 | 12.3 | 5,267 ms | 81.7 ms | 1.7 | 0 | 618 | 29.9 s |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.