Speed test matrix
Qwen3.5-9B-AWQ-4bit on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · W4A16 g326 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- W4A16 g32
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve cyankiwi/Qwen3.5-9B-AWQ-4bit --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only
Prefill
prompt tok/s by context length8K context6,2311,307 ms TTFT
16K context5,9332,717 ms TTFT
32K context5,6075,722 ms TTFT
64K context4,95312,925 ms TTFT
128K context3,99132,045 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 146.4114 ms TTFT | 279.5153 ms TTFT | 529.5177 ms TTFT | 960.2250 ms TTFT | 1,571.1373 ms TTFT | 2,364.4580 ms TTFT | 3,083.7906 ms TTFT | 3,715.51,388 ms TTFT |
| 16K | 136.2211 ms TTFT | 247.4290 ms TTFT | 434.4413 ms TTFT | 701.1629 ms TTFT | 993.8993 ms TTFT | 1,272.71,862 ms TTFT | 1,435.63,760 ms TTFT | Over capacity |
| 32K | 127.7322 ms TTFT | 222.3401 ms TTFT | 368.3529 ms TTFT | 549.9798 ms TTFT | 722.51,303 ms TTFT | 862.32,619 ms TTFT | Over capacity | Over capacity |
| 64K | 113.8483 ms TTFT | 184.8656 ms TTFT | 281.4886 ms TTFT | 383.31,401 ms TTFT | 463.12,424 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 93.6898 ms TTFT | 137.5983 ms TTFT | 190.71,536 ms TTFT | 237.52,532 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 69.71,689 ms TTFT | 92.92,003 ms TTFT | 118.63,121 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,231.0 | — | 1,307 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,933.0 | — | 2,717 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,607.0 | — | 5,722 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,953.0 | — | 12,925 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,991.0 | — | 32,045 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 146.4 | 146.7 | 114 ms | 6.8 ms | 1.0 | 0 | 4,391 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 279.5 | 140.4 | 153 ms | 7.1 ms | 2.0 | 0 | 8,386 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 529.5 | 133.4 | 177 ms | 7.5 ms | 4.0 | 0 | 15,896 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 960.2 | 121.4 | 250 ms | 8.2 ms | 8.0 | 0 | 28,802 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,571.1 | 99.7 | 373 ms | 10.0 ms | 16.0 | 0 | 47,136 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,364.4 | 75.0 | 580 ms | 13.3 ms | 32.0 | 0 | 70,944 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 3,083.7 | 48.7 | 906 ms | 20.5 ms | 64.0 | 0 | 92,544 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,715.5 | 28.8 | 1,388 ms | 34.7 ms | 128.0 | 0 | 111,488 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 136.2 | 136.5 | 211 ms | 7.3 ms | 1.0 | 0 | 4,085 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 247.4 | 124.0 | 290 ms | 8.1 ms | 2.0 | 0 | 7,422 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 434.4 | 108.7 | 413 ms | 9.2 ms | 4.0 | 0 | 13,032 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 701.1 | 87.5 | 629 ms | 11.4 ms | 8.0 | 0 | 21,040 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 993.8 | 61.6 | 993 ms | 16.2 ms | 16.0 | 0 | 29,824 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,272.7 | 38.8 | 1,862 ms | 25.8 ms | 32.0 | 0 | 38,176 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,435.6 | 20.9 | 3,760 ms | 47.7 ms | 64.0 | 0 | 43,072 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 127.7 | 128.1 | 322 ms | 7.8 ms | 1.0 | 0 | 3,830 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 222.3 | 111.5 | 401 ms | 9.0 ms | 2.0 | 0 | 6,670 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 368.3 | 92.0 | 529 ms | 10.9 ms | 4.0 | 0 | 11,048 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 549.9 | 68.3 | 798 ms | 14.6 ms | 8.0 | 0 | 16,496 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 722.5 | 44.2 | 1,303 ms | 22.6 ms | 16.0 | 0 | 21,664 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 862.3 | 25.6 | 2,619 ms | 39.1 ms | 32.0 | 0 | 25,856 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 113.8 | 114.3 | 483 ms | 8.8 ms | 1.0 | 0 | 3,414 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 184.8 | 92.7 | 656 ms | 10.8 ms | 2.0 | 0 | 5,544 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 281.4 | 70.4 | 886 ms | 14.2 ms | 4.0 | 0 | 8,440 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 383.3 | 47.7 | 1,401 ms | 21.0 ms | 8.0 | 0 | 11,496 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 463.1 | 28.6 | 2,424 ms | 34.9 ms | 16.0 | 0 | 13,888 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 93.6 | 94.0 | 898 ms | 10.6 ms | 1.0 | 0 | 2,807 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 137.5 | 69.1 | 983 ms | 14.5 ms | 2.0 | 0 | 4,125 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 190.7 | 47.8 | 1,536 ms | 20.9 ms | 4.0 | 0 | 5,720 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 237.5 | 29.6 | 2,532 ms | 33.8 ms | 8.0 | 0 | 7,120 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 69.7 | 70.2 | 1,689 ms | 14.2 ms | 1.0 | 0 | 2,091 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 92.9 | 46.8 | 2,003 ms | 21.4 ms | 2.0 | 0 | 2,786 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 118.6 | 29.9 | 3,121 ms | 33.4 ms | 4.0 | 0 | 3,556 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.