Speed test matrix
GLM-5.3-Flash-W4A16-MTP on CMP 170HX 64GB ×4
vllm · 0.32.0rc1.dev215+g819146023 · W4A167 contexts (0–504K) × 5 concurrency levels (1–16) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- GPU count
- 4
- VRAM
- 64 GB (4 × 16 GB)
- CPU
- Intel Core i7-9800X
- RAM
- 128 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev215+g819146023
- Quantization
- W4A16
- Backend
- cuda
- Repository
- github.com/Morrowmake/vllm-cmp170hx
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:18a2f26207568128e1d792ce923e86a74e40def1e43e492db83132cab8cdbe14
Run
- Served model
- m
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# x299 recipe as deployed (not an untuned baseline). Spec decode: none. Cool room, SYSTIN 24-26 C; instantaneous power peaks up to 302 W (GA100 reports instantaneous draw); enforced limit 180 W # 4x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 per card | i7-9800X | 128 GB DDR4 | Ubuntu 26.04.1, driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve /models/GLM-5.3-Flash-W4A16-MTP --served-model-name m --host 0.0.0.0 --pipeline-parallel-size 4 --tensor-parallel-size 1 --max-model-len 524288 --max-num-seqs 8 --max-num-batched-tokens 2312 --long-prefill-token-threshold 0 --gpu-memory-utilization 0.95 --trust-remote-code --reasoning-parser glm47 --enable-auto-tool-choice --tool-call-parser glm47 --port 8000 --prefill-chunk-with-decodes 384 --max-num-partial-prefills 2 --block-size 4608
Prefill
prompt tok/s by context length8K context4,3331,892 ms TTFT
16K context5,2533,089 ms TTFT
32K context5,9725,412 ms TTFT
64K context6,40910,067 ms TTFT
128K context6,57719,596 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 54.9222 ms TTFT | 107.8294 ms TTFT | 173.0464 ms TTFT | 281.91,819 ms TTFT | 276.4Requests queued508 ms TTFT |
| 16K | 54.01,151 ms TTFT | 105.71,621 ms TTFT | 170.13,267 ms TTFT | 276.83,871 ms TTFT | 275.7Requests queued3,921 ms TTFT |
| 32K | 54.1330 ms TTFT | 105.9396 ms TTFT | 170.2533 ms TTFT | 277.1704 ms TTFT | 276.3Requests queued858 ms TTFT |
| 64K | 54.0253 ms TTFT | 105.8362 ms TTFT | 169.7497 ms TTFT | 277.2811 ms TTFT | 275.9Requests queued840 ms TTFT |
| 128K | 53.71,858 ms TTFT | 105.22,564 ms TTFT | 169.14,437 ms TTFT | 275.66,812 ms TTFT | 274.5Requests queued6,939 ms TTFT |
| 256K | 53.52,353 ms TTFT | 104.93,088 ms TTFT | 168.44,876 ms TTFT | 274.97,204 ms TTFT | Over capacity |
| 504K | 53.32,192 ms TTFT | 104.72,843 ms TTFT | 168.14,317 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,333.0 | — | 1,892 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,253.0 | — | 3,089 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,972.0 | — | 5,412 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,409.0 | — | 10,067 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,577.0 | — | 19,596 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 54.9 | 54.9 | 222 ms | 18.2 ms | 1.0 | 0 | 1,646 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 107.8 | 53.9 | 294 ms | 18.6 ms | 2.0 | 0 | 3,234 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 173.0 | 43.3 | 464 ms | 23.1 ms | 4.0 | 0 | 5,190 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 281.9 | 35.3 | 1,819 ms | 28.4 ms | 8.0 | 0 | 8,456 | 30.0 s |
| Sustained decode | 0 | 16 | Requests queued | 276.4 | 34.9 | 508 ms | 28.7 ms | 8.0 | 0 | 8,290 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 54.0 | 54.0 | 1,151 ms | 18.5 ms | 1.0 | 0 | 1,618 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 105.7 | 52.2 | 1,621 ms | 19.2 ms | 2.0 | 0 | 3,170 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 170.1 | 41.9 | 3,267 ms | 23.9 ms | 4.0 | 0 | 5,104 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 276.8 | 32.4 | 3,871 ms | 30.9 ms | 8.0 | 0 | 8,304 | 30.0 s |
| Sustained decode | 16K | 16 | Requests queued | 275.7 | 33.9 | 3,921 ms | 29.5 ms | 8.0 | 0 | 8,270 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 54.1 | 54.1 | 330 ms | 18.5 ms | 1.0 | 0 | 1,623 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 105.9 | 52.9 | 396 ms | 18.9 ms | 2.0 | 0 | 3,177 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 170.2 | 42.4 | 533 ms | 23.6 ms | 4.0 | 0 | 5,107 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 277.1 | 34.5 | 704 ms | 29.0 ms | 8.0 | 0 | 8,313 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 276.3 | 34.5 | 858 ms | 29.0 ms | 8.0 | 0 | 8,289 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 54.0 | 54.1 | 253 ms | 18.5 ms | 1.0 | 0 | 1,619 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 105.8 | 53.0 | 362 ms | 18.9 ms | 2.0 | 0 | 3,173 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 169.7 | 42.5 | 497 ms | 23.5 ms | 4.0 | 0 | 5,090 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 277.2 | 34.7 | 811 ms | 28.8 ms | 8.0 | 0 | 8,316 | 30.0 s |
| Sustained decode | 64K | 16 | Requests queued | 275.9 | 34.6 | 840 ms | 28.9 ms | 8.0 | 0 | 8,277 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 53.7 | 53.9 | 1,858 ms | 18.6 ms | 1.0 | 0 | 1,611 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 105.2 | 51.8 | 2,564 ms | 19.3 ms | 2.0 | 0 | 3,157 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 169.1 | 41.1 | 4,437 ms | 24.4 ms | 4.0 | 0 | 5,073 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 275.6 | 29.7 | 6,812 ms | 33.6 ms | 8.0 | 0 | 8,267 | 30.0 s |
| Sustained decode | 128K | 16 | Requests queued | 274.5 | 33.8 | 6,939 ms | 29.6 ms | 8.0 | 0 | 8,236 | 30.0 s |
| Sustained decode | 256K | 1 | Valid | 53.5 | 53.8 | 2,353 ms | 18.6 ms | 1.0 | 0 | 1,606 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 104.9 | 51.8 | 3,088 ms | 19.3 ms | 2.0 | 0 | 3,148 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 168.4 | 41.0 | 4,876 ms | 24.4 ms | 4.0 | 0 | 5,052 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 274.9 | 31.5 | 7,204 ms | 31.7 ms | 8.0 | 0 | 8,247 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 504K | 1 | Valid | 53.3 | 53.8 | 2,192 ms | 18.6 ms | 1.0 | 0 | 1,600 | 30.0 s |
| Sustained decode | 504K | 2 | Valid | 104.7 | 52.6 | 2,843 ms | 19.0 ms | 2.0 | 0 | 3,138 | 30.0 s |
| Sustained decode | 504K | 4 | Valid | 168.1 | 42.3 | 4,317 ms | 23.6 ms | 4.0 | 0 | 5,044 | 30.0 s |
| Sustained decode | 504K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 504K | 16 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.