Speed test matrix
gemma-4-31B-it-qat-w4a16-ct on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · W4A166 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- W4A16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve google/gemma-4-31B-it-qat-w4a16-ct --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only
Prefill
prompt tok/s by context length8K context1,3685,827 ms TTFT
16K context1,15113,705 ms TTFT
32K context921.034,083 ms TTFT
64K context659.095,057 ms TTFT
128K context419.0298,640 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 42.993 ms TTFT | 82.9913 ms TTFT | 154.7157 ms TTFT | 271.9200 ms TTFT | 451.7302 ms TTFT | 683.8478 ms TTFT | Over capacity | Over capacity |
| 16K | 37.1201 ms TTFT | 69.4314 ms TTFT | 126.5403 ms TTFT | 207.3600 ms TTFT | 300.01,008 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 34.0392 ms TTFT | 62.4464 ms TTFT | 109.5660 ms TTFT | 174.31,048 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 30.6603 ms TTFT | 54.6857 ms TTFT | 89.91,161 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 25.1Requests queued1,395 ms TTFT | 42.2Requests queued1,760 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 1,368.0 | — | 5,827 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 1,151.0 | — | 13,705 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 921.0 | — | 34,083 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 659.0 | — | 95,057 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 419.0 | — | 298,640 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 42.9 | 43.2 | 93 ms | 23.1 ms | 1.0 | 0 | 1,288 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 82.9 | 41.8 | 913 ms | 23.9 ms | 2.0 | 0 | 2,486 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 154.7 | 39.3 | 157 ms | 25.4 ms | 4.0 | 0 | 4,640 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 271.9 | 34.9 | 200 ms | 28.6 ms | 8.0 | 0 | 8,152 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 451.7 | 29.4 | 302 ms | 34.1 ms | 16.0 | 0 | 13,552 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 683.8 | 22.4 | 478 ms | 44.7 ms | 32.0 | 0 | 20,512 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 37.1 | 37.2 | 201 ms | 26.9 ms | 1.0 | 0 | 1,114 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 69.4 | 34.7 | 314 ms | 28.8 ms | 2.0 | 0 | 2,080 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 126.5 | 31.7 | 403 ms | 31.6 ms | 4.0 | 0 | 3,792 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 207.3 | 25.9 | 600 ms | 38.6 ms | 8.0 | 0 | 6,216 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 300.0 | 18.7 | 1,008 ms | 53.4 ms | 16.0 | 0 | 8,992 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 34.0 | 34.1 | 392 ms | 29.3 ms | 1.0 | 0 | 1,019 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 62.4 | 31.3 | 464 ms | 31.9 ms | 2.0 | 0 | 1,871 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 109.5 | 27.5 | 660 ms | 36.4 ms | 4.0 | 0 | 3,284 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 174.3 | 21.9 | 1,048 ms | 45.7 ms | 8.0 | 0 | 5,226 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 30.6 | 30.8 | 603 ms | 32.5 ms | 1.0 | 0 | 919 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 54.6 | 27.4 | 857 ms | 36.5 ms | 2.0 | 0 | 1,638 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 89.9 | 22.6 | 1,161 ms | 44.3 ms | 4.0 | 0 | 2,696 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Requests queued | 25.1 | 25.8 | 1,395 ms | 38.7 ms | 1.0 | 0 | 752 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 42.2 | 22.0 | 1,760 ms | 45.4 ms | 1.9 | 0 | 1,267 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Over capacity | — | — | — | — | 0.6 | 0 | 0 | 30.0 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.