Speed test matrix
gemma-4-12B-it-FP8-Dynamic on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP8 dynamic6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8 dynamic
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve RedHatAI/gemma-4-12B-it-FP8-Dynamic --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only
Prefill
prompt tok/s by context length8K context3,8572,067 ms TTFT
16K context3,2094,916 ms TTFT
32K context2,56012,266 ms TTFT
64K context1,80834,646 ms TTFT
128K context1,141109,663 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 75.889 ms TTFT | 146.9127 ms TTFT | 276.8144 ms TTFT | 498.1174 ms TTFT | 827.0241 ms TTFT | 1,156.9340 ms TTFT | 1,713.7578 ms TTFT | 2,388.8838 ms TTFT |
| 16K | 62.3262 ms TTFT | 122.3273 ms TTFT | Looped | Looped | 691.2965 ms TTFT | Looped | 1,197.13,115 ms TTFT | Over capacity |
| 32K | 56.6370 ms TTFT | Looped | Looped | Looped | Looped | Looped | Over capacity | Over capacity |
| 64K | 50.6598 ms TTFT | 99.3920 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
| 128K | Looped | Looped | 144.52,744 ms TTFT | Looped | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 29.4Requests queued2,981 ms TTFT | Looped | 105.85,866 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,857.0 | — | 2,067 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 3,209.0 | — | 4,916 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 2,560.0 | — | 12,266 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,808.0 | — | 34,646 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,141.0 | — | 109,663 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 75.8 | 76.2 | 89 ms | 13.1 ms | 1.0 | 0 | 2,273 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 146.9 | 74.1 | 127 ms | 13.5 ms | 2.0 | 0 | 4,406 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 276.8 | 70.1 | 144 ms | 14.3 ms | 4.0 | 0 | 8,300 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 498.1 | 63.6 | 174 ms | 15.7 ms | 8.0 | 0 | 14,952 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 827.0 | 53.6 | 241 ms | 18.7 ms | 16.0 | 0 | 24,800 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,156.9 | 38.0 | 340 ms | 26.3 ms | 32.0 | 0 | 34,688 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 1,713.7 | 28.3 | 578 ms | 35.3 ms | 64.0 | 0 | 51,392 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 2,388.8 | 19.6 | 838 ms | 51.1 ms | 128.0 | 0 | 71,680 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 62.3 | 62.6 | 262 ms | 16.0 ms | 1.0 | 0 | 1,869 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 122.3 | 61.4 | 273 ms | 16.3 ms | 2.0 | 0 | 3,668 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Valid | 691.2 | 43.3 | 965 ms | 23.1 ms | 16.0 | 0 | 20,737 | 30.0 s |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Valid | 1,197.1 | 18.8 | 3,115 ms | 53.2 ms | 64.0 | 0 | 35,904 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 56.6 | 56.8 | 370 ms | 17.6 ms | 1.0 | 0 | 1,698 | 30.0 s |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 50.6 | 50.9 | 598 ms | 19.7 ms | 1.0 | 0 | 1,519 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 99.3 | 49.9 | 920 ms | 20.0 ms | 2.0 | 0 | 2,980 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 4 | Valid | 144.5 | 36.3 | 2,744 ms | 27.6 ms | 4.0 | 0 | 4,336 | 30.0 s |
| Sustained decode | 128K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 29.4 | 32.0 | 2,981 ms | 31.3 ms | 0.9 | 0 | 881 | 30.0 s |
| Sustained decode | 248K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 248K | 4 | Valid | 105.8 | 26.7 | 5,866 ms | 37.4 ms | 4.0 | 0 | 3,172 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.