Speed test matrix
Qwen3.5-4B-FP8-dynamic on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP8 dynamic6 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8 dynamic
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve RedHatAI/Qwen3.5-4B-FP8-dynamic --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only
Prefill
prompt tok/s by context length8K context11K740 ms TTFT
16K context10.3K1,561 ms TTFT
32K context9,4703,388 ms TTFT
64K context7,8718,134 ms TTFT
128K context5,70922,401 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 145.5117 ms TTFT | 277.8144 ms TTFT | 525.0205 ms TTFT | 954.2248 ms TTFT | 1,681.1346 ms TTFT | 2,562.7448 ms TTFT | 3,680.2666 ms TTFT | 4,738.81,065 ms TTFT |
| 16K | 135.7169 ms TTFT | 247.8290 ms TTFT | 434.8397 ms TTFT | 701.3656 ms TTFT | 1,038.61,036 ms TTFT | 1,334.61,890 ms TTFT | 1,553.93,512 ms TTFT | Over capacity |
| 32K | 127.9294 ms TTFT | 223.7372 ms TTFT | 368.9509 ms TTFT | 549.7747 ms TTFT | 744.21,252 ms TTFT | 890.32,364 ms TTFT | Over capacity | Over capacity |
| 64K | 114.9471 ms TTFT | 186.2650 ms TTFT | 282.2878 ms TTFT | 382.91,424 ms TTFT | 473.12,465 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 94.9875 ms TTFT | 138.91,131 ms TTFT | 191.11,635 ms TTFT | 237.72,559 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 71.21,636 ms TTFT | 94.32,159 ms TTFT | 118.93,451 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11K | — | 740 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.3K | — | 1,561 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,470.0 | — | 3,388 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,871.0 | — | 8,134 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,709.0 | — | 22,401 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 145.5 | 145.9 | 117 ms | 6.9 ms | 1.0 | 0 | 4,364 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 277.8 | 139.6 | 144 ms | 7.2 ms | 2.0 | 0 | 8,334 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 525.0 | 132.1 | 205 ms | 7.6 ms | 4.0 | 0 | 15,752 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 954.2 | 120.5 | 248 ms | 8.3 ms | 8.0 | 0 | 28,648 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,681.1 | 106.7 | 346 ms | 9.4 ms | 16.0 | 0 | 50,432 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 2,562.7 | 81.6 | 448 ms | 12.3 ms | 32.0 | 0 | 76,896 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 3,680.2 | 58.6 | 666 ms | 17.1 ms | 64.0 | 0 | 110,425 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 4,738.8 | 37.2 | 1,065 ms | 26.9 ms | 128.0 | 0 | 142,208 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 135.7 | 136.1 | 169 ms | 7.3 ms | 1.0 | 0 | 4,072 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 247.8 | 124.2 | 290 ms | 8.1 ms | 2.0 | 0 | 7,434 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 434.8 | 108.8 | 397 ms | 9.2 ms | 4.0 | 0 | 13,048 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 701.3 | 87.4 | 656 ms | 11.4 ms | 8.0 | 0 | 21,048 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,038.6 | 64.5 | 1,036 ms | 15.5 ms | 16.0 | 0 | 31,168 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,334.6 | 40.9 | 1,890 ms | 24.4 ms | 32.0 | 0 | 40,032 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,553.9 | 23.1 | 3,512 ms | 43.3 ms | 64.0 | 0 | 46,592 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 127.9 | 128.3 | 294 ms | 7.8 ms | 1.0 | 0 | 3,835 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 223.7 | 112.1 | 372 ms | 8.9 ms | 2.0 | 0 | 6,710 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 368.9 | 92.2 | 509 ms | 10.8 ms | 4.0 | 0 | 11,065 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 549.7 | 68.4 | 747 ms | 14.6 ms | 8.0 | 0 | 16,488 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 744.2 | 45.9 | 1,252 ms | 21.8 ms | 16.0 | 0 | 22,320 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 890.3 | 26.8 | 2,364 ms | 37.3 ms | 32.0 | 0 | 26,720 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 114.9 | 115.3 | 471 ms | 8.7 ms | 1.0 | 0 | 3,447 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 186.2 | 93.4 | 650 ms | 10.7 ms | 2.0 | 0 | 5,585 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 282.2 | 70.6 | 878 ms | 14.2 ms | 4.0 | 0 | 8,464 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 382.9 | 47.8 | 1,424 ms | 20.9 ms | 8.0 | 0 | 11,488 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 473.1 | 29.3 | 2,465 ms | 34.1 ms | 16.0 | 0 | 14,192 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 94.9 | 95.4 | 875 ms | 10.5 ms | 1.0 | 0 | 2,845 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 138.9 | 69.8 | 1,131 ms | 14.3 ms | 2.0 | 0 | 4,166 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 191.1 | 48.0 | 1,635 ms | 20.9 ms | 4.0 | 0 | 5,732 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 237.7 | 29.7 | 2,559 ms | 33.7 ms | 8.0 | 0 | 7,128 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 71.2 | 71.8 | 1,636 ms | 13.9 ms | 1.0 | 0 | 2,134 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 94.3 | 47.6 | 2,159 ms | 21.0 ms | 2.0 | 0 | 2,828 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 118.9 | 30.1 | 3,451 ms | 33.3 ms | 4.0 | 0 | 3,564 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.