Speed test matrix
Hy-MT2-7B-FP8 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve tencent/Hy-MT2-7B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --enforce-eager
Prefill
prompt tok/s by context length8K context6,5281,255 ms TTFT
16K context5,7872,804 ms TTFT
32K context4,8016,733 ms TTFT
64K context3,54818,186 ms TTFT
128K context2,29656,153 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 19.6225 ms TTFT | 36.7293 ms TTFT | 73.0441 ms TTFT | 145.3397 ms TTFT | 287.3455 ms TTFT | 573.6539 ms TTFT | Over capacity | Over capacity |
| 16K | 19.5285 ms TTFT | 36.3333 ms TTFT | 72.7392 ms TTFT | 145.1553 ms TTFT | 286.7806 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 19.6437 ms TTFT | 36.3408 ms TTFT | 72.4591 ms TTFT | 144.6825 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 19.6517 ms TTFT | 36.2697 ms TTFT | 72.41,097 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | Looped | 36.31,258 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 18.7Requests queued1,658 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,528.0 | — | 1,255 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,787.0 | — | 2,804 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,801.0 | — | 6,733 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,548.0 | — | 18,186 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 2,296.0 | — | 56,153 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 19.6 | 19.6 | 225 ms | 51.1 ms | 1.0 | 0 | 587 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 36.7 | 18.3 | 293 ms | 54.6 ms | 2.0 | 0 | 1,100 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 73.0 | 18.2 | 441 ms | 54.8 ms | 4.0 | 0 | 2,188 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 145.3 | 18.2 | 397 ms | 55.0 ms | 8.0 | 0 | 4,352 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 287.3 | 17.9 | 455 ms | 55.8 ms | 16.0 | 0 | 8,608 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 573.6 | 17.9 | 539 ms | 55.9 ms | 32.0 | 0 | 17,184 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 19.5 | 19.5 | 285 ms | 51.3 ms | 1.0 | 0 | 584 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 36.3 | 18.2 | 333 ms | 55.1 ms | 2.0 | 0 | 1,090 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 72.7 | 18.2 | 392 ms | 55.1 ms | 4.0 | 0 | 2,180 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 145.1 | 18.2 | 553 ms | 55.1 ms | 8.0 | 0 | 4,352 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 286.7 | 17.8 | 806 ms | 56.1 ms | 16.0 | 0 | 8,592 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 19.6 | 19.6 | 437 ms | 51.1 ms | 1.0 | 0 | 588 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 36.3 | 18.2 | 408 ms | 55.0 ms | 2.0 | 0 | 1,088 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 72.4 | 18.1 | 591 ms | 55.2 ms | 4.0 | 0 | 2,168 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 144.6 | 18.1 | 825 ms | 55.4 ms | 8.0 | 0 | 4,336 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 19.6 | 19.6 | 517 ms | 51.0 ms | 1.0 | 0 | 587 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 36.2 | 18.1 | 697 ms | 55.3 ms | 2.0 | 0 | 1,084 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 72.4 | 18.1 | 1,097 ms | 55.2 ms | 4.0 | 0 | 2,172 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 128K | 2 | Valid | 36.3 | 18.2 | 1,258 ms | 55.1 ms | 2.0 | 0 | 1,088 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 18.7 | 19.4 | 1,658 ms | 51.5 ms | 1.0 | 0 | 561 | 30.0 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.