Speed test matrix
Ornith-1.5-35B-A3B-W4A16-SYM on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · W4A166 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- W4A16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve MIRALABS/Ornith-1.5-35B-A3B-W4A16-SYM --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only
Prefill
prompt tok/s by context length8K context6,9021,181 ms TTFT
16K context6,7072,403 ms TTFT
32K context7,4314,318 ms TTFT
64K context6,17710,366 ms TTFT
128K context4,55828,060 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 132.5139 ms TTFT | 238.6228 ms TTFT | 412.8265 ms TTFT | 681.5360 ms TTFT | 1,072.0394 ms TTFT | 1,626.0618 ms TTFT | 2,373.2853 ms TTFT | 3,131.61,264 ms TTFT |
| 16K | 127.2262 ms TTFT | 222.5379 ms TTFT | 373.2570 ms TTFT | 578.9940 ms TTFT | 854.61,488 ms TTFT | 1,162.92,721 ms TTFT | 1,501.55,295 ms TTFT | Over capacity |
| 32K | 121.4317 ms TTFT | 209.7478 ms TTFT | 337.8650 ms TTFT | 505.11,008 ms TTFT | 703.91,679 ms TTFT | 908.73,066 ms TTFT | Over capacity | Over capacity |
| 64K | 113.4621 ms TTFT | 187.4723 ms TTFT | 288.51,073 ms TTFT | 402.51,428 ms TTFT | 521.92,545 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 100.3883 ms TTFT | 154.91,014 ms TTFT | 221.01,660 ms TTFT | 286.22,636 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 82.01,893 ms TTFT | 116.32,449 ms TTFT | 153.03,482 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,902.0 | — | 1,181 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,707.0 | — | 2,403 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,431.0 | — | 4,318 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,177.0 | — | 10,366 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,558.0 | — | 28,060 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 132.5 | 132.6 | 139 ms | 7.5 ms | 1.0 | 0 | 3,975 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 238.6 | 119.5 | 228 ms | 8.4 ms | 2.0 | 0 | 7,157 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 412.8 | 103.6 | 265 ms | 9.6 ms | 4.0 | 0 | 12,384 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 681.5 | 85.6 | 360 ms | 11.7 ms | 8.0 | 0 | 20,456 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,072.0 | 67.7 | 394 ms | 14.8 ms | 16.0 | 0 | 32,160 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,626.0 | 51.6 | 618 ms | 19.4 ms | 32.0 | 0 | 48,768 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,373.2 | 37.6 | 853 ms | 26.6 ms | 64.0 | 0 | 71,168 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,131.6 | 24.7 | 1,264 ms | 40.5 ms | 128.0 | 0 | 93,952 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 127.2 | 127.3 | 262 ms | 7.9 ms | 1.0 | 0 | 3,815 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 222.5 | 111.2 | 379 ms | 9.0 ms | 2.0 | 0 | 6,675 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 373.2 | 93.0 | 570 ms | 10.7 ms | 4.0 | 0 | 11,196 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 578.9 | 71.6 | 940 ms | 14.0 ms | 8.0 | 0 | 17,360 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 854.6 | 52.4 | 1,488 ms | 19.1 ms | 16.0 | 0 | 25,664 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,162.9 | 35.2 | 2,721 ms | 28.4 ms | 32.0 | 0 | 34,880 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,501.5 | 21.7 | 5,295 ms | 46.0 ms | 64.0 | 0 | 45,056 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 121.4 | 121.7 | 317 ms | 8.2 ms | 1.0 | 0 | 3,643 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 209.7 | 104.9 | 478 ms | 9.5 ms | 2.0 | 0 | 6,290 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 337.8 | 84.3 | 650 ms | 11.9 ms | 4.0 | 0 | 10,132 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 505.1 | 62.6 | 1,008 ms | 16.0 ms | 8.0 | 0 | 15,153 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 703.9 | 43.2 | 1,679 ms | 23.2 ms | 16.0 | 0 | 21,120 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 908.7 | 26.9 | 3,066 ms | 37.1 ms | 32.0 | 0 | 27,264 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 113.4 | 113.8 | 621 ms | 8.8 ms | 1.0 | 0 | 3,402 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 187.4 | 93.8 | 723 ms | 10.7 ms | 2.0 | 0 | 5,623 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 288.5 | 71.7 | 1,073 ms | 13.9 ms | 4.0 | 0 | 8,652 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 402.5 | 49.5 | 1,428 ms | 20.2 ms | 8.0 | 0 | 12,072 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 521.9 | 31.3 | 2,545 ms | 32.0 ms | 16.0 | 0 | 15,648 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 100.3 | 100.8 | 883 ms | 9.9 ms | 1.0 | 0 | 3,009 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 154.9 | 77.8 | 1,014 ms | 12.9 ms | 2.0 | 0 | 4,648 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 221.0 | 55.3 | 1,660 ms | 18.1 ms | 4.0 | 0 | 6,628 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 286.2 | 35.6 | 2,636 ms | 28.1 ms | 8.0 | 0 | 8,584 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 82.0 | 82.6 | 1,893 ms | 12.1 ms | 1.0 | 0 | 2,459 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 116.3 | 58.3 | 2,449 ms | 17.1 ms | 2.0 | 0 | 3,488 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 153.0 | 37.9 | 3,482 ms | 26.4 ms | 4.0 | 0 | 4,588 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.