Speed test matrix
Ornith-1.5-35B-A3B-FP8 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve ornith-ai/Ornith-1.5-35B-A3B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only
Prefill
prompt tok/s by context length8K context7,7971,045 ms TTFT
16K context7,1542,254 ms TTFT
32K context6,9834,595 ms TTFT
64K context6,00010,672 ms TTFT
128K context4,43228,856 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 128.8130 ms TTFT | 226.2149 ms TTFT | 378.4232 ms TTFT | 611.5349 ms TTFT | 936.3417 ms TTFT | 1,395.7634 ms TTFT | 2,030.9968 ms TTFT | 2,797.61,262 ms TTFT |
| 16K | 123.6278 ms TTFT | 210.1349 ms TTFT | 343.3536 ms TTFT | 532.9952 ms TTFT | 763.51,531 ms TTFT | 1,029.52,804 ms TTFT | Over capacity | Over capacity |
| 32K | 118.2367 ms TTFT | 198.7503 ms TTFT | 312.9752 ms TTFT | 468.81,010 ms TTFT | 643.91,788 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 110.8582 ms TTFT | 180.3803 ms TTFT | 272.91,133 ms TTFT | 377.11,610 ms TTFT | 488.12,557 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 98.4889 ms TTFT | 149.81,110 ms TTFT | 210.81,557 ms TTFT | 273.22,544 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 80.91,844 ms TTFT | 113.22,384 ms TTFT | 149.23,422 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,797.0 | — | 1,045 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,154.0 | — | 2,254 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,983.0 | — | 4,595 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,000.0 | — | 10,672 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,432.0 | — | 28,856 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 128.8 | 128.9 | 130 ms | 7.8 ms | 1.0 | 0 | 3,864 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 226.2 | 113.2 | 149 ms | 8.8 ms | 2.0 | 0 | 6,784 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 378.4 | 94.8 | 232 ms | 10.5 ms | 4.0 | 0 | 11,351 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 611.5 | 76.8 | 349 ms | 13.0 ms | 8.0 | 0 | 18,344 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 936.3 | 59.2 | 417 ms | 16.9 ms | 16.0 | 0 | 28,080 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,395.7 | 44.2 | 634 ms | 22.6 ms | 32.0 | 0 | 41,856 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,030.9 | 32.3 | 968 ms | 30.9 ms | 64.0 | 0 | 60,928 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 2,797.6 | 22.1 | 1,262 ms | 45.2 ms | 128.0 | 0 | 83,968 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 123.6 | 123.7 | 278 ms | 8.1 ms | 1.0 | 0 | 3,706 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 210.1 | 105.1 | 349 ms | 9.5 ms | 2.0 | 0 | 6,302 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 343.3 | 85.6 | 536 ms | 11.7 ms | 4.0 | 0 | 10,295 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 532.9 | 65.9 | 952 ms | 15.2 ms | 8.0 | 0 | 15,984 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 763.5 | 46.9 | 1,531 ms | 21.3 ms | 16.0 | 0 | 22,912 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,029.5 | 31.2 | 2,804 ms | 32.0 ms | 32.0 | 0 | 30,880 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 118.2 | 118.5 | 367 ms | 8.4 ms | 1.0 | 0 | 3,546 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 198.7 | 99.3 | 503 ms | 10.1 ms | 2.0 | 0 | 5,960 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 312.9 | 78.1 | 752 ms | 12.8 ms | 4.0 | 0 | 9,396 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 468.8 | 58.1 | 1,010 ms | 17.2 ms | 8.0 | 0 | 14,064 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 643.9 | 39.5 | 1,788 ms | 25.3 ms | 16.0 | 0 | 19,312 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 110.8 | 111.1 | 582 ms | 9.0 ms | 1.0 | 0 | 3,322 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 180.3 | 90.3 | 803 ms | 11.1 ms | 2.0 | 0 | 5,408 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 272.9 | 67.9 | 1,133 ms | 14.7 ms | 4.0 | 0 | 8,186 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 377.1 | 46.4 | 1,610 ms | 21.6 ms | 8.0 | 0 | 11,312 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 488.1 | 29.2 | 2,557 ms | 34.2 ms | 16.0 | 0 | 14,640 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 98.4 | 98.8 | 889 ms | 10.1 ms | 1.0 | 0 | 2,952 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 149.8 | 75.2 | 1,110 ms | 13.3 ms | 2.0 | 0 | 4,492 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 210.8 | 52.8 | 1,557 ms | 19.0 ms | 4.0 | 0 | 6,324 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 273.2 | 34.0 | 2,544 ms | 29.4 ms | 8.0 | 0 | 8,192 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 80.9 | 81.4 | 1,844 ms | 12.3 ms | 1.0 | 0 | 2,426 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 113.2 | 56.7 | 2,384 ms | 17.6 ms | 2.0 | 0 | 3,396 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 149.2 | 36.9 | 3,422 ms | 27.1 ms | 4.0 | 0 | 4,476 | 30.0 s |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.