Speed test matrix
Ornith-1.5-35B-A3B-FP8 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve ornith-ai/Ornith-1.5-35B-A3B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}Prefill
prompt tok/s by context length8K context6,7451,208 ms TTFT
16K context7,2022,238 ms TTFT
32K context6,4404,982 ms TTFT
64K context5,71611,201 ms TTFT
128K context4,19430,498 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 162.7147 ms TTFT | 289.9286 ms TTFT | 471.2335 ms TTFT | 715.9458 ms TTFT | 1,143.9493 ms TTFT | 1,535.9667 ms TTFT | 2,073.61,026 ms TTFT | Over capacity |
| 16K | 111.6366 ms TTFT | 189.3438 ms TTFT | 341.3687 ms TTFT | 425.71,009 ms TTFT | 546.31,865 ms TTFT | 665.23,342 ms TTFT | Over capacity | Over capacity |
| 32K | 75.7587 ms TTFT | 145.5804 ms TTFT | 264.61,243 ms TTFT | 302.72,068 ms TTFT | 355.83,755 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 49.2815 ms TTFT | 97.61,044 ms TTFT | 169.31,741 ms TTFT | 193.32,632 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 29.41,265 ms TTFT | 55.61,614 ms TTFT | 102.72,445 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 16.22,132 ms TTFT | 32.32,710 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,745.0 | — | 1,208 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 7,202.0 | — | 2,238 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,440.0 | — | 4,982 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,716.0 | — | 11,201 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,194.0 | — | 30,498 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 162.7 | 164.8 | 147 ms | 6.1 ms | 1.0 | 0 | 4,881 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 289.9 | 148.6 | 286 ms | 6.7 ms | 2.0 | 0 | 8,695 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 471.2 | 120.3 | 335 ms | 8.3 ms | 4.0 | 0 | 14,137 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 715.9 | 92.6 | 458 ms | 10.8 ms | 8.0 | 0 | 21,463 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,143.9 | 73.2 | 493 ms | 13.7 ms | 16.0 | 0 | 34,320 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,535.9 | 49.8 | 667 ms | 20.1 ms | 32.0 | 0 | 46,068 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,073.6 | 33.2 | 1,026 ms | 30.1 ms | 64.0 | 0 | 62,230 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 111.6 | 111.8 | 366 ms | 8.9 ms | 1.0 | 0 | 3,348 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 189.3 | 95.5 | 438 ms | 10.5 ms | 2.0 | 0 | 5,677 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 341.3 | 86.0 | 687 ms | 11.6 ms | 4.0 | 0 | 10,239 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 425.7 | 53.8 | 1,009 ms | 18.6 ms | 8.0 | 0 | 12,759 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 546.3 | 33.5 | 1,865 ms | 29.8 ms | 16.0 | 0 | 16,366 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 665.2 | 20.1 | 3,342 ms | 49.7 ms | 32.0 | 0 | 19,911 | 29.9 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 75.7 | 76.4 | 587 ms | 13.1 ms | 1.0 | 0 | 2,271 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 145.5 | 73.3 | 804 ms | 13.6 ms | 2.0 | 0 | 4,365 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 264.6 | 66.1 | 1,243 ms | 15.1 ms | 4.0 | 0 | 7,933 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 302.7 | 37.0 | 2,068 ms | 27.0 ms | 8.0 | 0 | 9,072 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 355.8 | 21.3 | 3,755 ms | 47.0 ms | 16.0 | 0 | 10,645 | 29.9 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 49.2 | 49.4 | 815 ms | 20.2 ms | 1.0 | 0 | 1,477 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 97.6 | 49.1 | 1,044 ms | 20.4 ms | 2.0 | 0 | 2,926 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 169.3 | 42.1 | 1,741 ms | 23.8 ms | 4.0 | 0 | 5,070 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 193.3 | 23.9 | 2,632 ms | 41.9 ms | 8.0 | 0 | 5,793 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 29.4 | 29.7 | 1,265 ms | 33.6 ms | 1.0 | 0 | 879 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 55.6 | 27.9 | 1,614 ms | 35.9 ms | 2.0 | 0 | 1,666 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 102.7 | 25.5 | 2,445 ms | 39.2 ms | 4.0 | 0 | 3,078 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 16.2 | 16.7 | 2,132 ms | 59.9 ms | 1.0 | 0 | 486 | 29.9 s |
| Sustained decode | 248K | 2 | Valid | 32.3 | 16.4 | 2,710 ms | 60.9 ms | 2.0 | 0 | 970 | 30.0 s |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.