Speed test matrix
Ornith-1.5-9B-NVFP4 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · NVFP46 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve ornith-ai/Ornith-1.5-9B-NVFP4 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}Prefill
prompt tok/s by context length8K context5,5921,457 ms TTFT
16K context5,2253,085 ms TTFT
32K context4,8326,641 ms TTFT
64K context4,14315,453 ms TTFT
128K context3,21339,808 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 108.1111 ms TTFT | 200.1136 ms TTFT | 345.6263 ms TTFT | 660.2315 ms TTFT | 1,245.1465 ms TTFT | 1,556.2714 ms TTFT | 2,147.71,201 ms TTFT | 2,410.91,813 ms TTFT |
| 16K | 98.4358 ms TTFT | 197.5612 ms TTFT | 386.3956 ms TTFT | 700.71,476 ms TTFT | 1,050.52,507 ms TTFT | 1,296.14,637 ms TTFT | 1,649.68,847 ms TTFT | Over capacity |
| 32K | 105.3704 ms TTFT | 208.41,102 ms TTFT | 385.11,768 ms TTFT | 630.12,869 ms TTFT | 926.05,035 ms TTFT | 1,095.19,703 ms TTFT | 1,347.819,602 ms TTFT | Over capacity |
| 64K | 97.4897 ms TTFT | 213.71,602 ms TTFT | 362.42,496 ms TTFT | 537.83,755 ms TTFT | 735.06,236 ms TTFT | 835.411,222 ms TTFT | Over capacity | Over capacity |
| 128K | 97.71,417 ms TTFT | 188.52,138 ms TTFT | 289.03,152 ms TTFT | 391.74,580 ms TTFT | 496.37,621 ms TTFT | Over capacity | Over capacity | Over capacity |
| 248K | 91.02,292 ms TTFT | 183.52,743 ms TTFT | 235.04,204 ms TTFT | 273.56,159 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,592.0 | — | 1,457 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,225.0 | — | 3,085 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,832.0 | — | 6,641 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,143.0 | — | 15,453 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,213.0 | — | 39,808 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 108.1 | 109.8 | 111 ms | 9.1 ms | 1.0 | 0 | 3,243 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 200.1 | 101.3 | 136 ms | 9.9 ms | 2.0 | 0 | 6,001 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 345.6 | 87.5 | 263 ms | 11.4 ms | 4.0 | 0 | 10,369 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 660.2 | 83.9 | 315 ms | 11.9 ms | 8.0 | 0 | 19,808 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,245.1 | 79.4 | 465 ms | 12.6 ms | 16.0 | 0 | 37,343 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,556.2 | 49.5 | 714 ms | 20.2 ms | 32.0 | 0 | 46,688 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,147.7 | 34.1 | 1,201 ms | 29.3 ms | 64.0 | 0 | 64,404 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 2,410.9 | 19.1 | 1,813 ms | 52.5 ms | 128.0 | 0 | 72,327 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 98.4 | 102.0 | 358 ms | 9.8 ms | 1.0 | 0 | 2,952 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 197.5 | 102.9 | 612 ms | 9.7 ms | 2.0 | 0 | 5,922 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 386.3 | 100.1 | 956 ms | 10.0 ms | 4.0 | 0 | 11,589 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 700.7 | 90.0 | 1,476 ms | 11.1 ms | 8.0 | 0 | 21,021 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,050.5 | 65.1 | 2,507 ms | 15.4 ms | 16.0 | 0 | 31,508 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,296.1 | 37.3 | 4,637 ms | 26.8 ms | 32.0 | 0 | 38,859 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,649.6 | 22.2 | 8,847 ms | 45.0 ms | 64.0 | 0 | 49,543 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 105.3 | 108.1 | 704 ms | 9.2 ms | 1.0 | 0 | 3,157 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 208.4 | 104.7 | 1,102 ms | 9.6 ms | 2.0 | 0 | 6,253 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 385.1 | 96.9 | 1,768 ms | 10.3 ms | 4.0 | 0 | 11,555 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 630.1 | 78.7 | 2,869 ms | 12.7 ms | 8.0 | 0 | 18,902 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 926.0 | 54.6 | 5,035 ms | 18.3 ms | 16.0 | 0 | 27,785 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,095.1 | 28.7 | 9,703 ms | 34.8 ms | 32.0 | 0 | 32,856 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,347.8 | 15.5 | 19,602 ms | 64.4 ms | 64.0 | 0 | 40,379 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 97.4 | 101.2 | 897 ms | 9.9 ms | 1.0 | 0 | 2,921 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 213.7 | 109.3 | 1,602 ms | 9.3 ms | 2.0 | 0 | 6,409 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 362.4 | 92.0 | 2,496 ms | 10.9 ms | 4.0 | 0 | 10,867 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 537.8 | 67.0 | 3,755 ms | 14.9 ms | 8.0 | 0 | 16,130 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 735.0 | 42.2 | 6,236 ms | 23.7 ms | 16.0 | 0 | 22,039 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 835.4 | 21.3 | 11,222 ms | 46.9 ms | 32.0 | 0 | 25,017 | 29.9 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 97.7 | 100.4 | 1,417 ms | 10.0 ms | 1.0 | 0 | 2,929 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 188.5 | 96.8 | 2,138 ms | 10.3 ms | 2.0 | 0 | 5,655 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 289.0 | 73.0 | 3,152 ms | 13.7 ms | 4.0 | 0 | 8,667 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 391.7 | 48.0 | 4,580 ms | 20.8 ms | 8.0 | 0 | 11,747 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 496.3 | 28.5 | 7,621 ms | 35.1 ms | 16.0 | 0 | 14,876 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 91.0 | 93.6 | 2,292 ms | 10.7 ms | 1.0 | 0 | 2,730 | 30.0 s |
| Sustained decode | 248K | 2 | Valid | 183.5 | 92.0 | 2,743 ms | 10.9 ms | 2.0 | 0 | 5,501 | 30.0 s |
| Sustained decode | 248K | 4 | Valid | 235.0 | 58.6 | 4,204 ms | 17.1 ms | 4.0 | 0 | 7,043 | 30.0 s |
| Sustained decode | 248K | 8 | Valid | 273.5 | 33.0 | 6,159 ms | 30.3 ms | 8.0 | 0 | 8,201 | 30.0 s |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.