Speed test matrix
Sage-A1.5-FP8 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve InternScience/Sage-A1.5-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}Prefill
prompt tok/s by context length8K context7,5181,083 ms TTFT
16K context6,8172,364 ms TTFT
32K context7,0224,569 ms TTFT
64K context5,78911,059 ms TTFT
128K context4,19830,461 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 183.4133 ms TTFT | 322.6277 ms TTFT | 540.7287 ms TTFT | 790.3391 ms TTFT | 1,195.7443 ms TTFT | 1,674.3611 ms TTFT | 2,232.1963 ms TTFT | Over capacity |
| 16K | 118.6346 ms TTFT | 206.8443 ms TTFT | 350.2698 ms TTFT | 454.41,056 ms TTFT | 562.21,792 ms TTFT | 680.43,377 ms TTFT | Over capacity | Over capacity |
| 32K | 83.1562 ms TTFT | 150.6807 ms TTFT | 266.11,232 ms TTFT | 307.12,059 ms TTFT | 362.73,722 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 52.0808 ms TTFT | 98.9988 ms TTFT | 176.01,685 ms TTFT | 191.02,605 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 30.61,264 ms TTFT | 57.51,483 ms TTFT | 101.42,344 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 16.32,147 ms TTFT | 32.02,796 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,518.0 | — | 1,083 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,817.0 | — | 2,364 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 7,022.0 | — | 4,569 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,789.0 | — | 11,059 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,198.0 | — | 30,461 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 183.4 | 187.1 | 133 ms | 5.3 ms | 1.0 | 0 | 5,500 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 322.6 | 165.2 | 277 ms | 6.1 ms | 2.0 | 0 | 9,675 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 540.7 | 137.7 | 287 ms | 7.3 ms | 4.0 | 0 | 16,215 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 790.3 | 101.3 | 391 ms | 9.9 ms | 8.0 | 0 | 23,710 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,195.7 | 77.3 | 443 ms | 12.9 ms | 16.0 | 0 | 35,846 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,674.3 | 54.1 | 611 ms | 18.5 ms | 32.0 | 0 | 50,218 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,232.1 | 36.2 | 963 ms | 27.6 ms | 64.0 | 0 | 66,916 | 30.0 s |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 118.6 | 119.4 | 346 ms | 8.4 ms | 1.0 | 0 | 3,558 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 206.8 | 104.5 | 443 ms | 9.6 ms | 2.0 | 0 | 6,204 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 350.2 | 88.7 | 698 ms | 11.3 ms | 4.0 | 0 | 10,501 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 454.4 | 56.9 | 1,056 ms | 17.6 ms | 8.0 | 0 | 13,622 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 562.2 | 35.1 | 1,792 ms | 28.5 ms | 16.0 | 0 | 16,841 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 680.4 | 20.9 | 3,377 ms | 47.7 ms | 32.0 | 0 | 20,410 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 83.1 | 84.6 | 562 ms | 11.8 ms | 1.0 | 0 | 2,492 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 150.6 | 75.6 | 807 ms | 13.2 ms | 2.0 | 0 | 4,516 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 266.1 | 66.2 | 1,232 ms | 15.1 ms | 4.0 | 0 | 7,983 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 307.1 | 38.2 | 2,059 ms | 26.2 ms | 8.0 | 0 | 9,210 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 362.7 | 22.3 | 3,722 ms | 44.9 ms | 16.0 | 0 | 10,877 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 52.0 | 52.9 | 808 ms | 18.9 ms | 1.0 | 0 | 1,557 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 98.9 | 49.7 | 988 ms | 20.1 ms | 2.0 | 0 | 2,964 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 176.0 | 44.1 | 1,685 ms | 22.7 ms | 4.0 | 0 | 5,280 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 191.0 | 23.8 | 2,605 ms | 42.0 ms | 8.0 | 0 | 5,717 | 29.9 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 30.6 | 31.2 | 1,264 ms | 32.1 ms | 1.0 | 0 | 918 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 57.5 | 28.9 | 1,483 ms | 34.6 ms | 2.0 | 0 | 1,721 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 101.4 | 25.3 | 2,344 ms | 39.5 ms | 4.0 | 0 | 3,043 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Valid | 16.3 | 16.7 | 2,147 ms | 59.8 ms | 1.0 | 0 | 486 | 29.9 s |
| Sustained decode | 248K | 2 | Valid | 32.0 | 16.3 | 2,796 ms | 61.5 ms | 2.0 | 0 | 958 | 29.9 s |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.