Speed test matrix
gpt-oss-20b on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · MXFP45 contexts (0–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- MXFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve openai/gpt-oss-20b --max-model-len 131072 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context8,710931 ms TTFT
16K context6,8242,344 ms TTFT
32K context4,9306,448 ms TTFT
64K context3,11220,366 ms TTFT
120K context1,87763,201 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 217.087 ms TTFT | 325.797 ms TTFT | 357.0146 ms TTFT | 671.4168 ms TTFT | 1,169.7256 ms TTFT | 1,497.0287 ms TTFT | 2,928.7484 ms TTFT | 3,766.5821 ms TTFT |
| 16K | 162.1153 ms TTFT | 262.9224 ms TTFT | 316.4369 ms TTFT | 563.3614 ms TTFT | 704.81,094 ms TTFT | 977.62,016 ms TTFT | 1,481.43,014 ms TTFT | Over capacity |
| 32K | 136.9221 ms TTFT | 228.6331 ms TTFT | 193.8557 ms TTFT | 446.61,065 ms TTFT | 584.71,855 ms TTFT | 705.33,308 ms TTFT | Over capacity | Over capacity |
| 64K | 109.6328 ms TTFT | 177.5564 ms TTFT | 225.41,003 ms TTFT | 266.71,692 ms TTFT | 305.83,059 ms TTFT | Over capacity | Over capacity | Over capacity |
| 120K | 76.1 | 80.6839 ms TTFT | 143.0967 ms TTFT | 161.72,336 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,710.0 | — | 931 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,824.0 | — | 2,344 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,930.0 | — | 6,448 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,112.0 | — | 20,366 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 1,877.0 | — | 63,201 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 217.0 | 194.1 | 87 ms | 5.2 ms | 1.0 | 0 | 6,519 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 325.7 | 153.8 | 97 ms | 6.5 ms | 2.0 | 0 | 9,762 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 357.0 | 118.7 | 146 ms | 8.4 ms | 4.0 | 0 | 10,708 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 671.4 | 96.6 | 168 ms | 10.4 ms | 8.0 | 0 | 20,133 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 1,169.7 | 75.4 | 256 ms | 13.3 ms | 16.0 | 0 | 35,083 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,497.0 | 62.2 | 287 ms | 16.1 ms | 32.0 | 0 | 44,902 | 30.0 s |
| Sustained decode | 0 | 64 | Valid | 2,928.7 | 50.9 | 484 ms | 19.7 ms | 64.0 | 0 | 87,836 | 30.0 s |
| Sustained decode | 0 | 128 | Valid | 3,766.5 | 36.3 | 821 ms | 27.6 ms | 128.0 | 0 | 112,947 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 162.1 | 160.8 | 153 ms | 6.2 ms | 1.0 | 0 | 4,852 | 29.9 s |
| Sustained decode | 16K | 2 | Valid | 262.9 | 125.4 | 224 ms | 8.0 ms | 2.0 | 0 | 7,887 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 316.4 | 92.8 | 369 ms | 10.8 ms | 4.0 | 0 | 9,493 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 563.3 | 71.2 | 614 ms | 14.0 ms | 8.0 | 0 | 16,897 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 704.8 | 51.1 | 1,094 ms | 19.6 ms | 16.0 | 0 | 21,149 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 977.6 | 39.9 | 2,016 ms | 25.1 ms | 32.0 | 0 | 29,325 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,481.4 | 29.6 | 3,014 ms | 33.7 ms | 64.0 | 0 | 44,420 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 136.9 | 137.5 | 221 ms | 7.3 ms | 1.0 | 0 | 4,108 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 228.6 | 109.8 | 331 ms | 9.1 ms | 2.0 | 0 | 6,860 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 193.8 | 79.8 | 557 ms | 12.5 ms | 4.0 | 0 | 5,814 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 446.6 | 56.7 | 1,065 ms | 17.6 ms | 8.0 | 0 | 13,398 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 584.7 | 39.2 | 1,855 ms | 25.5 ms | 16.0 | 0 | 17,536 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 705.3 | 29.4 | 3,308 ms | 34.0 ms | 32.0 | 0 | 21,173 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 109.6 | 106.5 | 328 ms | 9.4 ms | 1.0 | 0 | 3,085 | 28.2 s |
| Sustained decode | 64K | 2 | Valid | 177.5 | 87.1 | 564 ms | 11.5 ms | 2.0 | 0 | 5,325 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 225.4 | 58.0 | 1,003 ms | 17.2 ms | 4.0 | 0 | 6,762 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 266.7 | 40.4 | 1,692 ms | 24.7 ms | 8.0 | 0 | 7,999 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 305.8 | 25.9 | 3,059 ms | 38.7 ms | 16.0 | 0 | 9,165 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 76.1 | — | — | — | 1.0 | 0 | 2,135 | 28.1 s |
| Sustained decode | 120K | 2 | Valid | 80.6 | 64.2 | 839 ms | 15.6 ms | 2.0 | 0 | 2,418 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 143.0 | 39.4 | 967 ms | 25.4 ms | 4.0 | 0 | 4,262 | 29.8 s |
| Sustained decode | 120K | 8 | Valid | 161.7 | 26.5 | 2,336 ms | 37.8 ms | 8.0 | 0 | 4,848 | 30.0 s |
| Sustained decode | 120K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.