Speed test matrix
Qwen3.6-27B-FP8 on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP86 contexts (0–248K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: MTP, 2 tokens lookahead
# 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image
vllm serve Qwen/Qwen3.6-27B-FP8 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --language-model-only --speculative-config {"method":"mtp","num_speculative_tokens":2}Prefill
prompt tok/s by context length8K context1,6934,810 ms TTFT
16K context1,60910,018 ms TTFT
32K context1,52221,080 ms TTFT
64K context1,35947,103 ms TTFT
128K context1,117114,469 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 74.6234 ms TTFT | 148.0394 ms TTFT | 253.3489 ms TTFT | 429.5608 ms TTFT | 666.61,039 ms TTFT | 747.01,677 ms TTFT | Over capacity | Over capacity |
| 16K | 55.9739 ms TTFT | 105.71,259 ms TTFT | 157.72,380 ms TTFT | 232.64,027 ms TTFT | 282.86,573 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 44.6847 ms TTFT | 81.21,396 ms TTFT | 110.22,648 ms TTFT | 157.34,463 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 30.41,138 ms TTFT | 57.41,740 ms TTFT | 71.13,186 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 18.12,487 ms TTFT | 35.94,034 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 248K | 10.4Requests queued3,611 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 1,693.0 | — | 4,810 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 1,609.0 | — | 10,018 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 1,522.0 | — | 21,080 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 1,359.0 | — | 47,103 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 1,117.0 | — | 114,469 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 74.6 | 75.9 | 234 ms | 13.2 ms | 1.0 | 0 | 2,235 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 148.0 | 74.1 | 394 ms | 13.5 ms | 2.0 | 0 | 4,439 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 253.3 | 64.6 | 489 ms | 15.5 ms | 4.0 | 0 | 7,592 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 429.5 | 54.7 | 608 ms | 18.3 ms | 8.0 | 0 | 12,877 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 666.6 | 42.9 | 1,039 ms | 23.3 ms | 16.0 | 0 | 19,995 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 747.0 | 23.9 | 1,677 ms | 41.9 ms | 32.0 | 0 | 22,415 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 55.9 | 56.4 | 739 ms | 17.7 ms | 1.0 | 0 | 1,675 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 105.7 | 53.3 | 1,259 ms | 18.8 ms | 2.0 | 0 | 3,167 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 157.7 | 39.6 | 2,380 ms | 25.3 ms | 4.0 | 0 | 4,731 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 232.6 | 28.5 | 4,027 ms | 35.1 ms | 8.0 | 0 | 6,968 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 282.8 | 16.4 | 6,573 ms | 60.9 ms | 16.0 | 0 | 8,462 | 29.9 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 44.6 | 44.6 | 847 ms | 22.4 ms | 1.0 | 0 | 1,337 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 81.2 | 41.2 | 1,396 ms | 24.3 ms | 2.0 | 0 | 2,435 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 110.2 | 27.9 | 2,648 ms | 35.9 ms | 4.0 | 0 | 3,298 | 29.9 s |
| Sustained decode | 32K | 8 | Valid | 157.3 | 19.5 | 4,463 ms | 51.3 ms | 8.0 | 0 | 4,703 | 29.9 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 30.4 | 30.9 | 1,138 ms | 32.3 ms | 1.0 | 0 | 911 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 57.4 | 29.3 | 1,740 ms | 34.2 ms | 2.0 | 0 | 1,722 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 71.1 | 17.9 | 3,186 ms | 55.8 ms | 4.0 | 0 | 2,125 | 29.9 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 18.1 | 18.6 | 2,487 ms | 53.9 ms | 1.0 | 0 | 540 | 29.9 s |
| Sustained decode | 128K | 2 | Valid | 35.9 | 18.2 | 4,034 ms | 55.1 ms | 2.0 | 0 | 1,073 | 29.9 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 1 | Requests queued | 10.4 | 11.6 | 3,611 ms | 86.4 ms | 0.9 | 0 | 312 | 30.0 s |
| Sustained decode | 248K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 248K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.