Speed test matrix
Reka-Flash-3-21B-FP8-dynamic on CMP 170HX 64GB
vllm · 0.32.0rc1.dev249+g18305948a · FP8 dynamic3 contexts (0–24K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 24K
Hardware
- Type
- Discrete GPU
- GPU
- CMP 170HX
- VRAM
- 64 GB
- CPU
- 2x Intel Xeon E5-2630 v3
- RAM
- 64 GB
- OS
- Ubuntu 26.04.1
Engine
- Engine
- vllm
- Version
- 0.32.0rc1.dev249+g18305948a
- Quantization
- FP8 dynamic
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- ghcr.io/morrowmake/vllm-cmp170hx@sha256:5b45c4d6feafefe2eba962375b0a9a0b64fd22b9e05abb4d1966f14a32a2db79
Run
- Served model
- bench
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 8,192 tokens
Launch command
# Untuned out-of-the-box baseline. Spec decode: none # 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image vllm serve qingy2024/Reka-Flash-3-21B-FP8-dynamic --max-model-len 32768 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code
Prefill
prompt tok/s by context length8K context2,7322,997 ms TTFT
16K context2,4456,627 ms TTFT
24K context2,21210,955 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 0 | 50.465 ms TTFT | 98.6108 ms TTFT | 193.6132 ms TTFT | 373.8148 ms TTFT | 703.2232 ms TTFT | 1,012.0301 ms TTFT | Over capacity | Over capacity |
| 16K | 45.5192 ms TTFT | 84.0228 ms TTFT | 148.2303 ms TTFT | 236.4492 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 24K | 43.5216 ms TTFT | 78.3298 ms TTFT | 131.8401 ms TTFT | 197.7720 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,732.0 | — | 2,997 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 2,445.0 | — | 6,627 ms | — | — | 1 | — | — |
| Prefill | 24K | 1 | Valid | 2,212.0 | — | 10,955 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 50.4 | 50.5 | 65 ms | 19.8 ms | 1.0 | 0 | 1,511 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 98.6 | 49.4 | 108 ms | 20.2 ms | 2.0 | 0 | 2,956 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 193.6 | 48.6 | 132 ms | 20.6 ms | 4.0 | 0 | 5,804 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 373.8 | 47.1 | 148 ms | 21.2 ms | 8.0 | 0 | 11,216 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 703.2 | 44.6 | 232 ms | 22.4 ms | 16.0 | 0 | 21,104 | 30.0 s |
| Sustained decode | 0 | 32 | Valid | 1,012.0 | 32.1 | 301 ms | 31.1 ms | 32.0 | 0 | 30,336 | 30.0 s |
| Sustained decode | 0 | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 0 | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 45.5 | 45.6 | 192 ms | 21.9 ms | 1.0 | 0 | 1,364 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 84.0 | 42.1 | 228 ms | 23.7 ms | 2.0 | 0 | 2,520 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 148.2 | 37.1 | 303 ms | 27.0 ms | 4.0 | 0 | 4,444 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 236.4 | 29.5 | 492 ms | 33.9 ms | 8.0 | 0 | 7,088 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 24K | 1 | Valid | 43.5 | 43.6 | 216 ms | 23.0 ms | 1.0 | 0 | 1,304 | 30.0 s |
| Sustained decode | 24K | 2 | Valid | 78.3 | 39.2 | 298 ms | 25.5 ms | 2.0 | 0 | 2,348 | 30.0 s |
| Sustained decode | 24K | 4 | Valid | 131.8 | 33.0 | 401 ms | 30.3 ms | 4.0 | 0 | 3,952 | 30.0 s |
| Sustained decode | 24K | 8 | Valid | 197.7 | 24.5 | 720 ms | 40.8 ms | 8.0 | 0 | 5,928 | 30.0 s |
| Sustained decode | 24K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 24K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 24K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 24K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.