Speed test matrix
LFM2.5-230M on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-liquidai-vllm/bin/vllm" serve LiquidAI/LFM2.5-230M --served-model-name lfm --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser lfm2 --max-model-len 128000 --gpu-memory-utilization 0.881103 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context162.6K50 ms TTFT
16K context151.4K107 ms TTFT
32K context177.3K182 ms TTFT
64K context192.2K336 ms TTFT
120K context142.2K849 ms TTFT
125K context136.2K924 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 910.539 ms TTFT | 1,699.254 ms TTFT | 3,087.156 ms TTFT | 5,245.747 ms TTFT | 8,109.886 ms TTFT | 11.1K113 ms TTFT | 13.3K296 ms TTFT | 12.9K844 ms TTFT |
| 16K | 901.973 ms TTFT | 1,672.577 ms TTFT | 3,039.180 ms TTFT | 5,171.485 ms TTFT | 7,836.2109 ms TTFT | 10.4K114 ms TTFT | 12.3K364 ms TTFT | 13.3K1,317 ms TTFT |
| 32K | 864.593 ms TTFT | 1,599.5111 ms TTFT | 2,764.0117 ms TTFT | 4,540.9112 ms TTFT | 6,154.1122 ms TTFT | 7,600.5351 ms TTFT | 8,337.2515 ms TTFT | 9,564.92,372 ms TTFT |
| 64K | 788.2145 ms TTFT | 1,374.9179 ms TTFT | 2,202.1185 ms TTFT | 3,175.2181 ms TTFT | 3,933.0205 ms TTFT | 4,508.0468 ms TTFT | 5,047.82,551 ms TTFT | Over capacity |
| 120K | 679.7264 ms TTFT | 1,114.5310 ms TTFT | 1,615.1327 ms TTFT | 2,076.1330 ms TTFT | 2,415.2394 ms TTFT | 2,737.52,571 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 162.6K | — | 50 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 151.4K | — | 107 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 177.3K | — | 182 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 192.2K | — | 336 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 142.2K | — | 849 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 136.2K | — | 924 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 910.5 | 925.1 | 39 ms | 1.1 ms | 1.0 | 7 | 27,314 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 1,699.2 | 861.0 | 54 ms | 1.2 ms | 2.0 | 14 | 50,976 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 3,087.1 | 784.8 | 56 ms | 1.3 ms | 4.0 | 24 | 92,613 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 5,245.7 | 661.9 | 47 ms | 1.5 ms | 8.0 | 40 | 157,383 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 8,109.8 | 513.3 | 86 ms | 1.9 ms | 16.0 | 64 | 243,287 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 11.1K | 363.3 | 113 ms | 2.8 ms | 31.9 | 96 | 334,045 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 13.3K | 224.8 | 296 ms | 4.4 ms | 63.8 | 64 | 400,202 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 12.9K | 113.5 | 844 ms | 8.8 ms | 128.0 | 0 | 387,732 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 901.9 | 917.4 | 73 ms | 1.1 ms | 1.0 | 7 | 27,058 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 1,672.5 | 852.4 | 77 ms | 1.2 ms | 2.0 | 14 | 50,205 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 3,039.1 | 771.5 | 80 ms | 1.3 ms | 3.9 | 24 | 91,174 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 5,171.4 | 655.4 | 85 ms | 1.5 ms | 8.0 | 40 | 155,140 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 7,836.2 | 501.1 | 109 ms | 2.0 ms | 15.7 | 64 | 235,094 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 10.4K | 330.6 | 114 ms | 3.0 ms | 31.9 | 64 | 311,702 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 12.3K | 214.0 | 364 ms | 4.7 ms | 63.8 | 64 | 369,009 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 13.3K | 107.1 | 1,317 ms | 9.3 ms | 128.0 | 0 | 397,662 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 864.5 | 884.8 | 93 ms | 1.1 ms | 1.0 | 7 | 25,935 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 1,599.5 | 818.8 | 111 ms | 1.2 ms | 2.0 | 12 | 47,988 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 2,764.0 | 708.5 | 117 ms | 1.4 ms | 3.9 | 20 | 82,917 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 4,540.9 | 576.6 | 112 ms | 1.7 ms | 7.8 | 32 | 136,228 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 6,154.1 | 391.3 | 122 ms | 2.6 ms | 15.9 | 48 | 184,627 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 7,600.5 | 239.2 | 351 ms | 4.2 ms | 31.9 | 32 | 228,020 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 8,337.2 | 132.4 | 515 ms | 7.6 ms | 63.2 | 64 | 250,086 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 9,564.9 | 74.2 | 2,372 ms | 13.5 ms | 128.0 | 0 | 286,960 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 788.2 | 811.3 | 145 ms | 1.2 ms | 1.0 | 6 | 23,646 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 1,374.9 | 709.0 | 179 ms | 1.4 ms | 1.9 | 10 | 41,245 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 2,202.1 | 565.2 | 185 ms | 1.8 ms | 3.8 | 16 | 66,061 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 3,175.2 | 404.7 | 181 ms | 2.5 ms | 7.8 | 24 | 95,258 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,933.0 | 250.4 | 205 ms | 4.0 ms | 15.8 | 32 | 117,991 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,508.0 | 142.2 | 468 ms | 7.0 ms | 31.7 | 32 | 135,227 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 5,047.8 | 78.6 | 2,551 ms | 12.7 ms | 64.0 | 0 | 151,424 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 679.7 | 710.1 | 264 ms | 1.4 ms | 1.0 | 5 | 20,400 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 1,114.5 | 582.9 | 310 ms | 1.7 ms | 2.0 | 8 | 33,436 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 1,615.1 | 418.2 | 327 ms | 2.4 ms | 3.9 | 12 | 48,451 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 2,076.1 | 266.0 | 330 ms | 3.8 ms | 7.9 | 16 | 62,281 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 2,415.2 | 151.8 | 394 ms | 6.6 ms | 15.8 | 16 | 72,448 | 30.0 s |
| Sustained decode | 120K | 32 | Valid | 2,737.5 | 84.7 | 2,571 ms | 11.8 ms | 32.0 | 0 | 82,112 | 30.0 s |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.