Speed test matrix
LFM2.5-350M on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-liquidai-vllm/bin/vllm" serve LiquidAI/LFM2.5-350M --served-model-name lfm --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser lfm2 --max-model-len 128000 --gpu-memory-utilization 0.881068 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context143.1K57 ms TTFT
16K context140.1K116 ms TTFT
32K context164.5K196 ms TTFT
64K context171.8K375 ms TTFT
120K context116.6K1,036 ms TTFT
125K context130.8K962 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 959.947 ms TTFT | 1,819.248 ms TTFT | 3,419.654 ms TTFT | 6,317.355 ms TTFT | 10.3K60 ms TTFT | 15K161 ms TTFT | 19.9K290 ms TTFT | 24.3K521 ms TTFT |
| 16K | 928.870 ms TTFT | 1,722.279 ms TTFT | 3,144.084 ms TTFT | 5,464.591 ms TTFT | 8,357.3110 ms TTFT | 11.3K136 ms TTFT | 14K432 ms TTFT | 15.6K404 ms TTFT |
| 32K | 878.589 ms TTFT | 1,599.299 ms TTFT | 2,728.4115 ms TTFT | 4,345.0117 ms TTFT | 6,070.1182 ms TTFT | 7,587.7394 ms TTFT | 8,650.1337 ms TTFT | 9,668.82,370 ms TTFT |
| 64K | 799.4156 ms TTFT | 1,382.1191 ms TTFT | 2,173.5175 ms TTFT | 3,081.9183 ms TTFT | 3,905.2167 ms TTFT | 4,504.5668 ms TTFT | 5,071.12,550 ms TTFT | 5,317.84,579 ms TTFT |
| 120K | 687.9443 ms TTFT | 1,113.1Requests queued417 ms TTFT | 1,585.8522 ms TTFT | 2,004.5587 ms TTFT | 2,382.91,212 ms TTFT | 2,747.53,757 ms TTFT | 2,886.08,393 ms TTFT | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 143.1K | — | 57 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 140.1K | — | 116 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 164.5K | — | 196 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 171.8K | — | 375 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 116.6K | — | 1,036 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 130.8K | — | 962 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 959.9 | 972.0 | 47 ms | 1.0 ms | 1.0 | 7 | 28,796 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 1,819.2 | 919.9 | 48 ms | 1.1 ms | 2.0 | 14 | 54,577 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 3,419.6 | 866.9 | 54 ms | 1.2 ms | 4.0 | 28 | 102,590 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 6,317.3 | 800.2 | 55 ms | 1.2 ms | 7.8 | 48 | 189,524 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 10.3K | 652.1 | 60 ms | 1.5 ms | 15.7 | 80 | 309,008 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 15K | 478.4 | 161 ms | 2.1 ms | 32.0 | 96 | 449,512 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 19.9K | 318.1 | 290 ms | 3.1 ms | 62.8 | 128 | 596,716 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 24.3K | 195.1 | 521 ms | 5.1 ms | 126.2 | 128 | 739,238 | 30.5 s |
| Sustained decode | 16K | 1 | Valid | 928.8 | 944.6 | 70 ms | 1.1 ms | 1.0 | 7 | 27,864 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 1,722.2 | 878.8 | 79 ms | 1.1 ms | 2.0 | 14 | 51,666 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 3,144.0 | 798.6 | 84 ms | 1.3 ms | 3.9 | 24 | 94,323 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 5,464.5 | 693.6 | 91 ms | 1.4 ms | 8.0 | 40 | 163,937 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 8,357.3 | 530.2 | 110 ms | 1.9 ms | 15.9 | 64 | 250,790 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 11.3K | 359.0 | 136 ms | 2.8 ms | 32.0 | 64 | 339,675 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 14K | 222.5 | 432 ms | 4.5 ms | 63.5 | 64 | 420,159 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 15.6K | 124.8 | 404 ms | 8.0 ms | 126.7 | 128 | 467,845 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 878.5 | 898.0 | 89 ms | 1.1 ms | 1.0 | 7 | 26,372 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 1,599.2 | 815.8 | 99 ms | 1.2 ms | 2.0 | 12 | 47,974 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 2,728.4 | 695.8 | 115 ms | 1.4 ms | 4.0 | 20 | 81,851 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 4,345.0 | 551.9 | 117 ms | 1.8 ms | 7.7 | 32 | 130,350 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 6,070.1 | 387.6 | 182 ms | 2.6 ms | 15.9 | 48 | 182,108 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 7,587.7 | 239.7 | 394 ms | 4.2 ms | 31.9 | 32 | 227,639 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 8,650.1 | 136.5 | 337 ms | 7.3 ms | 63.6 | 64 | 259,479 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 9,668.8 | 75.8 | 2,370 ms | 13.2 ms | 128.0 | 0 | 290,050 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 799.4 | 824.8 | 156 ms | 1.2 ms | 1.0 | 6 | 23,983 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 1,382.1 | 714.5 | 191 ms | 1.4 ms | 2.0 | 10 | 41,461 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 2,173.5 | 556.7 | 175 ms | 1.8 ms | 3.9 | 16 | 65,209 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 3,081.9 | 393.3 | 183 ms | 2.5 ms | 7.9 | 24 | 92,459 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,905.2 | 247.7 | 167 ms | 4.0 ms | 15.7 | 32 | 117,160 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,504.5 | 142.6 | 668 ms | 7.0 ms | 31.5 | 32 | 135,138 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 5,071.1 | 78.8 | 2,550 ms | 12.7 ms | 64.0 | 0 | 152,124 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 5,317.8 | 41.9 | 4,579 ms | 23.9 ms | 128.0 | 0 | 159,600 | 30.0 s |
| Sustained decode | 120K | 1 | Valid | 687.9 | 745.9 | 443 ms | 1.3 ms | 0.9 | 5 | 20,637 | 30.0 s |
| Sustained decode | 120K | 2 | Requests queued | 1,113.1 | 588.1 | 417 ms | 1.7 ms | 1.9 | 8 | 33,392 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 1,585.8 | 419.3 | 522 ms | 2.4 ms | 3.8 | 12 | 47,575 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 2,004.5 | 261.4 | 587 ms | 3.8 ms | 7.7 | 16 | 60,130 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 2,382.9 | 154.1 | 1,212 ms | 6.5 ms | 15.6 | 16 | 71,493 | 30.0 s |
| Sustained decode | 120K | 32 | Valid | 2,747.5 | 86.5 | 3,757 ms | 11.6 ms | 32.0 | 0 | 82,424 | 30.0 s |
| Sustained decode | 120K | 64 | Valid | 2,886.0 | 47.4 | 8,393 ms | 21.1 ms | 64.0 | 0 | 86,571 | 30.0 s |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.