Speed test matrix
LFM2.5-VL-3B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF163 contexts (8K–27K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 27K, 32K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- lfm-vl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-liquidai-vllm/bin/vllm" serve LiquidAI/LFM2.5-VL-3B --served-model-name lfm-vl --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser lfm2 --max-model-len 32768 --gpu-memory-utilization 0.880400 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context69.4K118 ms TTFT
16K context65.4K249 ms TTFT
27K context64.7K422 ms TTFT
32K context60.5K535 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 349.051 ms TTFT | 670.360 ms TTFT | 1,302.163 ms TTFT | 2,479.082 ms TTFT | 4,422.7130 ms TTFT | 7,362.7247 ms TTFT | 11.2K438 ms TTFT | 14.9K868 ms TTFT |
| 16K | 343.986 ms TTFT | 655.086 ms TTFT | 1,245.6105 ms TTFT | 2,281.8169 ms TTFT | 3,866.1232 ms TTFT | 6,040.0437 ms TTFT | 8,311.1410 ms TTFT | 10.7K1,338 ms TTFT |
| 27K | 336.8115 ms TTFT | 635.1136 ms TTFT | 1,175.8172 ms TTFT | 2,070.9255 ms TTFT | 3,357.3342 ms TTFT | 4,844.3422 ms TTFT | 6,391.41,287 ms TTFT | 7,621.62,066 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 69.4K | — | 118 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 65.4K | — | 249 ms | — | — | 1 | — | — |
| Prefill | 27K | 1 | Valid | 64.7K | — | 422 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 60.5K | — | 535 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 349.0 | 350.2 | 51 ms | 2.9 ms | 1.0 | 2 | 10,471 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 670.3 | 336.8 | 60 ms | 3.0 ms | 2.0 | 4 | 20,107 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,302.1 | 327.6 | 63 ms | 3.1 ms | 4.0 | 8 | 39,061 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,479.0 | 312.4 | 82 ms | 3.2 ms | 8.0 | 16 | 74,372 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,422.7 | 279.7 | 130 ms | 3.6 ms | 16.0 | 32 | 132,681 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 7,362.7 | 233.5 | 247 ms | 4.3 ms | 32.0 | 32 | 220,875 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 11.2K | 178.6 | 438 ms | 5.6 ms | 62.1 | 64 | 334,705 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 14.9K | 117.4 | 868 ms | 8.5 ms | 128.0 | 0 | 445,819 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 343.9 | 346.1 | 86 ms | 2.9 ms | 1.0 | 2 | 10,317 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 655.0 | 330.3 | 86 ms | 3.0 ms | 2.0 | 4 | 19,650 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,245.6 | 314.2 | 105 ms | 3.2 ms | 4.0 | 8 | 37,366 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,281.8 | 288.8 | 169 ms | 3.5 ms | 8.0 | 16 | 68,454 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,866.1 | 245.8 | 232 ms | 4.1 ms | 16.0 | 32 | 115,985 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 6,040.0 | 191.8 | 437 ms | 5.2 ms | 31.7 | 32 | 181,184 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 8,311.1 | 131.6 | 410 ms | 7.6 ms | 62.9 | 64 | 249,325 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 10.7K | 83.3 | 1,338 ms | 12.0 ms | 128.0 | 0 | 320,476 | 30.0 s |
| Sustained decode | 27K | 1 | Valid | 336.8 | 339.4 | 115 ms | 2.9 ms | 1.0 | 2 | 10,103 | 30.0 s |
| Sustained decode | 27K | 2 | Valid | 635.1 | 320.9 | 136 ms | 3.1 ms | 2.0 | 4 | 19,052 | 30.0 s |
| Sustained decode | 27K | 4 | Valid | 1,175.8 | 297.9 | 172 ms | 3.4 ms | 3.9 | 8 | 35,276 | 30.0 s |
| Sustained decode | 27K | 8 | Valid | 2,070.9 | 263.1 | 255 ms | 3.8 ms | 7.7 | 16 | 62,124 | 30.0 s |
| Sustained decode | 27K | 16 | Valid | 3,357.3 | 212.9 | 342 ms | 4.7 ms | 15.6 | 16 | 100,720 | 30.0 s |
| Sustained decode | 27K | 32 | Valid | 4,844.3 | 153.4 | 422 ms | 6.5 ms | 32.0 | 32 | 145,332 | 30.0 s |
| Sustained decode | 27K | 64 | Valid | 6,391.4 | 99.8 | 1,287 ms | 10.0 ms | 64.0 | 0 | 191,721 | 30.0 s |
| Sustained decode | 27K | 128 | Valid | 7,621.6 | 59.0 | 2,066 ms | 16.9 ms | 128.0 | 0 | 228,608 | 30.0 s |