Speed test matrix
LFM2.5-1.2B-Instruct on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-liquidai-vllm/bin/vllm" serve LiquidAI/LFM2.5-1.2B-Instruct --served-model-name lfm --trust-remote-code --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser lfm2 --max-model-len 128000 --gpu-memory-utilization 0.888088 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context105.2K78 ms TTFT
16K context108.8K149 ms TTFT
32K context105.4K307 ms TTFT
64K context99.3K649 ms TTFT
120K context77K1,568 ms TTFT
125K context73.4K1,715 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 684.451 ms TTFT | 1,328.351 ms TTFT | 2,545.256 ms TTFT | 4,687.657 ms TTFT | 8,077.685 ms TTFT | 12.6K168 ms TTFT | 17.4K187 ms TTFT | 22.2K543 ms TTFT |
| 16K | 669.862 ms TTFT | 1,284.379 ms TTFT | 2,381.391 ms TTFT | 4,225.4104 ms TTFT | 6,824.4119 ms TTFT | 9,881.0156 ms TTFT | 12.7K403 ms TTFT | 14.9K1,231 ms TTFT |
| 32K | 644.895 ms TTFT | 1,205.4127 ms TTFT | 2,142.4128 ms TTFT | 3,540.7133 ms TTFT | 5,254.6224 ms TTFT | 6,896.8247 ms TTFT | 7,988.9499 ms TTFT | 9,324.22,417 ms TTFT |
| 64K | 604.0144 ms TTFT | 1,077.9180 ms TTFT | 1,795.9209 ms TTFT | 2,661.3230 ms TTFT | 3,585.3330 ms TTFT | 4,232.1456 ms TTFT | 4,887.22,518 ms TTFT | 5,216.34,812 ms TTFT |
| 120K | 538.0261 ms TTFT | 915.2263 ms TTFT | 1,406.0286 ms TTFT | 1,899.8346 ms TTFT | 2,243.8397 ms TTFT | 2,651.52,507 ms TTFT | 2,817.25,077 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 105.2K | — | 78 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 108.8K | — | 149 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 105.4K | — | 307 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 99.3K | — | 649 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 77K | — | 1,568 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 73.4K | — | 1,715 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 684.4 | 690.6 | 51 ms | 1.4 ms | 1.0 | 5 | 20,532 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 1,328.3 | 670.3 | 51 ms | 1.5 ms | 2.0 | 10 | 39,850 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 2,545.2 | 642.4 | 56 ms | 1.6 ms | 4.0 | 20 | 76,358 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 4,687.6 | 591.7 | 57 ms | 1.7 ms | 7.9 | 32 | 140,630 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 8,077.6 | 510.8 | 85 ms | 2.0 ms | 16.0 | 64 | 242,331 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 12.6K | 401.5 | 168 ms | 2.5 ms | 31.5 | 96 | 377,270 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 17.4K | 275.8 | 187 ms | 3.6 ms | 63.5 | 128 | 521,850 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 22.2K | 179.8 | 543 ms | 5.6 ms | 126.5 | 128 | 666,738 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 669.8 | 676.6 | 62 ms | 1.5 ms | 1.0 | 5 | 20,095 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 1,284.3 | 651.4 | 79 ms | 1.5 ms | 1.9 | 10 | 38,528 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 2,381.3 | 604.6 | 91 ms | 1.7 ms | 4.0 | 20 | 71,439 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 4,225.4 | 536.2 | 104 ms | 1.9 ms | 7.8 | 32 | 126,765 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 6,824.4 | 432.4 | 119 ms | 2.3 ms | 15.7 | 48 | 204,735 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 9,881.0 | 311.5 | 156 ms | 3.2 ms | 31.5 | 64 | 296,424 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 12.7K | 200.9 | 403 ms | 5.0 ms | 63.6 | 64 | 380,733 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 14.9K | 117.8 | 1,231 ms | 8.5 ms | 127.6 | 37 | 447,869 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 644.8 | 655.2 | 95 ms | 1.5 ms | 1.0 | 5 | 19,345 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 1,205.4 | 615.7 | 127 ms | 1.6 ms | 2.0 | 10 | 36,161 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 2,142.4 | 545.3 | 128 ms | 1.8 ms | 4.0 | 16 | 64,272 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 3,540.7 | 449.7 | 133 ms | 2.2 ms | 7.8 | 24 | 106,221 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 5,254.6 | 333.3 | 224 ms | 3.0 ms | 16.0 | 32 | 157,637 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 6,896.8 | 216.7 | 247 ms | 4.6 ms | 31.9 | 32 | 206,905 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 7,988.9 | 127.1 | 499 ms | 7.9 ms | 63.0 | 64 | 239,657 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 9,324.2 | 72.4 | 2,417 ms | 13.8 ms | 128.0 | 0 | 279,681 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 604.0 | 616.8 | 144 ms | 1.6 ms | 1.0 | 4 | 18,121 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 1,077.9 | 552.9 | 180 ms | 1.8 ms | 1.9 | 8 | 32,337 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,795.9 | 459.0 | 209 ms | 2.2 ms | 4.0 | 12 | 53,878 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,661.3 | 337.5 | 230 ms | 3.0 ms | 7.9 | 16 | 79,839 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,585.3 | 225.3 | 330 ms | 4.4 ms | 15.9 | 16 | 107,549 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 4,232.1 | 133.4 | 456 ms | 7.5 ms | 31.7 | 32 | 126,964 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,887.2 | 75.8 | 2,518 ms | 13.2 ms | 64.0 | 0 | 146,607 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 5,216.3 | 41.0 | 4,812 ms | 24.4 ms | 128.0 | 0 | 156,499 | 30.0 s |
| Sustained decode | 120K | 1 | Valid | 538.0 | 557.2 | 261 ms | 1.8 ms | 1.0 | 4 | 16,141 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 915.2 | 470.3 | 263 ms | 2.1 ms | 2.0 | 6 | 27,456 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 1,406.0 | 359.1 | 286 ms | 2.8 ms | 3.9 | 8 | 42,180 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 1,899.8 | 239.7 | 346 ms | 4.2 ms | 7.9 | 8 | 56,992 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 2,243.8 | 141.9 | 397 ms | 7.0 ms | 15.8 | 16 | 67,310 | 30.0 s |
| Sustained decode | 120K | 32 | Valid | 2,651.5 | 82.2 | 2,507 ms | 12.2 ms | 32.0 | 0 | 79,551 | 30.0 s |
| Sustained decode | 120K | 64 | Valid | 2,817.2 | 44.1 | 5,077 ms | 22.7 ms | 64.0 | 0 | 84,497 | 30.0 s |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.