Speed test matrix
LFM2.5-2.6B on H200 NVL 141GB
vllm · 0.26.1rc1.dev638+g52be12cfa · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.26.1rc1.dev638+g52be12cfa
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
52be12cfac0c
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-liquidai-vllm/bin/vllm" serve LiquidAI/LFM2.5-2.6B --served-model-name lfm --trust-remote-code --tensor-parallel-size 1 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser lfm2 --max-model-len 131072 --gpu-memory-utilization 0.882025 --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context65.4K125 ms TTFT
16K context66.2K245 ms TTFT
32K context65K496 ms TTFT
64K context59.5K1,080 ms TTFT
123K context47.7K2,588 ms TTFT
128K context47.1K2,725 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 343.853 ms TTFT | 663.853 ms TTFT | 1,282.672 ms TTFT | 2,426.687 ms TTFT | 4,288.6143 ms TTFT | 7,042.7249 ms TTFT | 10.3K370 ms TTFT | 12.7K842 ms TTFT |
| 16K | 339.183 ms TTFT | 649.794 ms TTFT | 1,234.5118 ms TTFT | 2,255.6146 ms TTFT | 3,799.7212 ms TTFT | 5,835.8448 ms TTFT | 7,743.5499 ms TTFT | 10.1K1,228 ms TTFT |
| 32K | 330.9105 ms TTFT | 624.2119 ms TTFT | 1,143.0153 ms TTFT | 1,968.6234 ms TTFT | 3,107.5257 ms TTFT | 4,281.0497 ms TTFT | 5,563.61,373 ms TTFT | 6,482.72,480 ms TTFT |
| 64K | 316.0159 ms TTFT | 576.1204 ms TTFT | 996.7216 ms TTFT | 1,599.3275 ms TTFT | 2,267.9399 ms TTFT | 2,936.91,430 ms TTFT | 3,417.52,269 ms TTFT | Over capacity |
| 123K | 290.4268 ms TTFT | 501.5341 ms TTFT | 818.7316 ms TTFT | 1,162.5455 ms TTFT | 1,543.01,520 ms TTFT | 1,819.32,911 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 65.4K | — | 125 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 66.2K | — | 245 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 65K | — | 496 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 59.5K | — | 1,080 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 47.7K | — | 2,588 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 47.1K | — | 2,725 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 343.8 | 345.1 | 53 ms | 2.9 ms | 1.0 | 2 | 10,319 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 663.8 | 333.4 | 53 ms | 3.0 ms | 2.0 | 4 | 19,912 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,282.6 | 322.3 | 72 ms | 3.1 ms | 4.0 | 8 | 38,475 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,426.6 | 305.3 | 87 ms | 3.3 ms | 8.0 | 16 | 72,800 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,288.6 | 271.3 | 143 ms | 3.7 ms | 16.0 | 32 | 128,654 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 7,042.7 | 223.3 | 249 ms | 4.5 ms | 31.0 | 32 | 211,268 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 10.3K | 165.5 | 370 ms | 6.0 ms | 62.4 | 64 | 308,540 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 12.7K | 101.5 | 842 ms | 9.9 ms | 128.0 | 0 | 381,952 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 339.1 | 341.2 | 83 ms | 2.9 ms | 1.0 | 2 | 10,177 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 649.7 | 327.2 | 94 ms | 3.1 ms | 2.0 | 4 | 19,492 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,234.5 | 310.9 | 118 ms | 3.2 ms | 4.0 | 8 | 37,034 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,255.6 | 284.9 | 146 ms | 3.5 ms | 7.7 | 16 | 67,669 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,799.7 | 240.1 | 212 ms | 4.2 ms | 16.0 | 17 | 113,990 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,835.8 | 185.5 | 448 ms | 5.4 ms | 31.2 | 32 | 175,061 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,743.5 | 123.7 | 499 ms | 8.1 ms | 62.7 | 64 | 232,317 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 10.1K | 78.5 | 1,228 ms | 12.7 ms | 128.0 | 0 | 302,208 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 330.9 | 333.4 | 105 ms | 3.0 ms | 1.0 | 2 | 9,929 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 624.2 | 315.0 | 119 ms | 3.2 ms | 2.0 | 4 | 18,728 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,143.0 | 288.6 | 153 ms | 3.5 ms | 4.0 | 8 | 34,290 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,968.6 | 250.4 | 234 ms | 4.0 ms | 7.8 | 16 | 59,058 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,107.5 | 195.9 | 257 ms | 5.1 ms | 16.0 | 16 | 93,218 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,281.0 | 135.5 | 497 ms | 7.4 ms | 31.6 | 32 | 128,432 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,563.6 | 86.6 | 1,373 ms | 11.6 ms | 64.0 | 0 | 166,912 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,482.7 | 50.0 | 2,480 ms | 20.0 ms | 128.0 | 0 | 194,432 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 316.0 | 319.4 | 159 ms | 3.1 ms | 1.0 | 2 | 9,481 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 576.1 | 291.7 | 204 ms | 3.4 ms | 2.0 | 4 | 17,281 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 996.7 | 252.8 | 216 ms | 4.0 ms | 4.0 | 8 | 29,899 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,599.3 | 201.1 | 275 ms | 5.0 ms | 7.9 | 8 | 48,011 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,267.9 | 143.1 | 399 ms | 7.0 ms | 15.8 | 16 | 68,034 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,936.9 | 91.0 | 1,430 ms | 11.0 ms | 32.0 | 0 | 88,096 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,417.5 | 52.8 | 2,269 ms | 18.9 ms | 64.0 | 0 | 102,529 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 290.4 | 296.3 | 268 ms | 3.4 ms | 1.0 | 2 | 8,713 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 501.5 | 257.2 | 341 ms | 3.9 ms | 2.0 | 4 | 15,044 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 818.7 | 206.4 | 316 ms | 4.8 ms | 4.0 | 4 | 24,561 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 1,162.5 | 147.3 | 455 ms | 6.8 ms | 7.9 | 8 | 34,873 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,543.0 | 95.5 | 1,520 ms | 10.5 ms | 16.0 | 0 | 46,288 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,819.3 | 56.2 | 2,911 ms | 17.8 ms | 32.0 | 0 | 54,560 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.