Speed test matrix
LFM2.5-350M on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-liquidai-sglang/bin/sglang" serve --model-path LiquidAI/LFM2.5-350M --served-model-name lfm --trust-remote-code --tp 1 --tool-call-parser lfm2 --context-length 128000 --mem-fraction-static 0.865156 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context114.3K72 ms TTFT
16K context165.8K98 ms TTFT
32K context189.8K170 ms TTFT
64K context173.3K372 ms TTFT
120K context139K869 ms TTFT
125K context136K925 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 16K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 32K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 64K | 571.9165 ms TTFT | 1,005.2171 ms TTFT | Looped | Looped | Looped | Looped | Looped | Over capacity |
| 120K | 511.5252 ms TTFT | 865.5345 ms TTFT | 1,334.7425 ms TTFT | 1,827.5828 ms TTFT | 2,213.71,489 ms TTFT | 2,584.45,234 ms TTFT | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 114.3K | — | 72 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 165.8K | — | 98 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 189.8K | — | 170 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 173.3K | — | 372 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 139K | — | 869 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 136K | — | 925 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Valid | 571.9 | 584.9 | 165 ms | 1.7 ms | 1.0 | 4 | 17,158 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 1,005.2 | 514.5 | 171 ms | 1.9 ms | 2.0 | 8 | 30,156 | 30.0 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 511.5 | 530.9 | 252 ms | 1.9 ms | 1.0 | 4 | 15,344 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 865.5 | 449.8 | 345 ms | 2.2 ms | 1.9 | 6 | 25,965 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 1,334.7 | 343.4 | 425 ms | 2.9 ms | 3.9 | 8 | 40,041 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 1,827.5 | 233.3 | 828 ms | 4.3 ms | 7.9 | 8 | 54,820 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 2,213.7 | 142.7 | 1,489 ms | 7.0 ms | 15.8 | 16 | 66,404 | 30.0 s |
| Sustained decode | 120K | 32 | Valid | 2,584.4 | 84.9 | 5,234 ms | 11.8 ms | 32.0 | 0 | 77,535 | 30.0 s |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.