Speed test matrix
LFM2.5-2.6B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9418+gc26ae0439 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9418+gc26ae0439
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/tugot17/sglang
- Build
- cuda-13.2
- Commit
c26ae043924f
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_h200-liquidai-sglang-pr-31041/bin/sglang" serve --model-path LiquidAI/LFM2.5-2.6B --served-model-name lfm --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser lfm2 --context-length 131072 --mem-fraction-static 0.862142 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-algorithm DSPARK --speculative-draft-model-path LiquidAI/LFM2.5-2.6B-DSpark --speculative-draft-attention-backend flashinfer --disable-radix-cache
Prefill
prompt tok/s by context length8K context59.8K137 ms TTFT
16K context65.6K247 ms TTFT
32K context65.3K493 ms TTFT
64K context57.8K1,113 ms TTFT
123K context46.4K2,662 ms TTFT
128K context46K2,794 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 531.7123 ms TTFT | 892.7130 ms TTFT | 1,814.9124 ms TTFT | 3,107.0213 ms TTFT | 4,340.3216 ms TTFT | 5,643.7Requests queued270 ms TTFT | Not run | Not run |
| 16K | 521.7238 ms TTFT | 773.0245 ms TTFT | 1,622.9249 ms TTFT | 2,625.4299 ms TTFT | 3,829.8383 ms TTFT | 3,850.3Requests queued1,165 ms TTFT | Not run | Not run |
| 32K | 453.7490 ms TTFT | 569.6508 ms TTFT | 1,287.8504 ms TTFT | 1,906.5Requests queued618 ms TTFT | 1,914.8Requests queued1,954 ms TTFT | 2,575.97,059 ms TTFT | Not run | Over capacity |
| 64K | 342.41,104 ms TTFT | 385.6Requests queued1,612 ms TTFT | 886.41,603 ms TTFT | 1,109.4Requests queued1,984 ms TTFT | 1,117.38,691 ms TTFT | 1,660.816,654 ms TTFT | Not run | Over capacity |
| 123K | 239.12,648 ms TTFT | 179.54,065 ms TTFT | 489.1Requests queued4,080 ms TTFT | 520.411,485 ms TTFT | 549.221,330 ms TTFT | 430.0Requests queued26,308 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 59.8K | — | 137 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 65.6K | — | 247 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 65.3K | — | 493 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 57.8K | — | 1,113 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 46.4K | — | 2,662 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 46K | — | 2,794 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 531.7 | 552.5 | 123 ms | 1.8 ms | 0.9 | 4 | 15,950 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 892.7 | 464.6 | 130 ms | 2.2 ms | 2.4 | 6 | 26,779 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,814.9 | 456.7 | 124 ms | 2.2 ms | 4.0 | 12 | 54,446 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 3,107.0 | 392.8 | 213 ms | 2.5 ms | 7.9 | 24 | 93,199 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,340.3 | 278.5 | 216 ms | 3.6 ms | 17.4 | 32 | 130,199 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 5,643.7 | 161.0 | 270 ms | 6.2 ms | 31.6 | 53 | 169,128 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 521.7 | 559.5 | 238 ms | 1.8 ms | 0.9 | 4 | 15,650 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 773.0 | 479.0 | 245 ms | 2.1 ms | 9.0 | 6 | 23,189 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,622.9 | 420.0 | 249 ms | 2.4 ms | 4.2 | 12 | 48,684 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,625.4 | 329.5 | 299 ms | 3.0 ms | 8.0 | 16 | 78,749 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,829.8 | 239.9 | 383 ms | 4.2 ms | 15.9 | 20 | 114,874 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 3,850.3 | 110.6 | 1,165 ms | 9.0 ms | 34.1 | 25 | 115,498 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 453.7 | 477.3 | 490 ms | 2.1 ms | 0.9 | 3 | 13,611 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 569.6 | 407.2 | 508 ms | 2.5 ms | 14.6 | 4 | 17,087 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,287.8 | 338.7 | 504 ms | 3.0 ms | 4.0 | 8 | 38,629 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 1,906.5 | 252.7 | 618 ms | 4.0 ms | 7.8 | 13 | 57,129 | 30.0 s |
| Sustained decode | 32K | 16 | Requests queued | 1,914.8 | 113.4 | 1,954 ms | 8.8 ms | 19.2 | 11 | 57,313 | 29.9 s |
| Sustained decode | 32K | 32 | Valid | 2,575.9 | 71.6 | 7,059 ms | 14.0 ms | 43.0 | 0 | 77,217 | 30.0 s |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 342.4 | 372.2 | 1,104 ms | 2.7 ms | 0.9 | 2 | 10,273 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 385.6 | 223.7 | 1,612 ms | 4.9 ms | 14.7 | 2 | 11,566 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 886.4 | 222.7 | 1,603 ms | 4.5 ms | 4.1 | 4 | 26,588 | 30.0 s |
| Sustained decode | 64K | 8 | Requests queued | 1,109.4 | 135.1 | 1,984 ms | 7.4 ms | 7.7 | 8 | 33,273 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,117.3 | 60.2 | 8,691 ms | 16.6 ms | 24.0 | 0 | 33,496 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,660.8 | 38.0 | 16,654 ms | 26.3 ms | 34.9 | 0 | 49,797 | 30.0 s |
| Sustained decode | 64K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 239.1 | 313.6 | 2,648 ms | 3.2 ms | 0.9 | 2 | 7,051 | 29.5 s |
| Sustained decode | 123K | 2 | Valid | 179.5 | 81.7 | 4,065 ms | 12.2 ms | 20.6 | 0 | 5,385 | 30.0 s |
| Sustained decode | 123K | 4 | Requests queued | 489.1 | 130.8 | 4,080 ms | 7.6 ms | 3.9 | 3 | 14,213 | 29.1 s |
| Sustained decode | 123K | 8 | Valid | 520.4 | 52.9 | 11,485 ms | 18.9 ms | 11.0 | 0 | 15,612 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 549.2 | 22.3 | 21,330 ms | 44.8 ms | 26.8 | 0 | 16,462 | 30.0 s |
| Sustained decode | 123K | 32 | Requests queued | 430.0 | 17.8 | 26,308 ms | 56.2 ms | 36.0 | 0 | 12,871 | 29.9 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.