Speed test matrix
LFM2.5-VL-3B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF163 contexts (8K–27K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 27K, 32K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- lfm-vl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_USE_CUDA_IPC_TRANSPORT=1 SGLANG_USE_IPC_POOL_HANDLE_CACHE=1 "$HOME/env_h200-liquidai-sglang/bin/sglang" serve --model-path LiquidAI/LFM2.5-VL-3B --served-model-name lfm-vl --trust-remote-code --tp 1 --tool-call-parser lfm2 --context-length 32768 --mem-fraction-static 0.882265 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context52.7K156 ms TTFT
16K context61.9K263 ms TTFT
27K context61.7K443 ms TTFT
32K context65.3K496 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 334.365 ms TTFT | 632.347 ms TTFT | 1,211.185 ms TTFT | 2,316.7125 ms TTFT | 4,089.7306 ms TTFT | 6,669.2457 ms TTFT | 10.5K1,047 ms TTFT | 11.6K2,245 ms TTFT |
| 16K | 326.4161 ms TTFT | 567.0219 ms TTFT | 1,106.9259 ms TTFT | 2,084.8292 ms TTFT | 3,434.5472 ms TTFT | 5,823.3911 ms TTFT | 7,475.72,252 ms TTFT | 10.3K4,610 ms TTFT |
| 27K | 321.8143 ms TTFT | 473.6201 ms TTFT | 1,094.7225 ms TTFT | 1,877.7542 ms TTFT | 3,169.9921 ms TTFT | 4,392.01,711 ms TTFT | 4,616.54,029 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 52.7K | — | 156 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 61.9K | — | 263 ms | — | — | 1 | — | — |
| Prefill | 27K | 1 | Valid | 61.7K | — | 443 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 65.3K | — | 496 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 334.3 | 335.7 | 65 ms | 3.0 ms | 1.0 | 2 | 10,029 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 632.3 | 316.6 | 47 ms | 3.2 ms | 2.0 | 4 | 18,968 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,211.1 | 304.2 | 85 ms | 3.3 ms | 4.0 | 8 | 36,334 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,316.7 | 292.0 | 125 ms | 3.4 ms | 8.0 | 16 | 69,504 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,089.7 | 263.0 | 306 ms | 3.8 ms | 16.6 | 32 | 122,695 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 6,669.2 | 212.6 | 457 ms | 4.7 ms | 37.9 | 32 | 200,063 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 10.5K | 168.0 | 1,047 ms | 6.0 ms | 63.1 | 64 | 315,081 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 11.6K | 92.1 | 2,245 ms | 10.9 ms | 175.3 | 0 | 347,200 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 326.4 | 330.3 | 161 ms | 3.0 ms | 1.0 | 2 | 9,792 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 567.0 | 303.4 | 219 ms | 3.3 ms | 16.5 | 4 | 17,008 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,106.9 | 285.0 | 259 ms | 3.5 ms | 4.3 | 8 | 33,209 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,084.8 | 270.2 | 292 ms | 3.7 ms | 8.4 | 16 | 62,545 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,434.5 | 218.5 | 472 ms | 4.6 ms | 18.7 | 16 | 103,034 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,823.3 | 186.8 | 911 ms | 5.4 ms | 31.6 | 32 | 174,702 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 7,475.7 | 118.0 | 2,252 ms | 8.5 ms | 75.2 | 0 | 224,235 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 10.3K | 86.1 | 4,610 ms | 11.6 ms | 128.0 | 0 | 308,791 | 30.0 s |
| Sustained decode | 27K | 1 | Valid | 321.8 | 324.3 | 143 ms | 3.1 ms | 1.0 | 2 | 9,654 | 30.0 s |
| Sustained decode | 27K | 2 | Valid | 473.6 | 239.3 | 201 ms | 4.5 ms | 35.4 | 2 | 14,207 | 30.0 s |
| Sustained decode | 27K | 4 | Valid | 1,094.7 | 277.8 | 225 ms | 3.6 ms | 3.9 | 8 | 32,838 | 30.0 s |
| Sustained decode | 27K | 8 | Valid | 1,877.7 | 236.2 | 542 ms | 4.2 ms | 8.9 | 8 | 56,327 | 30.0 s |
| Sustained decode | 27K | 16 | Valid | 3,169.9 | 202.9 | 921 ms | 4.9 ms | 15.8 | 16 | 95,100 | 30.0 s |
| Sustained decode | 27K | 32 | Valid | 4,392.0 | 141.2 | 1,711 ms | 7.1 ms | 33.5 | 32 | 131,744 | 30.0 s |
| Sustained decode | 27K | 64 | Valid | 4,616.5 | 73.9 | 4,029 ms | 13.5 ms | 96.0 | 0 | 138,492 | 30.0 s |
| Sustained decode | 27K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.