Speed test matrix
LFM2.5-VL-3B on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF163 contexts (8K–27K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 27K, 32K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- lfm-vl
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_USE_CUDA_IPC_TRANSPORT=1 SGLANG_USE_IPC_POOL_HANDLE_CACHE=1 "$HOME/env_rtxpro6k-liquidai-sglang/bin/sglang" serve --model-path LiquidAI/LFM2.5-VL-3B --served-model-name lfm-vl --trust-remote-code --tp 1 --tool-call-parser lfm2 --context-length 32768 --mem-fraction-static 0.81 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context44.2K186 ms TTFT
16K context47.1K345 ms TTFT
27K context45.7K598 ms TTFT
32K context44.9K720 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 212.037 ms TTFT | 404.048 ms TTFT | 788.379 ms TTFT | Looped | 2,724.9232 ms TTFT | Looped | Looped | 11.6K2,619 ms TTFT |
| 16K | 207.4196 ms TTFT | 390.8257 ms TTFT | Looped | 1,434.1328 ms TTFT | 2,479.0451 ms TTFT | 3,824.11,317 ms TTFT | 6,299.12,737 ms TTFT | Over capacity |
| 27K | 202.1133 ms TTFT | 382.1150 ms TTFT | 735.0292 ms TTFT | 1,338.9554 ms TTFT | 2,140.21,000 ms TTFT | 3,261.51,986 ms TTFT | 5,220.93,887 ms TTFT | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 44.2K | — | 186 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 47.1K | — | 345 ms | — | — | 1 | — | — |
| Prefill | 27K | 1 | Valid | 45.7K | — | 598 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 44.9K | — | 720 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 212.0 | 212.2 | 37 ms | 4.7 ms | 1.0 | 1 | 6,360 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 404.0 | 202.4 | 48 ms | 4.9 ms | 2.0 | 2 | 12,118 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 788.3 | 197.9 | 79 ms | 5.1 ms | 4.0 | 4 | 23,648 | 30.0 s |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 2,724.9 | 172.7 | 232 ms | 5.8 ms | 16.0 | 16 | 81,742 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Valid | 11.6K | 95.3 | 2,619 ms | 10.5 ms | 128.0 | 0 | 346,925 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 207.4 | 208.9 | 196 ms | 4.8 ms | 1.0 | 1 | 6,223 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 390.8 | 196.7 | 257 ms | 5.1 ms | 2.0 | 2 | 11,724 | 30.0 s |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Valid | 1,434.1 | 181.5 | 328 ms | 5.5 ms | 7.9 | 8 | 43,024 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,479.0 | 158.3 | 451 ms | 6.3 ms | 15.9 | 16 | 74,367 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,824.1 | 122.0 | 1,317 ms | 8.2 ms | 32.0 | 19 | 114,634 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 6,299.1 | 103.6 | 2,737 ms | 9.7 ms | 64.0 | 0 | 188,967 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 27K | 1 | Valid | 202.1 | 203.1 | 133 ms | 4.9 ms | 1.0 | 1 | 6,062 | 30.0 s |
| Sustained decode | 27K | 2 | Valid | 382.1 | 191.9 | 150 ms | 5.2 ms | 2.0 | 2 | 11,461 | 30.0 s |
| Sustained decode | 27K | 4 | Valid | 735.0 | 185.7 | 292 ms | 5.4 ms | 3.9 | 4 | 22,052 | 30.0 s |
| Sustained decode | 27K | 8 | Valid | 1,338.9 | 170.6 | 554 ms | 5.9 ms | 7.9 | 8 | 40,164 | 30.0 s |
| Sustained decode | 27K | 16 | Valid | 2,140.2 | 139.4 | 1,000 ms | 7.2 ms | 15.9 | 16 | 64,198 | 30.0 s |
| Sustained decode | 27K | 32 | Valid | 3,261.5 | 102.3 | 1,986 ms | 9.8 ms | 32.0 | 0 | 97,844 | 30.0 s |
| Sustained decode | 27K | 64 | Valid | 5,220.9 | 81.9 | 3,887 ms | 12.2 ms | 64.0 | 0 | 156,598 | 30.0 s |
| Sustained decode | 27K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.