Speed test matrix
LFM2.5-350M on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_rtxpro6k-liquidai-sglang/bin/sglang" serve --model-path LiquidAI/LFM2.5-350M --served-model-name lfm --trust-remote-code --tp 1 --tool-call-parser lfm2 --context-length 128000 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context151.9K54 ms TTFT
16K context175.7K92 ms TTFT
32K context176.3K183 ms TTFT
64K context147.3K438 ms TTFT
120K context78.8K1,532 ms TTFT
125K context110.4K1,140 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 16K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 32K | Looped | Looped | Looped | Looped | Looped | Looped | Looped | Over capacity |
| 64K | Looped | Looped | 1,726.7314 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity |
| 120K | 388.0267 ms TTFT | 717.8313 ms TTFT | 1,393.5474 ms TTFT | 2,199.9574 ms TTFT | 2,956.51,616 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 151.9K | — | 54 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 175.7K | — | 92 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 176.3K | — | 183 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 147.3K | — | 438 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 78.8K | — | 1,532 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 110.4K | — | 1,140 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Valid | 1,726.7 | 446.1 | 314 ms | 2.2 ms | 3.8 | 12 | 51,801 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 388.0 | 399.5 | 267 ms | 2.5 ms | 0.9 | 3 | 11,641 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 717.8 | 370.6 | 313 ms | 2.7 ms | 1.9 | 6 | 21,532 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 1,393.5 | 359.9 | 474 ms | 2.8 ms | 3.9 | 8 | 41,806 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 2,199.9 | 284.5 | 574 ms | 3.5 ms | 7.8 | 16 | 65,996 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 2,956.5 | 190.4 | 1,616 ms | 5.3 ms | 15.8 | 16 | 88,688 | 30.0 s |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.