Speed test matrix
LFM2.5-2.6B on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-liquidai-sglang/bin/sglang" serve --model-path LiquidAI/LFM2.5-2.6B --served-model-name lfm --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser lfm2 --context-length 131072 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context47.5K173 ms TTFT
16K context49.5K329 ms TTFT
32K context47K689 ms TTFT
64K context41.2K1,569 ms TTFT
123K context32.8K3,780 ms TTFT
128K context32.2K4,003 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 190.826 ms TTFT | 363.931 ms TTFT | 710.356 ms TTFT | 1,358.0112 ms TTFT | 2,453.7199 ms TTFT | 4,051.7383 ms TTFT | 6,788.9761 ms TTFT | 10.5K1,556 ms TTFT |
| 16K | 187.0181 ms TTFT | 353.4262 ms TTFT | 681.9414 ms TTFT | 1,303.3209 ms TTFT | 2,251.5323 ms TTFT | 3,564.2742 ms TTFT | 5,852.61,495 ms TTFT | Over capacity |
| 32K | 181.3220 ms TTFT | 341.2310 ms TTFT | 653.9Requests queued489 ms TTFT | 1,196.0350 ms TTFT | 1,916.3719 ms TTFT | 2,865.51,439 ms TTFT | 4,519.22,822 ms TTFT | Over capacity |
| 64K | 169.8293 ms TTFT | 319.6402 ms TTFT | 607.8557 ms TTFT | 1,051.0443 ms TTFT | 1,552.81,422 ms TTFT | 2,013.32,824 ms TTFT | Over capacity | Over capacity |
| 123K | 153.0224 ms TTFT | 289.0328 ms TTFT | 538.1507 ms TTFT | 851.41,320 ms TTFT | 1,114.22,648 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 47.5K | — | 173 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 49.5K | — | 329 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 47K | — | 689 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 41.2K | — | 1,569 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 32.8K | — | 3,780 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 32.2K | — | 4,003 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 190.8 | 191.1 | 26 ms | 5.2 ms | 1.0 | 1 | 5,725 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 363.9 | 180.6 | 31 ms | 5.5 ms | 2.0 | 2 | 10,917 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 710.3 | 178.3 | 56 ms | 5.6 ms | 4.0 | 4 | 21,308 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,358.0 | 170.7 | 112 ms | 5.9 ms | 8.0 | 8 | 40,739 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,453.7 | 155.0 | 199 ms | 6.5 ms | 16.0 | 16 | 73,606 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 4,051.7 | 129.1 | 383 ms | 7.7 ms | 32.0 | 32 | 121,548 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 6,788.9 | 108.2 | 761 ms | 9.2 ms | 64.0 | 0 | 203,649 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 10.5K | 83.9 | 1,556 ms | 11.9 ms | 128.0 | 0 | 316,304 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 187.0 | 188.3 | 181 ms | 5.3 ms | 1.0 | 1 | 5,610 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 353.4 | 178.0 | 262 ms | 5.6 ms | 2.0 | 2 | 10,602 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 681.9 | 172.2 | 414 ms | 5.8 ms | 3.9 | 4 | 20,456 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,303.3 | 164.6 | 209 ms | 6.1 ms | 7.9 | 8 | 39,094 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,251.5 | 142.3 | 323 ms | 7.0 ms | 16.0 | 16 | 67,541 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,564.2 | 112.7 | 742 ms | 8.9 ms | 32.0 | 0 | 106,912 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 5,852.6 | 93.2 | 1,495 ms | 10.7 ms | 64.0 | 0 | 175,552 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 181.3 | 182.7 | 220 ms | 5.5 ms | 1.0 | 1 | 5,438 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 341.2 | 172.1 | 310 ms | 5.8 ms | 2.0 | 2 | 10,236 | 30.0 s |
| Sustained decode | 32K | 4 | Requests queued | 653.9 | 165.1 | 489 ms | 6.1 ms | 3.9 | 4 | 19,617 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,196.0 | 151.4 | 350 ms | 6.6 ms | 7.9 | 8 | 35,880 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,916.3 | 119.8 | 719 ms | 8.3 ms | 16.0 | 0 | 57,488 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,865.5 | 89.6 | 1,439 ms | 11.2 ms | 32.0 | 0 | 85,952 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,519.2 | 70.9 | 2,822 ms | 14.1 ms | 64.0 | 0 | 135,529 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 169.8 | 171.6 | 293 ms | 5.8 ms | 1.0 | 1 | 5,094 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 319.6 | 161.7 | 402 ms | 6.2 ms | 1.9 | 2 | 9,586 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 607.8 | 153.5 | 557 ms | 6.5 ms | 3.9 | 4 | 18,232 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,051.0 | 133.7 | 443 ms | 7.5 ms | 8.0 | 8 | 31,530 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,552.8 | 99.0 | 1,422 ms | 10.1 ms | 16.0 | 0 | 46,576 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,013.3 | 65.5 | 2,824 ms | 15.3 ms | 32.0 | 0 | 60,384 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 153.0 | 154.4 | 224 ms | 6.5 ms | 1.0 | 1 | 4,589 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 289.0 | 146.4 | 328 ms | 6.8 ms | 2.0 | 2 | 8,667 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 538.1 | 137.0 | 507 ms | 7.3 ms | 4.0 | 4 | 16,142 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 851.4 | 108.0 | 1,320 ms | 9.3 ms | 8.0 | 0 | 25,539 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,114.2 | 71.5 | 2,648 ms | 14.0 ms | 16.0 | 0 | 33,424 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.