Speed test matrix
LFM2.5-8B-A1B on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_rtxpro6k-liquidai-sglang/bin/sglang" serve --model-path LiquidAI/LFM2.5-8B-A1B --served-model-name lfm --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser lfm2 --context-length 128000 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context16.1K508 ms TTFT
16K context31.9K510 ms TTFT
32K context31.7K1,019 ms TTFT
64K context42.2K1,528 ms TTFT
120K context39.5K3,060 ms TTFT
125K context41K3,073 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 261.037 ms TTFT | 384.993 ms TTFT | 596.9106 ms TTFT | 912.5120 ms TTFT | 1,505.5192 ms TTFT | 2,496.5376 ms TTFT | 3,503.1694 ms TTFT | 6,034.81,275 ms TTFT |
| 16K | 248.8155 ms TTFT | 375.1276 ms TTFT | 582.7516 ms TTFT | 917.4428 ms TTFT | 1,440.3382 ms TTFT | 2,338.2673 ms TTFT | 3,222.41,310 ms TTFT | 5,455.92,604 ms TTFT |
| 32K | 242.4192 ms TTFT | 365.6333 ms TTFT | 568.2559 ms TTFT | 878.3356 ms TTFT | 1,354.9714 ms TTFT | 2,065.91,410 ms TTFT | 2,854.92,911 ms TTFT | Over capacity |
| 64K | 233.5259 ms TTFT | 346.3427 ms TTFT | 515.5730 ms TTFT | 820.7917 ms TTFT | 1,160.81,426 ms TTFT | 1,700.22,857 ms TTFT | Over capacity | Over capacity |
| 120K | 210.8377 ms TTFT | 320.4592 ms TTFT | 487.51,015 ms TTFT | 727.81,393 ms TTFT | 984.02,613 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.1K | — | 508 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.9K | — | 510 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.7K | — | 1,019 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 42.2K | — | 1,528 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 39.5K | — | 3,060 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 41K | — | 3,073 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 261.0 | 261.7 | 37 ms | 3.8 ms | 1.0 | 2 | 7,829 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 384.9 | 193.4 | 93 ms | 5.2 ms | 2.0 | 2 | 11,548 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 596.9 | 152.2 | 106 ms | 6.6 ms | 4.0 | 4 | 17,904 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 912.5 | 115.9 | 120 ms | 8.6 ms | 8.0 | 0 | 27,376 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,505.5 | 95.5 | 192 ms | 10.5 ms | 16.0 | 0 | 45,152 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,496.5 | 79.8 | 376 ms | 12.5 ms | 32.0 | 0 | 74,896 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,503.1 | 56.4 | 694 ms | 17.7 ms | 64.0 | 0 | 105,088 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,034.8 | 48.6 | 1,275 ms | 20.6 ms | 128.0 | 0 | 180,992 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 248.8 | 251.5 | 155 ms | 4.0 ms | 1.0 | 2 | 7,464 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 375.1 | 189.9 | 276 ms | 5.3 ms | 2.0 | 2 | 11,252 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 582.7 | 150.4 | 516 ms | 6.6 ms | 4.0 | 4 | 17,480 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 917.4 | 115.8 | 428 ms | 8.6 ms | 8.0 | 0 | 27,520 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,440.3 | 91.3 | 382 ms | 11.0 ms | 16.0 | 0 | 43,200 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,338.2 | 74.6 | 673 ms | 13.4 ms | 32.0 | 0 | 70,144 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,222.4 | 51.6 | 1,310 ms | 19.4 ms | 64.0 | 0 | 96,640 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,455.9 | 43.8 | 2,604 ms | 22.8 ms | 128.0 | 0 | 163,584 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 242.4 | 244.0 | 192 ms | 4.1 ms | 1.0 | 1 | 7,273 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 365.6 | 185.3 | 333 ms | 5.4 ms | 1.9 | 2 | 10,966 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 568.2 | 147.7 | 559 ms | 6.8 ms | 4.0 | 4 | 17,044 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 878.3 | 110.5 | 356 ms | 9.0 ms | 8.0 | 0 | 26,344 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,354.9 | 85.5 | 714 ms | 11.7 ms | 16.0 | 0 | 40,640 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,065.9 | 65.7 | 1,410 ms | 15.2 ms | 32.0 | 0 | 61,952 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,854.9 | 45.5 | 2,911 ms | 22.0 ms | 64.0 | 0 | 85,632 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 233.5 | 235.6 | 259 ms | 4.2 ms | 1.0 | 1 | 7,004 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 346.3 | 175.7 | 427 ms | 5.7 ms | 1.9 | 2 | 10,386 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 515.5 | 140.2 | 730 ms | 7.2 ms | 3.9 | 4 | 15,464 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 820.7 | 103.6 | 917 ms | 9.7 ms | 8.0 | 0 | 24,616 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,160.8 | 74.2 | 1,426 ms | 13.5 ms | 16.0 | 0 | 34,816 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,700.2 | 55.1 | 2,857 ms | 18.1 ms | 32.0 | 0 | 50,976 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 210.8 | 213.6 | 377 ms | 4.7 ms | 1.0 | 1 | 6,324 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 320.4 | 164.3 | 592 ms | 6.1 ms | 1.9 | 2 | 9,612 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 487.5 | 156.1 | 1,015 ms | 6.6 ms | 3.9 | 4 | 14,624 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 727.8 | 92.2 | 1,393 ms | 10.8 ms | 8.0 | 0 | 21,835 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 984.0 | 63.7 | 2,613 ms | 15.7 ms | 16.0 | 0 | 29,504 | 30.0 s |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.