Speed test matrix
LFM2.5-2.6B on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9418+gc26ae0439 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9418+gc26ae0439
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/tugot17/sglang
- Build
- cuda-13.2
- Commit
c26ae043924f
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_rtxpro6k-liquidai-sglang-pr-31041/bin/sglang" serve --model-path LiquidAI/LFM2.5-2.6B --served-model-name lfm --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser lfm2 --context-length 131072 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-algorithm DSPARK --speculative-draft-model-path LiquidAI/LFM2.5-2.6B-DSpark --speculative-draft-attention-backend flashinfer --disable-radix-cache
Prefill
prompt tok/s by context length8K context45.7K180 ms TTFT
16K context48.4K336 ms TTFT
32K context46.2K701 ms TTFT
64K context40.7K1,585 ms TTFT
123K context32.6K3,803 ms TTFT
128K context32.1K4,014 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 335.0171 ms TTFT | 661.2209 ms TTFT | Looped | 1,970.2178 ms TTFT | 3,153.7389 ms TTFT | 3,759.1Requests queued1,327 ms TTFT | Not run | Not run |
| 16K | 371.7329 ms TTFT | 695.6333 ms TTFT | 1,114.4402 ms TTFT | 1,752.1539 ms TTFT | 2,386.4758 ms TTFT | 3,311.85,110 ms TTFT | Not run | Not run |
| 32K | 298.0693 ms TTFT | 505.5702 ms TTFT | 863.31,071 ms TTFT | 1,252.0Requests queued1,223 ms TTFT | 1,715.74,419 ms TTFT | 2,325.010,964 ms TTFT | Not run | Over capacity |
| 64K | 227.11,577 ms TTFT | 376.81,667 ms TTFT | 525.9Requests queued2,379 ms TTFT | Looped | 1,051.413,077 ms TTFT | 1,222.425,110 ms TTFT | Over capacity | Over capacity |
| 123K | 150.03,795 ms TTFT | 228.05,774 ms TTFT | 378.39,503 ms TTFT | 487.716,748 ms TTFT | Looped | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 45.7K | — | 180 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 48.4K | — | 336 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 46.2K | — | 701 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 40.7K | — | 1,585 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 32.6K | — | 3,803 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 32.1K | — | 4,014 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 335.0 | 342.0 | 171 ms | 2.9 ms | 1.0 | 2 | 10,049 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 661.2 | 340.0 | 209 ms | 2.9 ms | 2.0 | 4 | 19,837 | 30.0 s |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Valid | 1,970.2 | 247.7 | 178 ms | 4.0 ms | 7.9 | 15 | 59,103 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,153.7 | 200.8 | 389 ms | 5.0 ms | 15.9 | 16 | 94,607 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 3,759.1 | 112.5 | 1,327 ms | 8.9 ms | 31.9 | 21 | 112,492 | 29.9 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 371.7 | 352.4 | 329 ms | 2.8 ms | 1.0 | 2 | 11,149 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 695.6 | 335.8 | 333 ms | 3.0 ms | 2.0 | 5 | 20,868 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,114.4 | 284.4 | 402 ms | 3.5 ms | 3.9 | 8 | 33,427 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,752.1 | 219.9 | 539 ms | 4.5 ms | 7.9 | 8 | 52,551 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,386.4 | 150.3 | 758 ms | 6.7 ms | 15.7 | 16 | 71,593 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,311.8 | 95.2 | 5,110 ms | 10.5 ms | 32.0 | 1 | 99,189 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 298.0 | 321.1 | 693 ms | 3.1 ms | 1.0 | 2 | 8,939 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 505.5 | 295.3 | 702 ms | 3.4 ms | 2.0 | 4 | 15,163 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 863.3 | 220.1 | 1,071 ms | 4.5 ms | 3.9 | 4 | 25,895 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 1,252.0 | 161.6 | 1,223 ms | 6.2 ms | 7.8 | 8 | 37,554 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,715.7 | 101.9 | 4,419 ms | 9.8 ms | 15.9 | 4 | 51,459 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,325.0 | 59.0 | 10,964 ms | 16.9 ms | 32.0 | 0 | 69,755 | 30.0 s |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 227.1 | 242.6 | 1,577 ms | 4.1 ms | 0.9 | 1 | 6,811 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 376.8 | 200.5 | 1,667 ms | 5.0 ms | 1.9 | 2 | 11,302 | 30.0 s |
| Sustained decode | 64K | 4 | Requests queued | 525.9 | 137.4 | 2,379 ms | 7.3 ms | 3.7 | 4 | 15,772 | 30.0 s |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Valid | 1,051.4 | 52.7 | 13,077 ms | 19.0 ms | 16.0 | 0 | 31,520 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,222.4 | 24.1 | 25,110 ms | 41.5 ms | 32.0 | 0 | 36,645 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 150.0 | 180.7 | 3,795 ms | 5.5 ms | 0.9 | 1 | 4,499 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 228.0 | 135.1 | 5,774 ms | 7.5 ms | 2.0 | 1 | 6,713 | 29.4 s |
| Sustained decode | 123K | 4 | Valid | 378.3 | 82.1 | 9,503 ms | 12.2 ms | 4.0 | 0 | 11,347 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 487.7 | 44.8 | 16,748 ms | 22.4 ms | 8.0 | 0 | 14,621 | 30.0 s |
| Sustained decode | 123K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.