Speed test matrix
LFM2.5-1.2B-Instruct on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9418+gc26ae0439 · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9418+gc26ae0439
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/tugot17/sglang
- Build
- cuda-13.2
- Commit
c26ae043924f
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-liquidai-sglang-pr-31041/bin/sglang" serve --model-path LiquidAI/LFM2.5-1.2B-Instruct --served-model-name lfm --trust-remote-code --tp 1 --tool-call-parser lfm2 --context-length 128000 --mem-fraction-static 0.80 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --disable-radix-cache --speculative-algorithm DSPARK --speculative-draft-model-path LiquidAI/LFM2.5-1.2B-Instruct-DSpark --speculative-draft-attention-backend flashinfer
Prefill
prompt tok/s by context length8K context84.3K97 ms TTFT
16K context93.2K174 ms TTFT
32K context87.4K370 ms TTFT
64K context73.7K875 ms TTFT
120K context55.7K2,169 ms TTFT
125K context54.6K2,304 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 313.689 ms TTFT | 605.1223 ms TTFT | 1,104.792 ms TTFT | 1,787.2154 ms TTFT | 2,625.5183 ms TTFT | 3,655.71,238 ms TTFT | Not run | Not run |
| 16K | 273.8167 ms TTFT | 516.0167 ms TTFT | 904.7200 ms TTFT | 1,376.5281 ms TTFT | 1,930.9994 ms TTFT | 2,541.82,395 ms TTFT | Not run | Not run |
| 32K | 249.0360 ms TTFT | 447.9396 ms TTFT | 727.9397 ms TTFT | 1,004.6749 ms TTFT | 1,370.92,822 ms TTFT | 1,648.85,382 ms TTFT | Not run | Over capacity |
| 64K | 247.2866 ms TTFT | 413.5Requests queued1,264 ms TTFT | 597.8909 ms TTFT | 872.03,736 ms TTFT | 1,032.06,884 ms TTFT | 1,137.613,218 ms TTFT | Over capacity | Over capacity |
| 120K | 193.52,160 ms TTFT | 287.0Requests queued2,217 ms TTFT | 461.55,300 ms TTFT | 557.19,269 ms TTFT | 613.317,297 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 84.3K | — | 97 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 93.2K | — | 174 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 87.4K | — | 370 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 73.7K | — | 875 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 55.7K | — | 2,169 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 54.6K | — | 2,304 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 313.6 | 321.5 | 89 ms | 3.1 ms | 1.0 | 2 | 9,409 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 605.1 | 306.8 | 223 ms | 3.3 ms | 2.0 | 4 | 18,152 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,104.7 | 280.5 | 92 ms | 3.6 ms | 4.0 | 8 | 33,137 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,787.2 | 225.2 | 154 ms | 4.4 ms | 8.0 | 8 | 53,608 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,625.5 | 164.9 | 183 ms | 6.1 ms | 15.9 | 16 | 78,765 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 3,655.7 | 113.2 | 1,238 ms | 8.8 ms | 32.0 | 0 | 109,652 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 273.8 | 278.6 | 167 ms | 3.6 ms | 1.0 | 2 | 8,215 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 516.0 | 262.6 | 167 ms | 3.8 ms | 2.0 | 4 | 15,478 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 904.7 | 228.4 | 200 ms | 4.4 ms | 4.0 | 4 | 27,140 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,376.5 | 173.9 | 281 ms | 5.8 ms | 8.0 | 8 | 41,288 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,930.9 | 117.3 | 994 ms | 8.5 ms | 16.0 | 4 | 57,921 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,541.8 | 76.4 | 2,395 ms | 13.1 ms | 32.0 | 0 | 76,218 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 249.0 | 254.1 | 360 ms | 3.9 ms | 1.0 | 2 | 7,469 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 447.9 | 228.4 | 396 ms | 4.4 ms | 2.0 | 2 | 13,436 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 727.9 | 182.4 | 397 ms | 5.5 ms | 4.0 | 4 | 21,833 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,004.6 | 118.2 | 749 ms | 8.5 ms | 7.8 | 7 | 30,139 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,370.9 | 80.5 | 2,822 ms | 12.4 ms | 16.0 | 0 | 41,123 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,648.8 | 45.2 | 5,382 ms | 22.1 ms | 32.0 | 0 | 49,458 | 30.0 s |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 247.2 | 258.4 | 866 ms | 3.9 ms | 0.9 | 2 | 7,417 | 30.0 s |
| Sustained decode | 64K | 2 | Requests queued | 413.5 | 213.0 | 1,264 ms | 4.7 ms | 1.9 | 2 | 12,404 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 597.8 | 149.9 | 909 ms | 6.7 ms | 3.9 | 4 | 17,932 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 872.0 | 99.3 | 3,736 ms | 10.1 ms | 8.0 | 0 | 26,153 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,032.0 | 55.3 | 6,884 ms | 18.1 ms | 16.0 | 0 | 30,950 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,137.6 | 26.3 | 13,218 ms | 38.0 ms | 32.0 | 0 | 34,124 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 193.5 | 208.6 | 2,160 ms | 4.8 ms | 0.9 | 1 | 5,806 | 30.0 s |
| Sustained decode | 120K | 2 | Requests queued | 287.0 | 154.9 | 2,217 ms | 6.5 ms | 1.9 | 2 | 8,610 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 461.5 | 107.0 | 5,300 ms | 9.4 ms | 4.0 | 0 | 13,843 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 557.1 | 57.5 | 9,269 ms | 17.4 ms | 8.0 | 0 | 16,709 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 613.3 | 25.9 | 17,297 ms | 38.6 ms | 16.0 | 0 | 18,387 | 30.0 s |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.