Speed test matrix
LFM2.5-1.2B-Thinking on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_rtxpro6k-liquidai-sglang/bin/sglang" serve --model-path LiquidAI/LFM2.5-1.2B-Thinking --served-model-name lfm --trust-remote-code --tp 1 --reasoning-parser qwen3-thinking --tool-call-parser lfm2 --context-length 128000 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context16.1K510 ms TTFT
16K context32K508 ms TTFT
32K context63.4K510 ms TTFT
64K context63.3K1,019 ms TTFT
120K context56K2,156 ms TTFT
125K context55K2,286 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 368.526 ms TTFT | 690.326 ms TTFT | 1,347.141 ms TTFT | 2,540.276 ms TTFT | 4,443.5102 ms TTFT | 7,061.3308 ms TTFT | 11.1K609 ms TTFT | 15.5K1,257 ms TTFT |
| 16K | 360.5103 ms TTFT | 671.1181 ms TTFT | 1,286.3308 ms TTFT | 2,391.0171 ms TTFT | 3,914.7212 ms TTFT | 5,872.8474 ms TTFT | 8,717.3860 ms TTFT | 12.5K2,589 ms TTFT |
| 32K | 343.2141 ms TTFT | 636.5240 ms TTFT | 1,213.5348 ms TTFT | 2,147.6104 ms TTFT | 3,279.4463 ms TTFT | 4,348.0837 ms TTFT | 6,563.32,725 ms TTFT | Over capacity |
| 64K | 313.6206 ms TTFT | 581.5334 ms TTFT | 1,074.5572 ms TTFT | 1,696.5918 ms TTFT | 2,376.81,242 ms TTFT | 2,987.92,549 ms TTFT | Over capacity | Over capacity |
| 120K | 270.8328 ms TTFT | 500.5Requests queued432 ms TTFT | 925.3684 ms TTFT | 1,330.61,344 ms TTFT | 1,601.22,649 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.1K | — | 510 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 32K | — | 508 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 63.4K | — | 510 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 63.3K | — | 1,019 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 56K | — | 2,156 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 55K | — | 2,286 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 368.5 | 370.1 | 26 ms | 2.7 ms | 1.0 | 3 | 11,056 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 690.3 | 346.1 | 26 ms | 2.9 ms | 2.0 | 4 | 20,710 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,347.1 | 338.0 | 41 ms | 3.0 ms | 4.0 | 8 | 40,413 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,540.2 | 318.9 | 76 ms | 3.1 ms | 8.0 | 16 | 76,202 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,443.5 | 280.5 | 102 ms | 3.6 ms | 16.0 | 32 | 133,308 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 7,061.3 | 225.3 | 308 ms | 4.4 ms | 31.4 | 32 | 211,831 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 11.1K | 181.9 | 609 ms | 5.5 ms | 63.7 | 64 | 333,994 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 15.5K | 122.0 | 1,257 ms | 8.2 ms | 128.0 | 68 | 464,991 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 360.5 | 363.9 | 103 ms | 2.7 ms | 1.0 | 2 | 10,815 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 671.1 | 339.8 | 181 ms | 2.9 ms | 2.0 | 4 | 20,134 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,286.3 | 327.6 | 308 ms | 3.1 ms | 3.9 | 8 | 38,590 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,391.0 | 302.0 | 171 ms | 3.3 ms | 8.0 | 16 | 71,729 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,914.7 | 249.1 | 212 ms | 4.0 ms | 15.7 | 32 | 117,431 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,872.8 | 187.5 | 474 ms | 5.3 ms | 31.5 | 32 | 176,190 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 8,717.3 | 141.1 | 860 ms | 7.1 ms | 63.8 | 64 | 261,522 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 12.5K | 100.9 | 2,589 ms | 9.9 ms | 128.0 | 0 | 375,422 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 343.2 | 347.0 | 141 ms | 2.9 ms | 1.0 | 2 | 10,295 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 636.5 | 323.7 | 240 ms | 3.1 ms | 2.0 | 4 | 19,094 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,213.5 | 308.7 | 348 ms | 3.2 ms | 4.0 | 8 | 36,403 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,147.6 | 271.2 | 104 ms | 3.7 ms | 7.9 | 16 | 64,422 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,279.4 | 208.1 | 463 ms | 4.8 ms | 16.0 | 16 | 98,374 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,348.0 | 139.0 | 837 ms | 7.2 ms | 31.9 | 32 | 130,440 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 6,563.3 | 106.5 | 2,725 ms | 9.4 ms | 64.0 | 0 | 196,862 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 313.6 | 318.0 | 206 ms | 3.1 ms | 1.0 | 2 | 9,406 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 581.5 | 297.4 | 334 ms | 3.4 ms | 2.0 | 4 | 17,444 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,074.5 | 279.5 | 572 ms | 3.6 ms | 3.8 | 8 | 32,232 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,696.5 | 218.8 | 918 ms | 4.6 ms | 7.9 | 8 | 50,892 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,376.8 | 154.5 | 1,242 ms | 6.5 ms | 15.7 | 16 | 71,301 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,987.9 | 93.8 | 2,549 ms | 10.7 ms | 32.0 | 0 | 89,632 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 270.8 | 276.9 | 328 ms | 3.6 ms | 1.0 | 2 | 8,123 | 30.0 s |
| Sustained decode | 120K | 2 | Requests queued | 500.5 | 258.1 | 432 ms | 3.9 ms | 1.9 | 4 | 15,014 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 925.3 | 236.0 | 684 ms | 4.2 ms | 3.9 | 4 | 27,756 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 1,330.6 | 173.4 | 1,344 ms | 5.8 ms | 7.7 | 8 | 39,912 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 1,601.2 | 99.3 | 2,649 ms | 10.1 ms | 16.0 | 0 | 48,032 | 30.0 s |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.