Speed test matrix
LFM2.5-1.2B-Instruct on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_rtxpro6k-liquidai-sglang/bin/sglang" serve --model-path LiquidAI/LFM2.5-1.2B-Instruct --served-model-name lfm --trust-remote-code --tp 1 --tool-call-parser lfm2 --context-length 128000 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context86.7K95 ms TTFT
16K context93.1K175 ms TTFT
32K context88.6K365 ms TTFT
64K context74.3K867 ms TTFT
120K context56.1K2,151 ms TTFT
125K context53.6K2,346 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 366.823 ms TTFT | 690.526 ms TTFT | 1,337.146 ms TTFT | 2,525.874 ms TTFT | 4,423.3110 ms TTFT | 7,035.6271 ms TTFT | 11.1K515 ms TTFT | 15K1,202 ms TTFT |
| 16K | 358.6103 ms TTFT | 665.2138 ms TTFT | 1,277.2200 ms TTFT | 2,383.8154 ms TTFT | 3,927.4131 ms TTFT | 5,898.5448 ms TTFT | 8,821.0850 ms TTFT | 12.5K2,662 ms TTFT |
| 32K | 341.2137 ms TTFT | 636.1185 ms TTFT | 1,214.6203 ms TTFT | 2,152.0133 ms TTFT | 3,292.6426 ms TTFT | 4,379.8757 ms TTFT | 6,594.42,486 ms TTFT | Over capacity |
| 64K | 312.1203 ms TTFT | 581.2267 ms TTFT | 1,075.6391 ms TTFT | 1,774.2673 ms TTFT | 2,383.11,216 ms TTFT | 2,991.32,613 ms TTFT | Over capacity | Over capacity |
| 120K | 270.0329 ms TTFT | 500.3Requests queued417 ms TTFT | 905.1660 ms TTFT | 1,314.11,351 ms TTFT | 1,591.62,612 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 86.7K | — | 95 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 93.1K | — | 175 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 88.6K | — | 365 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 74.3K | — | 867 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 56.1K | — | 2,151 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 53.6K | — | 2,346 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 366.8 | 367.7 | 23 ms | 2.7 ms | 1.0 | 3 | 10,999 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 690.5 | 345.9 | 26 ms | 2.9 ms | 1.9 | 4 | 20,717 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,337.1 | 335.5 | 46 ms | 3.0 ms | 4.0 | 8 | 40,113 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,525.8 | 317.3 | 74 ms | 3.2 ms | 8.0 | 16 | 75,775 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,423.3 | 278.5 | 110 ms | 3.6 ms | 15.9 | 32 | 132,701 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 7,035.6 | 224.3 | 271 ms | 4.5 ms | 31.5 | 32 | 211,064 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 11.1K | 181.3 | 515 ms | 5.5 ms | 64.0 | 64 | 333,259 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 15K | 122.5 | 1,202 ms | 8.2 ms | 128.0 | 48 | 448,399 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 358.6 | 361.1 | 103 ms | 2.8 ms | 1.0 | 2 | 10,756 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 665.2 | 335.8 | 138 ms | 3.0 ms | 1.9 | 4 | 19,956 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,277.2 | 322.9 | 200 ms | 3.1 ms | 4.0 | 8 | 38,316 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,383.8 | 300.5 | 154 ms | 3.3 ms | 7.9 | 16 | 71,515 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,927.4 | 248.6 | 131 ms | 4.0 ms | 15.8 | 32 | 117,810 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,898.5 | 188.7 | 448 ms | 5.3 ms | 31.7 | 32 | 177,073 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 8,821.0 | 143.5 | 850 ms | 7.0 ms | 63.7 | 64 | 264,640 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 12.5K | 102.3 | 2,662 ms | 9.8 ms | 128.0 | 0 | 376,367 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 341.2 | 345.0 | 137 ms | 2.9 ms | 1.0 | 2 | 10,237 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 636.1 | 321.8 | 185 ms | 3.1 ms | 2.0 | 4 | 19,084 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,214.6 | 308.5 | 203 ms | 3.2 ms | 4.0 | 8 | 36,435 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,152.0 | 272.1 | 133 ms | 3.7 ms | 8.0 | 16 | 64,557 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,292.6 | 209.0 | 426 ms | 4.8 ms | 16.0 | 16 | 98,777 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,379.8 | 139.9 | 757 ms | 7.1 ms | 31.9 | 32 | 131,381 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 6,594.4 | 107.0 | 2,486 ms | 9.3 ms | 64.0 | 0 | 197,819 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 312.1 | 316.4 | 203 ms | 3.2 ms | 1.0 | 2 | 9,362 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 581.2 | 295.9 | 267 ms | 3.4 ms | 2.0 | 4 | 17,436 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,075.6 | 275.3 | 391 ms | 3.6 ms | 3.8 | 8 | 32,268 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,774.2 | 227.0 | 673 ms | 4.4 ms | 8.0 | 8 | 53,223 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,383.1 | 153.7 | 1,216 ms | 6.5 ms | 15.7 | 16 | 71,490 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,991.3 | 94.3 | 2,613 ms | 10.6 ms | 32.0 | 0 | 89,728 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 270.0 | 276.0 | 329 ms | 3.6 ms | 1.0 | 2 | 8,099 | 30.0 s |
| Sustained decode | 120K | 2 | Requests queued | 500.3 | 257.7 | 417 ms | 3.9 ms | 1.9 | 4 | 15,010 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 905.1 | 230.5 | 660 ms | 4.3 ms | 3.9 | 4 | 27,152 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 1,314.1 | 171.3 | 1,351 ms | 5.8 ms | 7.9 | 8 | 39,426 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 1,591.6 | 98.9 | 2,612 ms | 10.1 ms | 16.0 | 0 | 47,742 | 30.0 s |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.