Speed test matrix
LFM2.5-8B-A1B on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.6.post3.dev9418+gc26ae0439 · BF165 contexts (8K–120K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 120K, 125K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9418+gc26ae0439
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/tugot17/sglang
- Build
- cuda-13.2
- Commit
c26ae043924f
Run
- Served model
- lfm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 "$HOME/env_rtxpro6k-liquidai-sglang-pr-31041/bin/sglang" serve --model-path LiquidAI/LFM2.5-8B-A1B --served-model-name lfm --trust-remote-code --tp 1 --reasoning-parser qwen3 --tool-call-parser lfm2 --context-length 128000 --mem-fraction-static 0.79 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --disable-radix-cache --speculative-algorithm DSPARK --speculative-draft-model-path LiquidAI/LFM2.5-8B-A1B-DSpark --speculative-draft-attention-backend flashinfer
Prefill
prompt tok/s by context length8K context16.1K511 ms TTFT
16K context31.9K510 ms TTFT
32K context31.7K1,021 ms TTFT
64K context42.2K1,532 ms TTFT
120K context39.5K3,063 ms TTFT
125K context40.8K3,089 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 278.9156 ms TTFT | 463.1243 ms TTFT | 674.4353 ms TTFT | 1,103.2996 ms TTFT | 1,889.71,944 ms TTFT | Looped | Not run | Not run |
| 16K | 265.2276 ms TTFT | 451.5405 ms TTFT | 591.3647 ms TTFT | 1,008.21,951 ms TTFT | 1,702.93,882 ms TTFT | 2,702.47,743 ms TTFT | Not run | Not run |
| 32K | 250.0573 ms TTFT | 350.4842 ms TTFT | 508.12,124 ms TTFT | 804.64,195 ms TTFT | 1,317.68,357 ms TTFT | 1,936.916,688 ms TTFT | Not run | Over capacity |
| 64K | 152.01,287 ms TTFT | 229.52,487 ms TTFT | 341.14,872 ms TTFT | 518.29,670 ms TTFT | 792.619,271 ms TTFT | 1,095.638,534 ms TTFT | Over capacity | Over capacity |
| 120K | 113.12,928 ms TTFT | 179.95,703 ms TTFT | 244.311,231 ms TTFT | 345.222,339 ms TTFT | 428.244,549 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.1K | — | 511 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 31.9K | — | 510 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.7K | — | 1,021 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 42.2K | — | 1,532 ms | — | — | 1 | — | — |
| Prefill | 120K | 1 | Valid | 39.5K | — | 3,063 ms | — | — | 1 | — | — |
| Prefill | 125K | 1 | Valid | 40.8K | — | 3,089 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 278.9 | 291.3 | 156 ms | 3.4 ms | 1.0 | 2 | 8,365 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 463.1 | 238.9 | 243 ms | 4.2 ms | 2.0 | 2 | 13,892 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 674.4 | 153.2 | 353 ms | 6.5 ms | 4.0 | 4 | 20,225 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,103.2 | 135.8 | 996 ms | 7.4 ms | 7.9 | 6 | 33,080 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,889.7 | 117.4 | 1,944 ms | 8.5 ms | 15.9 | 5 | 56,657 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 265.2 | 267.6 | 276 ms | 3.7 ms | 1.0 | 2 | 7,955 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 451.5 | 231.5 | 405 ms | 4.3 ms | 2.0 | 2 | 13,541 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 591.3 | 164.2 | 647 ms | 6.1 ms | 4.0 | 4 | 17,737 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 1,008.2 | 130.0 | 1,951 ms | 7.7 ms | 8.0 | 4 | 30,236 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,702.9 | 103.4 | 3,882 ms | 9.7 ms | 15.9 | 4 | 51,056 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,702.4 | 83.4 | 7,743 ms | 12.0 ms | 32.0 | 1 | 81,042 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 250.0 | 262.8 | 573 ms | 3.8 ms | 1.0 | 2 | 7,498 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 350.4 | 182.0 | 842 ms | 5.5 ms | 2.0 | 2 | 10,508 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 508.1 | 122.0 | 2,124 ms | 8.2 ms | 4.0 | 1 | 15,238 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 804.6 | 102.4 | 4,195 ms | 9.8 ms | 8.0 | 0 | 24,133 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,317.6 | 81.9 | 8,357 ms | 12.2 ms | 16.0 | 0 | 39,503 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,936.9 | 61.6 | 16,688 ms | 16.2 ms | 32.0 | 0 | 58,110 | 30.0 s |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 152.0 | 161.0 | 1,287 ms | 6.2 ms | 1.0 | 1 | 4,561 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 229.5 | 131.9 | 2,487 ms | 7.6 ms | 2.0 | 1 | 6,884 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 341.1 | 85.0 | 4,872 ms | 11.8 ms | 4.0 | 0 | 10,228 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 518.2 | 66.0 | 9,670 ms | 15.2 ms | 8.0 | 0 | 15,547 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 792.6 | 49.8 | 19,271 ms | 20.1 ms | 16.0 | 0 | 23,772 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,095.6 | 34.8 | 38,534 ms | 28.8 ms | 32.0 | 0 | 32,826 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 1 | Valid | 113.1 | 113.6 | 2,928 ms | 8.8 ms | 1.0 | 0 | 3,391 | 30.0 s |
| Sustained decode | 120K | 2 | Valid | 179.9 | 90.1 | 5,703 ms | 11.1 ms | 2.0 | 0 | 5,396 | 30.0 s |
| Sustained decode | 120K | 4 | Valid | 244.3 | 60.1 | 11,231 ms | 16.6 ms | 4.0 | 0 | 7,326 | 30.0 s |
| Sustained decode | 120K | 8 | Valid | 345.2 | 43.2 | 22,339 ms | 23.1 ms | 8.0 | 0 | 10,348 | 30.0 s |
| Sustained decode | 120K | 16 | Valid | 428.2 | 27.1 | 44,549 ms | 36.9 ms | 16.0 | 0 | 12,830 | 30.0 s |
| Sustained decode | 120K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 120K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.