Speed test matrix
Ling-3.0-tiny on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev236+g1093c501d · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev236+g1093c501d
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1093c501dfef
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 SGLANG_BATCH_INVARIANT_OPS_ENABLE_MM_DEEPGEMM=0 "$HOME/env_rtxpro6k-inclusionai-sglang-1093c501dfef/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny --served-model-name ling --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 131072 --mem-fraction-static 0.77 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-deterministic-inference --attention-backend triton --moe-runner-backend triton
Prefill
prompt tok/s by context length8K context23.5K349 ms TTFT
16K context21.1K768 ms TTFT
32K context15K2,153 ms TTFT
64K context9,3546,884 ms TTFT
123K context4,88125,323 ms TTFT
128K context4,64427,692 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 101.562 ms TTFT | 193.9217 ms TTFT | 380.7143 ms TTFT | 755.9178 ms TTFT | 1,373.4220 ms TTFT | 2,559.0434 ms TTFT | 4,298.01,059 ms TTFT | 6,109.21,768 ms TTFT |
| 16K | 98.672 ms TTFT | 197.2121 ms TTFT | 381.6169 ms TTFT | 738.9236 ms TTFT | 1,322.3446 ms TTFT | 2,385.31,326 ms TTFT | 3,798.02,585 ms TTFT | 5,129.24,586 ms TTFT |
| 32K | 97.4124 ms TTFT | 185.8151 ms TTFT | 347.7247 ms TTFT | 663.7437 ms TTFT | 1,074.9894 ms TTFT | 1,850.81,774 ms TTFT | 2,761.23,505 ms TTFT | 3,903.06,437 ms TTFT |
| 64K | 96.1193 ms TTFT | 177.8254 ms TTFT | 311.0417 ms TTFT | 527.3832 ms TTFT | 796.41,681 ms TTFT | 1,267.93,423 ms TTFT | 1,849.66,536 ms TTFT | Over capacity |
| 123K | 90.9343 ms TTFT | 163.2462 ms TTFT | 279.6766 ms TTFT | 455.91,616 ms TTFT | 671.73,015 ms TTFT | 993.16,230 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 23.5K | — | 349 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.1K | — | 768 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15K | — | 2,153 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,354.0 | — | 6,884 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 4,881.0 | — | 25,323 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,644.0 | — | 27,692 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 101.5 | 101.6 | 62 ms | 9.8 ms | 1.0 | 0 | 3,045 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 193.9 | 96.7 | 217 ms | 10.3 ms | 2.0 | 0 | 5,818 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 380.7 | 95.2 | 143 ms | 10.5 ms | 4.0 | 0 | 11,418 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 755.9 | 94.3 | 178 ms | 10.6 ms | 8.0 | 0 | 22,672 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,373.4 | 85.8 | 220 ms | 11.7 ms | 16.0 | 0 | 41,200 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,559.0 | 80.0 | 434 ms | 12.5 ms | 32.0 | 0 | 76,768 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,298.0 | 67.3 | 1,059 ms | 14.9 ms | 64.0 | 0 | 128,925 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,109.2 | 47.9 | 1,768 ms | 20.9 ms | 128.0 | 0 | 183,232 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 98.6 | 98.6 | 72 ms | 10.1 ms | 1.0 | 0 | 2,958 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 197.2 | 98.6 | 121 ms | 10.1 ms | 2.0 | 0 | 5,916 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 381.6 | 95.3 | 169 ms | 10.5 ms | 4.0 | 0 | 11,448 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 738.9 | 92.3 | 236 ms | 10.8 ms | 8.0 | 0 | 22,176 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,322.3 | 82.5 | 446 ms | 12.1 ms | 16.0 | 0 | 39,664 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,385.3 | 74.6 | 1,326 ms | 13.4 ms | 32.0 | 0 | 71,552 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,798.0 | 59.4 | 2,585 ms | 16.8 ms | 64.0 | 0 | 113,920 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,129.2 | 40.1 | 4,586 ms | 24.9 ms | 128.0 | 0 | 153,856 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 97.4 | 97.4 | 124 ms | 10.3 ms | 1.0 | 0 | 2,921 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 185.8 | 92.8 | 151 ms | 10.8 ms | 2.0 | 0 | 5,574 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 347.7 | 86.8 | 247 ms | 11.5 ms | 4.0 | 0 | 10,429 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 663.7 | 82.8 | 437 ms | 12.1 ms | 8.0 | 0 | 19,904 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,074.9 | 67.3 | 894 ms | 14.9 ms | 16.0 | 0 | 32,240 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,850.8 | 58.0 | 1,774 ms | 17.2 ms | 32.0 | 0 | 55,520 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,761.2 | 43.2 | 3,505 ms | 23.1 ms | 64.0 | 0 | 82,816 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,903.0 | 30.5 | 6,437 ms | 32.8 ms | 128.0 | 0 | 117,053 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 96.1 | 96.1 | 193 ms | 10.4 ms | 1.0 | 0 | 2,882 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 177.8 | 88.8 | 254 ms | 11.3 ms | 2.0 | 0 | 5,332 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 311.0 | 77.8 | 417 ms | 12.9 ms | 4.0 | 0 | 9,329 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 527.3 | 66.0 | 832 ms | 15.1 ms | 8.0 | 0 | 15,816 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 796.4 | 50.1 | 1,681 ms | 20.0 ms | 16.0 | 0 | 23,888 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,267.9 | 40.1 | 3,423 ms | 25.0 ms | 32.0 | 0 | 38,016 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,849.6 | 29.4 | 6,536 ms | 34.0 ms | 64.0 | 0 | 55,488 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 90.9 | 90.9 | 343 ms | 11.0 ms | 1.0 | 0 | 2,727 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 163.2 | 81.5 | 462 ms | 12.3 ms | 2.0 | 0 | 4,894 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 279.6 | 69.8 | 766 ms | 14.3 ms | 4.0 | 0 | 8,388 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 455.9 | 57.1 | 1,616 ms | 17.5 ms | 8.0 | 0 | 13,671 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 671.7 | 42.3 | 3,015 ms | 23.6 ms | 16.0 | 0 | 20,144 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 993.1 | 31.8 | 6,230 ms | 31.5 ms | 32.0 | 0 | 29,792 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.