Speed test matrix
Ling-3.0-tiny-base-midtrain on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev236+g1093c501d · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev236+g1093c501d
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1093c501dfef
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 SGLANG_BATCH_INVARIANT_OPS_ENABLE_MM_DEEPGEMM=0 "$HOME/env_rtxpro6k-inclusionai-sglang-1093c501dfef/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny-base-midtrain --served-model-name ling --trust-remote-code --tp 1 --context-length 262144 --mem-fraction-static 0.72 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-deterministic-inference --attention-backend triton --moe-runner-backend triton
Prefill
prompt tok/s by context length8K context26.1K314 ms TTFT
16K context21.2K764 ms TTFT
32K context15K2,146 ms TTFT
64K context9,0047,150 ms TTFT
128K context4,63927,723 ms TTFT
251K context2,193114,919 ms TTFT
256K context2,135120,414 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 97.562 ms TTFT | 194.1431 ms TTFT | 381.2197 ms TTFT | 746.5170 ms TTFT | 1,364.4212 ms TTFT | 2,578.2389 ms TTFT | 4,574.5812 ms TTFT | 7,054.61,511 ms TTFT |
| 16K | 96.874 ms TTFT | 195.6112 ms TTFT | 373.4167 ms TTFT | 733.8236 ms TTFT | 1,316.3439 ms TTFT | 2,416.71,013 ms TTFT | 4,030.91,592 ms TTFT | 5,811.23,438 ms TTFT |
| 32K | 95.5115 ms TTFT | 185.2145 ms TTFT | 339.9240 ms TTFT | 653.7436 ms TTFT | 1,113.9818 ms TTFT | 1,956.91,628 ms TTFT | 3,081.83,309 ms TTFT | Over capacity |
| 64K | 92.4198 ms TTFT | 177.5264 ms TTFT | 314.5411 ms TTFT | 542.4832 ms TTFT | 796.71,746 ms TTFT | 1,281.93,358 ms TTFT | Over capacity | Over capacity |
| 128K | 88.6369 ms TTFT | 165.4511 ms TTFT | 282.5785 ms TTFT | 465.71,625 ms TTFT | 637.13,185 ms TTFT | 947.46,452 ms TTFT | Over capacity | Over capacity |
| 251K | 74.8734 ms TTFT | 137.4996 ms TTFT | 224.41,740 ms TTFT | 353.23,400 ms TTFT | 500.16,638 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 26.1K | — | 314 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.2K | — | 764 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15K | — | 2,146 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,004.0 | — | 7,150 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,639.0 | — | 27,723 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,193.0 | — | 114,919 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,135.0 | — | 120,414 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 97.5 | 97.5 | 62 ms | 10.3 ms | 1.0 | 0 | 2,924 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 194.1 | 96.2 | 431 ms | 10.4 ms | 2.0 | 0 | 5,822 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 381.2 | 95.3 | 197 ms | 10.5 ms | 4.0 | 0 | 11,432 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 746.5 | 93.3 | 170 ms | 10.7 ms | 8.0 | 0 | 22,392 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,364.4 | 85.1 | 212 ms | 11.7 ms | 16.0 | 0 | 40,928 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,578.2 | 80.5 | 389 ms | 12.4 ms | 32.0 | 0 | 77,344 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,574.5 | 71.7 | 812 ms | 14.0 ms | 64.0 | 0 | 137,216 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,054.6 | 55.4 | 1,511 ms | 18.1 ms | 128.0 | 0 | 211,584 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 96.8 | 96.8 | 74 ms | 10.3 ms | 1.0 | 0 | 2,904 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 195.6 | 97.9 | 112 ms | 10.2 ms | 2.0 | 0 | 5,866 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 373.4 | 93.3 | 167 ms | 10.7 ms | 4.0 | 0 | 11,200 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 733.8 | 91.6 | 236 ms | 10.9 ms | 8.0 | 0 | 21,998 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,316.3 | 82.1 | 439 ms | 12.2 ms | 16.0 | 0 | 39,488 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,416.7 | 75.6 | 1,013 ms | 13.2 ms | 32.0 | 0 | 72,480 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,030.9 | 63.1 | 1,592 ms | 15.8 ms | 64.0 | 0 | 120,896 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,811.2 | 45.5 | 3,438 ms | 22.0 ms | 128.0 | 0 | 174,336 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 95.5 | 95.6 | 115 ms | 10.5 ms | 1.0 | 0 | 2,866 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 185.2 | 92.8 | 145 ms | 10.8 ms | 2.0 | 0 | 5,557 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 339.9 | 84.9 | 240 ms | 11.8 ms | 4.0 | 0 | 10,194 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 653.7 | 81.6 | 436 ms | 12.3 ms | 8.0 | 0 | 19,607 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,113.9 | 69.7 | 818 ms | 14.4 ms | 16.0 | 0 | 33,408 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,956.9 | 61.2 | 1,628 ms | 16.3 ms | 32.0 | 0 | 58,687 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,081.8 | 48.2 | 3,309 ms | 20.8 ms | 64.0 | 0 | 92,416 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 92.4 | 92.5 | 198 ms | 10.8 ms | 1.0 | 0 | 2,770 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 177.5 | 88.8 | 264 ms | 11.3 ms | 2.0 | 0 | 5,322 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 314.5 | 78.4 | 411 ms | 12.8 ms | 4.0 | 0 | 9,434 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 542.4 | 67.7 | 832 ms | 14.8 ms | 8.0 | 0 | 16,272 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 796.7 | 50.1 | 1,746 ms | 20.0 ms | 16.0 | 0 | 23,888 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,281.9 | 40.4 | 3,358 ms | 24.7 ms | 32.0 | 0 | 38,432 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 88.6 | 88.8 | 369 ms | 11.3 ms | 1.0 | 0 | 2,658 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 165.4 | 82.6 | 511 ms | 12.1 ms | 2.0 | 0 | 4,962 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 282.5 | 70.4 | 785 ms | 14.2 ms | 4.0 | 0 | 8,471 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 465.7 | 58.0 | 1,625 ms | 17.2 ms | 8.0 | 0 | 13,968 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 637.1 | 40.3 | 3,185 ms | 24.8 ms | 16.0 | 0 | 19,104 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 947.4 | 30.4 | 6,452 ms | 32.9 ms | 32.0 | 0 | 28,416 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 74.8 | 74.9 | 734 ms | 13.3 ms | 1.0 | 0 | 2,243 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 137.4 | 68.5 | 996 ms | 14.6 ms | 2.0 | 0 | 4,120 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 224.4 | 56.0 | 1,740 ms | 17.8 ms | 4.0 | 0 | 6,732 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 353.2 | 44.3 | 3,400 ms | 22.6 ms | 8.0 | 0 | 10,592 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 500.1 | 31.7 | 6,638 ms | 31.5 ms | 16.0 | 0 | 14,992 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.