Speed test matrix
Ling-3.0-tiny-base on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev236+g1093c501d · BF166 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev236+g1093c501d
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1093c501dfef
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 SGLANG_BATCH_INVARIANT_OPS_ENABLE_MM_DEEPGEMM=0 "$HOME/env_rtxpro6k-inclusionai-sglang-1093c501dfef/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny-base --served-model-name ling --trust-remote-code --tp 1 --context-length 262144 --mem-fraction-static 0.72 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-deterministic-inference --attention-backend triton --moe-runner-backend triton
Prefill
prompt tok/s by context length8K context26.3K312 ms TTFT
16K context21.3K762 ms TTFT
32K context15K2,151 ms TTFT
64K context9,3736,868 ms TTFT
128K context4,64127,710 ms TTFT
251K context2,199114,599 ms TTFT
256K context2,152119,467 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 97.461 ms TTFT | 194.1211 ms TTFT | 381.1105 ms TTFT | 746.3175 ms TTFT | 1,360.3221 ms TTFT | 2,569.7377 ms TTFT | 4,584.5757 ms TTFT | 7,051.41,556 ms TTFT |
| 16K | 96.890 ms TTFT | 194.1120 ms TTFT | 373.4147 ms TTFT | 726.8207 ms TTFT | 1,311.0387 ms TTFT | 2,413.6758 ms TTFT | 4,031.51,519 ms TTFT | 5,966.73,052 ms TTFT |
| 32K | 95.6114 ms TTFT | 184.9141 ms TTFT | 349.8251 ms TTFT | 652.8449 ms TTFT | 1,046.8897 ms TTFT | 1,752.81,821 ms TTFT | 2,688.73,704 ms TTFT | Over capacity |
| 64K | 92.2201 ms TTFT | 177.4275 ms TTFT | 314.3421 ms TTFT | 538.9834 ms TTFT | 810.41,699 ms TTFT | 1,295.13,411 ms TTFT | Over capacity | Over capacity |
| 128K | 88.7375 ms TTFT | 160.7474 ms TTFT | 274.9877 ms TTFT | 449.81,604 ms TTFT | 660.63,157 ms TTFT | 973.36,467 ms TTFT | Over capacity | Over capacity |
| 251K | 75.0733 ms TTFT | 137.81,007 ms TTFT | 224.11,657 ms TTFT | 355.23,293 ms TTFT | 499.26,530 ms TTFT | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 26.3K | — | 312 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.3K | — | 762 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15K | — | 2,151 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,373.0 | — | 6,868 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,641.0 | — | 27,710 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 2,199.0 | — | 114,599 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 2,152.0 | — | 119,467 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 97.4 | 97.5 | 61 ms | 10.3 ms | 1.0 | 0 | 2,923 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 194.1 | 96.7 | 211 ms | 10.3 ms | 2.0 | 0 | 5,822 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 381.1 | 95.3 | 105 ms | 10.5 ms | 4.0 | 0 | 11,432 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 746.3 | 93.3 | 175 ms | 10.7 ms | 8.0 | 0 | 22,384 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,360.3 | 84.9 | 221 ms | 11.8 ms | 16.0 | 0 | 40,800 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,569.7 | 80.2 | 377 ms | 12.5 ms | 32.0 | 0 | 77,088 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,584.5 | 71.8 | 757 ms | 13.9 ms | 64.0 | 0 | 137,536 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 7,051.4 | 55.4 | 1,556 ms | 18.0 ms | 128.0 | 0 | 211,499 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 96.8 | 96.8 | 90 ms | 10.3 ms | 1.0 | 0 | 2,904 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 194.1 | 97.2 | 120 ms | 10.3 ms | 2.0 | 0 | 5,822 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 373.4 | 93.4 | 147 ms | 10.7 ms | 4.0 | 0 | 11,200 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 726.8 | 90.8 | 207 ms | 11.0 ms | 8.0 | 0 | 21,800 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,311.0 | 81.8 | 387 ms | 12.2 ms | 16.0 | 0 | 39,328 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,413.6 | 75.3 | 758 ms | 13.3 ms | 32.0 | 0 | 72,384 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,031.5 | 63.1 | 1,519 ms | 15.9 ms | 64.0 | 0 | 120,896 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,966.7 | 46.7 | 3,052 ms | 21.4 ms | 128.0 | 0 | 178,944 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 95.6 | 95.6 | 114 ms | 10.5 ms | 1.0 | 0 | 2,867 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 184.9 | 92.6 | 141 ms | 10.8 ms | 2.0 | 0 | 5,548 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 349.8 | 87.4 | 251 ms | 11.4 ms | 4.0 | 0 | 10,492 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 652.8 | 81.4 | 449 ms | 12.3 ms | 8.0 | 0 | 19,592 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,046.8 | 65.6 | 897 ms | 15.2 ms | 16.0 | 0 | 31,392 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,752.8 | 55.0 | 1,821 ms | 18.2 ms | 32.0 | 0 | 52,576 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,688.7 | 42.2 | 3,704 ms | 23.7 ms | 64.0 | 0 | 80,640 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 92.2 | 92.3 | 201 ms | 10.8 ms | 1.0 | 0 | 2,766 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 177.4 | 88.8 | 275 ms | 11.3 ms | 2.0 | 0 | 5,321 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 314.3 | 78.3 | 421 ms | 12.8 ms | 4.0 | 0 | 9,427 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 538.9 | 67.3 | 834 ms | 14.9 ms | 8.0 | 0 | 16,168 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 810.4 | 50.9 | 1,699 ms | 19.6 ms | 16.0 | 0 | 24,304 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,295.1 | 40.9 | 3,411 ms | 24.5 ms | 32.0 | 0 | 38,848 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 88.7 | 89.0 | 375 ms | 11.2 ms | 1.0 | 0 | 2,662 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 160.7 | 80.3 | 474 ms | 12.5 ms | 2.0 | 0 | 4,821 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 274.9 | 68.6 | 877 ms | 14.6 ms | 4.0 | 0 | 8,243 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 449.8 | 56.2 | 1,604 ms | 17.8 ms | 8.0 | 0 | 13,488 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 660.6 | 41.5 | 3,157 ms | 24.1 ms | 16.0 | 0 | 19,808 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 973.3 | 31.0 | 6,467 ms | 32.2 ms | 32.0 | 0 | 29,184 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 75.0 | 75.1 | 733 ms | 13.3 ms | 1.0 | 0 | 2,251 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 137.8 | 68.7 | 1,007 ms | 14.6 ms | 2.0 | 0 | 4,132 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 224.1 | 55.8 | 1,657 ms | 17.9 ms | 4.0 | 0 | 6,719 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 355.2 | 44.3 | 3,293 ms | 22.6 ms | 8.0 | 0 | 10,649 | 30.0 s |
| Sustained decode | 251K | 16 | Valid | 499.2 | 31.7 | 6,530 ms | 31.5 ms | 16.0 | 0 | 14,961 | 30.0 s |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.