Speed test matrix
Ling-3.0-tiny-int4 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev236+g1093c501d · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev236+g1093c501d
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1093c501dfef
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 SGLANG_BATCH_INVARIANT_OPS_ENABLE_MM_DEEPGEMM=0 "$HOME/env_rtxpro6k-inclusionai-sglang-1093c501dfef/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny-int4 --served-model-name ling --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 131072 --mem-fraction-static 0.77 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-deterministic-inference --attention-backend triton
Prefill
prompt tok/s by context length8K context23.6K348 ms TTFT
16K context21.5K753 ms TTFT
32K context15.1K2,133 ms TTFT
64K context9,4086,844 ms TTFT
123K context4,90225,214 ms TTFT
128K context4,66327,580 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 95.260 ms TTFT | Looped | 359.0145 ms TTFT | 684.4172 ms TTFT | 1,244.5225 ms TTFT | Looped | Looped | 5,636.21,567 ms TTFT |
| 16K | 94.574 ms TTFT | 185.2115 ms TTFT | 352.3168 ms TTFT | 668.6240 ms TTFT | 1,219.9472 ms TTFT | 2,175.1926 ms TTFT | 3,501.31,809 ms TTFT | 4,902.73,621 ms TTFT |
| 32K | 93.4111 ms TTFT | 178.9148 ms TTFT | 343.8244 ms TTFT | 645.3460 ms TTFT | 1,130.1924 ms TTFT | 1,929.21,911 ms TTFT | 2,901.63,705 ms TTFT | 3,886.57,564 ms TTFT |
| 64K | 90.0192 ms TTFT | 176.0272 ms TTFT | 336.0437 ms TTFT | 608.0892 ms TTFT | 996.21,884 ms TTFT | 1,568.63,524 ms TTFT | 2,173.36,743 ms TTFT | Over capacity |
| 123K | 86.1345 ms TTFT | 165.4453 ms TTFT | 300.3772 ms TTFT | 513.21,516 ms TTFT | 810.83,170 ms TTFT | 1,173.86,191 ms TTFT | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 23.6K | — | 348 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.5K | — | 753 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.1K | — | 2,133 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,408.0 | — | 6,844 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 4,902.0 | — | 25,214 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,663.0 | — | 27,580 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 95.2 | 95.4 | 60 ms | 10.5 ms | 1.0 | 0 | 2,856 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Valid | 359.0 | 89.8 | 145 ms | 11.1 ms | 4.0 | 0 | 10,768 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 684.4 | 85.7 | 172 ms | 11.7 ms | 8.0 | 0 | 20,528 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,244.5 | 77.9 | 225 ms | 12.8 ms | 16.0 | 0 | 37,328 | 30.0 s |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Valid | 5,636.2 | 44.3 | 1,567 ms | 22.6 ms | 128.0 | 0 | 169,053 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 94.5 | 94.6 | 74 ms | 10.6 ms | 1.0 | 0 | 2,836 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 185.2 | 93.0 | 115 ms | 10.8 ms | 2.0 | 0 | 5,554 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 352.3 | 88.1 | 168 ms | 11.3 ms | 4.0 | 0 | 10,568 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 668.6 | 83.7 | 240 ms | 11.9 ms | 8.0 | 0 | 20,055 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,219.9 | 76.3 | 472 ms | 13.1 ms | 16.0 | 0 | 36,592 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,175.1 | 68.2 | 926 ms | 14.7 ms | 32.0 | 0 | 65,248 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,501.3 | 55.1 | 1,809 ms | 18.2 ms | 64.0 | 0 | 105,024 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,902.7 | 38.5 | 3,621 ms | 25.9 ms | 128.0 | 0 | 147,072 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 93.4 | 93.4 | 111 ms | 10.7 ms | 1.0 | 0 | 2,800 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 178.9 | 89.6 | 148 ms | 11.2 ms | 2.0 | 0 | 5,366 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 343.8 | 85.9 | 244 ms | 11.6 ms | 4.0 | 0 | 10,314 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 645.3 | 80.6 | 460 ms | 12.4 ms | 8.0 | 0 | 19,352 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,130.1 | 70.9 | 924 ms | 14.1 ms | 16.0 | 0 | 33,905 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,929.2 | 60.7 | 1,911 ms | 16.5 ms | 32.0 | 0 | 57,856 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,901.6 | 45.7 | 3,705 ms | 21.9 ms | 64.0 | 0 | 87,050 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,886.5 | 30.9 | 7,564 ms | 32.3 ms | 128.0 | 0 | 116,572 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 90.0 | 90.0 | 192 ms | 11.1 ms | 1.0 | 0 | 2,700 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 176.0 | 88.2 | 272 ms | 11.3 ms | 2.0 | 0 | 5,278 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 336.0 | 83.8 | 437 ms | 11.9 ms | 4.0 | 0 | 10,080 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 608.0 | 76.1 | 892 ms | 13.1 ms | 8.0 | 0 | 18,239 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 996.2 | 62.8 | 1,884 ms | 15.9 ms | 16.0 | 0 | 29,872 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,568.6 | 49.7 | 3,524 ms | 20.1 ms | 32.0 | 0 | 47,040 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 2,173.3 | 34.9 | 6,743 ms | 28.7 ms | 64.0 | 0 | 65,152 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 86.1 | 86.5 | 345 ms | 11.6 ms | 1.0 | 0 | 2,582 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 165.4 | 82.6 | 453 ms | 12.1 ms | 2.0 | 0 | 4,961 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 300.3 | 75.0 | 772 ms | 13.3 ms | 4.0 | 0 | 9,007 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 513.2 | 64.2 | 1,516 ms | 15.6 ms | 8.0 | 0 | 15,391 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 810.8 | 51.1 | 3,170 ms | 19.6 ms | 16.0 | 0 | 24,320 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,173.8 | 37.7 | 6,191 ms | 26.6 ms | 32.0 | 0 | 35,200 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.