Speed test matrix
Ling-3.0-tiny-fp8 on RTX PRO 6000 Blackwell Server Edition 96GB
sglang · 0.5.22.dev236+g1093c501d · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.22.dev236+g1093c501d
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
1093c501dfef
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 SGLANG_BATCH_INVARIANT_OPS_ENABLE_MM_DEEPGEMM=0 "$HOME/env_rtxpro6k-inclusionai-sglang-1093c501dfef/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny-fp8 --served-model-name ling --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 131072 --mem-fraction-static 0.77 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-deterministic-inference --attention-backend triton --fp8-gemm-backend cutlass --moe-runner-backend triton
Prefill
prompt tok/s by context length8K context23.1K356 ms TTFT
16K context21K772 ms TTFT
32K context14.9K2,172 ms TTFT
64K context9,3096,917 ms TTFT
123K context4,91025,172 ms TTFT
128K context4,61227,887 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 122.864 ms TTFT | 238.0408 ms TTFT | 469.0156 ms TTFT | 899.5170 ms TTFT | 1,582.4237 ms TTFT | 2,895.0416 ms TTFT | 4,617.3837 ms TTFT | 6,226.51,651 ms TTFT |
| 16K | 121.777 ms TTFT | 237.1118 ms TTFT | 457.0165 ms TTFT | 870.9224 ms TTFT | 1,530.2388 ms TTFT | 2,671.2790 ms TTFT | 4,046.01,547 ms TTFT | 5,334.02,978 ms TTFT |
| 32K | 119.7115 ms TTFT | 224.4142 ms TTFT | 404.0252 ms TTFT | 767.7445 ms TTFT | 1,197.6883 ms TTFT | 2,026.71,731 ms TTFT | 2,987.53,435 ms TTFT | 3,729.76,876 ms TTFT |
| 64K | 115.0190 ms TTFT | 213.4261 ms TTFT | 373.9486 ms TTFT | 628.7921 ms TTFT | 916.91,776 ms TTFT | 1,426.33,728 ms TTFT | 1,971.87,049 ms TTFT | Over capacity |
| 123K | 108.7335 ms TTFT | 194.2461 ms TTFT | 329.2794 ms TTFT | 528.91,569 ms TTFT | 762.53,130 ms TTFT | 1,096.16,440 ms TTFT | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 23.1K | — | 356 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21K | — | 772 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.9K | — | 2,172 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,309.0 | — | 6,917 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 4,910.0 | — | 25,172 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,612.0 | — | 27,887 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 122.8 | 123.6 | 64 ms | 8.1 ms | 1.0 | 1 | 3,683 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 238.0 | 118.1 | 408 ms | 8.5 ms | 2.0 | 0 | 7,140 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 469.0 | 117.3 | 156 ms | 8.5 ms | 4.0 | 0 | 14,067 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 899.5 | 112.5 | 170 ms | 8.9 ms | 8.0 | 0 | 27,001 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,582.4 | 98.9 | 237 ms | 10.1 ms | 16.0 | 0 | 47,473 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,895.0 | 90.4 | 416 ms | 11.1 ms | 32.0 | 0 | 86,848 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 4,617.3 | 72.4 | 837 ms | 13.8 ms | 64.0 | 0 | 138,496 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 6,226.5 | 48.8 | 1,651 ms | 20.5 ms | 128.0 | 0 | 186,752 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 121.7 | 121.8 | 77 ms | 8.2 ms | 1.0 | 0 | 3,651 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 237.1 | 119.3 | 118 ms | 8.4 ms | 2.0 | 2 | 7,113 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 457.0 | 114.3 | 165 ms | 8.7 ms | 4.0 | 0 | 13,710 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 870.9 | 108.8 | 224 ms | 9.2 ms | 8.0 | 0 | 26,120 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,530.2 | 95.4 | 388 ms | 10.5 ms | 16.0 | 0 | 45,904 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,671.2 | 83.5 | 790 ms | 12.0 ms | 32.0 | 0 | 80,136 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 4,046.0 | 63.3 | 1,547 ms | 15.8 ms | 64.0 | 0 | 121,344 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,334.0 | 41.7 | 2,978 ms | 24.0 ms | 128.0 | 0 | 160,000 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 119.7 | 119.7 | 115 ms | 8.4 ms | 1.0 | 0 | 3,590 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 224.4 | 112.2 | 142 ms | 8.9 ms | 2.0 | 0 | 6,729 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 404.0 | 100.8 | 252 ms | 9.9 ms | 4.0 | 0 | 12,119 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 767.7 | 95.8 | 445 ms | 10.4 ms | 8.0 | 0 | 23,033 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,197.6 | 75.0 | 883 ms | 13.3 ms | 16.0 | 0 | 35,920 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,026.7 | 63.4 | 1,731 ms | 15.8 ms | 32.0 | 0 | 60,800 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,987.5 | 46.8 | 3,435 ms | 21.4 ms | 64.0 | 0 | 89,600 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 3,729.7 | 29.2 | 6,876 ms | 34.3 ms | 128.0 | 0 | 111,899 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 115.0 | 115.2 | 190 ms | 8.7 ms | 1.0 | 0 | 3,449 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 213.4 | 106.9 | 261 ms | 9.4 ms | 2.0 | 0 | 6,402 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 373.9 | 93.3 | 486 ms | 10.7 ms | 4.0 | 0 | 11,216 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 628.7 | 78.8 | 921 ms | 12.7 ms | 8.0 | 0 | 18,856 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 916.9 | 57.7 | 1,776 ms | 17.3 ms | 16.0 | 0 | 27,504 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,426.3 | 45.1 | 3,728 ms | 22.2 ms | 32.0 | 0 | 42,784 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,971.8 | 31.8 | 7,049 ms | 31.5 ms | 64.0 | 0 | 59,136 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 108.7 | 109.0 | 335 ms | 9.2 ms | 1.0 | 0 | 3,262 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 194.2 | 97.0 | 461 ms | 10.3 ms | 2.0 | 0 | 5,828 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 329.2 | 82.2 | 794 ms | 12.2 ms | 4.0 | 0 | 9,872 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 528.9 | 66.2 | 1,569 ms | 15.1 ms | 8.0 | 0 | 15,864 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 762.5 | 48.0 | 3,130 ms | 20.8 ms | 16.0 | 0 | 22,864 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,096.1 | 35.2 | 6,440 ms | 28.4 ms | 32.0 | 0 | 32,864 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.