Speed test matrix
Ling-3.0-tiny on H200 NVL 141GB
sglang · 0.5.6.post3.dev8719+ge1a24a189 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny --served-model-name ling --tp 1 --reasoning-parser deepseek-r1 --tool-call-parser glm45 --context-length 131072 --mem-fraction-static 0.872504 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context55.5K148 ms TTFT
16K context74.1K219 ms TTFT
32K context78.2K413 ms TTFT
64K context72.8K885 ms TTFT
123K context60K2,060 ms TTFT
128K context60.4K2,129 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 411.358 ms TTFT | 733.081 ms TTFT | 1,249.595 ms TTFT | 2,079.5147 ms TTFT | 3,145.1214 ms TTFT | 5,450.6422 ms TTFT | 7,925.3839 ms TTFT | 9,799.41,271 ms TTFT |
| 16K | 406.275 ms TTFT | 510.2107 ms TTFT | 1,275.2141 ms TTFT | 2,008.6229 ms TTFT | 2,977.4381 ms TTFT | 4,875.9697 ms TTFT | 6,567.11,559 ms TTFT | 11.5K2,939 ms TTFT |
| 32K | 399.5101 ms TTFT | 613.7140 ms TTFT | 1,215.0210 ms TTFT | 1,890.2456 ms TTFT | 2,891.4630 ms TTFT | 4,133.31,293 ms TTFT | 4,895.13,468 ms TTFT | 6,908.55,610 ms TTFT |
| 64K | 387.0159 ms TTFT | 198.2265 ms TTFT | 1,162.2243 ms TTFT | 1,724.6689 ms TTFT | 2,572.71,024 ms TTFT | 2,670.23,294 ms TTFT | 4,864.86,588 ms TTFT | Over capacity |
| 123K | 368.8255 ms TTFT | 356.4454 ms TTFT | 1,018.0468 ms TTFT | 1,379.01,054 ms TTFT | 1,954.93,223 ms TTFT | 1,591.26,516 ms TTFT | 2,232.513,141 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 55.5K | — | 148 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 74.1K | — | 219 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 78.2K | — | 413 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 72.8K | — | 885 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 60K | — | 2,060 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 60.4K | — | 2,129 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 411.3 | 413.5 | 58 ms | 2.4 ms | 1.0 | 3 | 12,339 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 733.0 | 386.4 | 81 ms | 2.6 ms | 2.2 | 6 | 21,988 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,249.5 | 328.1 | 95 ms | 3.0 ms | 4.4 | 8 | 37,484 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,079.5 | 264.0 | 147 ms | 3.8 ms | 7.8 | 16 | 62,385 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,145.1 | 199.2 | 214 ms | 5.0 ms | 18.9 | 16 | 94,356 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 5,450.6 | 172.8 | 422 ms | 5.8 ms | 33.6 | 32 | 163,522 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 7,925.3 | 122.9 | 839 ms | 8.1 ms | 78.7 | 64 | 232,128 | 29.3 s |
| Sustained decode | 8K | 128 | Valid | 9,799.4 | 77.2 | 1,271 ms | 13.0 ms | 192.0 | 0 | 293,984 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 406.2 | 409.2 | 75 ms | 2.4 ms | 1.0 | 3 | 12,185 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 510.2 | 275.8 | 107 ms | 4.3 ms | 74.5 | 2 | 15,304 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,275.2 | 322.6 | 141 ms | 3.1 ms | 4.0 | 8 | 38,253 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,008.6 | 256.1 | 229 ms | 3.9 ms | 8.0 | 16 | 60,260 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,977.4 | 190.6 | 381 ms | 5.3 ms | 18.9 | 16 | 89,312 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 4,875.9 | 160.0 | 697 ms | 6.3 ms | 35.6 | 32 | 146,283 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 6,567.1 | 102.5 | 1,559 ms | 9.8 ms | 89.6 | 0 | 197,088 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 11.5K | 87.9 | 2,939 ms | 11.4 ms | 132.3 | 0 | 346,288 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 399.5 | 403.6 | 101 ms | 2.5 ms | 1.0 | 3 | 11,985 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 613.7 | 374.8 | 140 ms | 2.7 ms | 31.8 | 4 | 18,409 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,215.0 | 313.7 | 210 ms | 3.2 ms | 4.1 | 8 | 36,447 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 1,890.2 | 235.8 | 456 ms | 4.2 ms | 8.3 | 8 | 56,702 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,891.4 | 184.5 | 630 ms | 5.4 ms | 15.9 | 16 | 86,745 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,133.3 | 129.7 | 1,293 ms | 7.7 ms | 36.7 | 32 | 124,000 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 4,895.1 | 79.3 | 3,468 ms | 12.6 ms | 96.0 | 0 | 146,816 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 6,908.5 | 55.0 | 5,610 ms | 18.2 ms | 176.4 | 0 | 207,264 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 387.0 | 393.0 | 159 ms | 2.5 ms | 1.0 | 3 | 11,610 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 198.2 | 95.2 | 265 ms | 10.5 ms | 114.3 | 0 | 5,946 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,162.2 | 296.0 | 243 ms | 3.4 ms | 4.0 | 8 | 34,865 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,724.6 | 217.5 | 689 ms | 4.6 ms | 8.5 | 8 | 51,735 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,572.7 | 164.6 | 1,024 ms | 6.1 ms | 15.9 | 16 | 77,176 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,670.2 | 86.0 | 3,294 ms | 11.6 ms | 44.7 | 0 | 80,096 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 4,864.8 | 77.9 | 6,588 ms | 12.8 ms | 70.8 | 0 | 145,920 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 368.8 | 375.5 | 255 ms | 2.7 ms | 0.9 | 3 | 11,032 | 29.9 s |
| Sustained decode | 123K | 2 | Valid | 356.4 | 238.8 | 454 ms | 5.1 ms | 36.3 | 2 | 10,691 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 1,018.0 | 268.1 | 468 ms | 3.7 ms | 4.2 | 8 | 30,540 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 1,379.0 | 184.8 | 1,054 ms | 5.4 ms | 9.8 | 8 | 41,367 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 1,954.9 | 129.2 | 3,223 ms | 7.7 ms | 18.5 | 14 | 57,610 | 29.5 s |
| Sustained decode | 123K | 32 | Valid | 1,591.2 | 55.8 | 6,516 ms | 17.9 ms | 46.0 | 0 | 47,712 | 30.0 s |
| Sustained decode | 123K | 64 | Valid | 2,232.5 | 40.8 | 13,141 ms | 24.5 ms | 104.7 | 0 | 66,944 | 30.0 s |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.