Speed test matrix
Ling-3.0-tiny-int4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev8719+ge1a24a189 · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny-int4 --served-model-name ling --tp 1 --reasoning-parser deepseek-r1 --tool-call-parser glm45 --context-length 131072 --mem-fraction-static 0.871460 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context55.6K148 ms TTFT
16K context69.1K235 ms TTFT
32K context73.6K439 ms TTFT
64K context68.4K941 ms TTFT
123K context57.2K2,162 ms TTFT
128K context57.3K2,245 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 439.464 ms TTFT | 825.885 ms TTFT | 1,502.791 ms TTFT | 2,561.3154 ms TTFT | 4,418.1235 ms TTFT | 6,818.5435 ms TTFT | 10.8K787 ms TTFT | 13.3K1,417 ms TTFT |
| 16K | 434.778 ms TTFT | 820.0105 ms TTFT | 1,450.0110 ms TTFT | 2,425.8216 ms TTFT | 4,121.6398 ms TTFT | 5,700.4752 ms TTFT | 8,442.11,389 ms TTFT | 9,370.62,786 ms TTFT |
| 32K | 426.0104 ms TTFT | 501.8157 ms TTFT | 1,418.4197 ms TTFT | 2,356.8179 ms TTFT | 3,796.6674 ms TTFT | 5,795.51,023 ms TTFT | 7,096.63,347 ms TTFT | 10.5K7,037 ms TTFT |
| 64K | 412.5145 ms TTFT | 619.7225 ms TTFT | 1,316.9273 ms TTFT | 2,113.1259 ms TTFT | 3,065.11,113 ms TTFT | 3,993.13,323 ms TTFT | 3,776.36,562 ms TTFT | 4,191.212,996 ms TTFT |
| 123K | 390.4267 ms TTFT | 102.0480 ms TTFT | 527.8864 ms TTFT | 1,653.81,117 ms TTFT | 2,594.52,266 ms TTFT | 1,864.66,543 ms TTFT | 2,526.313,502 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 55.6K | — | 148 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 69.1K | — | 235 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 73.6K | — | 439 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 68.4K | — | 941 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 57.2K | — | 2,162 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 57.3K | — | 2,245 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 439.4 | 442.5 | 64 ms | 2.3 ms | 1.0 | 3 | 13,183 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 825.8 | 421.1 | 85 ms | 2.4 ms | 2.1 | 6 | 24,774 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,502.7 | 378.8 | 91 ms | 2.6 ms | 4.0 | 12 | 45,081 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,561.3 | 328.1 | 154 ms | 3.0 ms | 8.4 | 16 | 76,840 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 4,418.1 | 281.0 | 235 ms | 3.6 ms | 15.8 | 32 | 132,544 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 6,818.5 | 215.8 | 435 ms | 4.7 ms | 36.3 | 32 | 204,545 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 10.8K | 173.1 | 787 ms | 5.8 ms | 64.0 | 64 | 325,312 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 13.3K | 102.8 | 1,417 ms | 9.7 ms | 168.5 | 0 | 397,984 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 434.7 | 438.1 | 78 ms | 2.3 ms | 1.0 | 3 | 13,041 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 820.0 | 415.3 | 105 ms | 2.4 ms | 1.9 | 6 | 24,600 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,450.0 | 369.4 | 110 ms | 2.7 ms | 4.1 | 12 | 43,496 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,425.8 | 314.9 | 216 ms | 3.2 ms | 8.9 | 16 | 72,772 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 4,121.6 | 264.8 | 398 ms | 3.8 ms | 16.2 | 32 | 123,641 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,700.4 | 186.1 | 752 ms | 5.4 ms | 40.0 | 32 | 171,019 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 8,442.1 | 135.3 | 1,389 ms | 7.4 ms | 74.4 | 64 | 253,241 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 9,370.6 | 72.8 | 2,786 ms | 13.7 ms | 192.0 | 0 | 281,184 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 426.0 | 430.7 | 104 ms | 2.3 ms | 1.0 | 3 | 12,779 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 501.8 | 284.7 | 157 ms | 4.3 ms | 78.3 | 2 | 15,054 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,418.4 | 359.1 | 197 ms | 2.8 ms | 3.9 | 8 | 42,548 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,356.8 | 299.0 | 179 ms | 3.3 ms | 7.9 | 16 | 70,703 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,796.6 | 238.8 | 674 ms | 4.2 ms | 16.6 | 16 | 113,901 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 5,795.5 | 184.8 | 1,023 ms | 5.4 ms | 31.9 | 32 | 173,853 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 7,096.6 | 111.3 | 3,347 ms | 9.0 ms | 76.2 | 0 | 212,884 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 10.5K | 87.6 | 7,037 ms | 11.4 ms | 128.0 | 0 | 315,008 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 412.5 | 418.7 | 145 ms | 2.4 ms | 1.0 | 3 | 12,376 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 619.7 | 390.5 | 225 ms | 2.6 ms | 29.7 | 4 | 18,589 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,316.9 | 337.2 | 273 ms | 3.0 ms | 4.1 | 8 | 39,505 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,113.1 | 270.6 | 259 ms | 3.7 ms | 7.8 | 16 | 63,394 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 3,065.1 | 200.5 | 1,113 ms | 5.0 ms | 18.6 | 16 | 91,950 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,993.1 | 124.3 | 3,323 ms | 8.0 ms | 35.2 | 32 | 120,567 | 30.2 s |
| Sustained decode | 64K | 64 | Valid | 3,776.3 | 64.1 | 6,562 ms | 15.6 ms | 96.0 | 0 | 113,281 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 4,191.2 | 37.7 | 12,996 ms | 26.6 ms | 204.7 | 0 | 125,695 | 30.0 s |
| Sustained decode | 123K | 1 | Valid | 390.4 | 401.0 | 267 ms | 2.5 ms | 0.9 | 3 | 11,643 | 29.8 s |
| Sustained decode | 123K | 2 | Valid | 102.0 | 49.0 | 480 ms | 20.4 ms | 133.6 | 0 | 3,059 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 527.8 | 120.7 | 864 ms | 8.3 ms | 76.9 | 4 | 15,780 | 29.9 s |
| Sustained decode | 123K | 8 | Valid | 1,653.8 | 217.4 | 1,117 ms | 4.6 ms | 9.5 | 8 | 49,614 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 2,594.5 | 168.6 | 2,266 ms | 5.9 ms | 15.7 | 16 | 77,831 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,864.6 | 63.9 | 6,543 ms | 15.6 ms | 46.1 | 0 | 55,937 | 30.0 s |
| Sustained decode | 123K | 64 | Valid | 2,526.3 | 46.1 | 13,502 ms | 21.7 ms | 88.2 | 0 | 75,776 | 30.0 s |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.