Speed test matrix
Ling-3.0-tiny-fp8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev8719+ge1a24a189 · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-tiny-fp8 --served-model-name ling --tp 1 --reasoning-parser deepseek-r1 --tool-call-parser glm45 --context-length 131072 --mem-fraction-static 0.871859 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context59.5K138 ms TTFT
16K context88.4K183 ms TTFT
32K context90K359 ms TTFT
64K context83.9K768 ms TTFT
123K context68.7K1,799 ms TTFT
128K context67.4K1,908 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 412.056 ms TTFT | 744.181 ms TTFT | 1,316.3123 ms TTFT | 2,239.5159 ms TTFT | 3,707.0219 ms TTFT | 6,266.6391 ms TTFT | 9,565.9772 ms TTFT | 11.4K1,391 ms TTFT |
| 16K | 406.971 ms TTFT | 709.0112 ms TTFT | 1,329.9138 ms TTFT | 2,164.1216 ms TTFT | 3,490.2393 ms TTFT | 5,785.1742 ms TTFT | 8,165.81,402 ms TTFT | 9,066.92,615 ms TTFT |
| 32K | 400.6101 ms TTFT | 419.1161 ms TTFT | 1,291.0166 ms TTFT | 2,043.1223 ms TTFT | 3,106.1642 ms TTFT | 4,811.11,126 ms TTFT | 5,438.83,491 ms TTFT | 8,938.76,119 ms TTFT |
| 64K | 388.5151 ms TTFT | 409.1238 ms TTFT | 1,210.2271 ms TTFT | 1,859.6631 ms TTFT | 2,891.01,320 ms TTFT | 3,615.93,411 ms TTFT | 5,723.36,814 ms TTFT | 6,954.813,585 ms TTFT |
| 123K | 371.3220 ms TTFT | 106.3441 ms TTFT | 1,035.2595 ms TTFT | 1,493.01,101 ms TTFT | 2,228.71,929 ms TTFT | 1,663.26,196 ms TTFT | 2,339.412,765 ms TTFT | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 59.5K | — | 138 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 88.4K | — | 183 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 90K | — | 359 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 83.9K | — | 768 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 68.7K | — | 1,799 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 67.4K | — | 1,908 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 412.0 | 414.1 | 56 ms | 2.4 ms | 1.0 | 3 | 12,359 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 744.1 | 388.9 | 81 ms | 2.6 ms | 2.2 | 6 | 22,324 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,316.3 | 342.3 | 123 ms | 2.9 ms | 4.3 | 8 | 39,488 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,239.5 | 282.0 | 159 ms | 3.5 ms | 7.8 | 16 | 67,184 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 3,707.0 | 231.8 | 219 ms | 4.3 ms | 17.3 | 16 | 111,200 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 6,266.6 | 199.9 | 391 ms | 5.0 ms | 32.0 | 32 | 188,002 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 9,565.9 | 156.8 | 772 ms | 6.4 ms | 67.9 | 64 | 286,978 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 11.4K | 89.2 | 1,391 ms | 11.2 ms | 187.7 | 0 | 340,864 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 406.9 | 409.7 | 71 ms | 2.4 ms | 1.0 | 3 | 12,207 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 709.0 | 384.3 | 112 ms | 2.6 ms | 14.8 | 4 | 21,270 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,329.9 | 333.9 | 138 ms | 3.0 ms | 4.0 | 8 | 39,896 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,164.1 | 275.0 | 216 ms | 3.6 ms | 7.7 | 16 | 64,928 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 3,490.2 | 220.5 | 393 ms | 4.5 ms | 17.7 | 16 | 104,706 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 5,785.1 | 185.3 | 742 ms | 5.4 ms | 32.0 | 32 | 173,536 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 8,165.8 | 125.9 | 1,402 ms | 7.9 ms | 75.6 | 64 | 236,864 | 29.0 s |
| Sustained decode | 16K | 128 | Valid | 9,066.9 | 71.3 | 2,615 ms | 14.0 ms | 192.0 | 0 | 272,000 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 400.6 | 404.6 | 101 ms | 2.5 ms | 1.0 | 3 | 12,017 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 419.1 | 262.7 | 161 ms | 4.7 ms | 95.9 | 2 | 12,571 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,291.0 | 325.8 | 166 ms | 3.1 ms | 4.0 | 8 | 38,728 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,043.1 | 258.4 | 223 ms | 3.9 ms | 7.9 | 16 | 61,296 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 3,106.1 | 199.5 | 642 ms | 5.0 ms | 19.1 | 16 | 93,186 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 4,811.1 | 150.2 | 1,126 ms | 6.7 ms | 34.4 | 32 | 144,337 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 5,438.8 | 86.9 | 3,491 ms | 11.5 ms | 93.3 | 0 | 163,168 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 8,938.7 | 70.2 | 6,119 ms | 14.2 ms | 149.0 | 0 | 268,160 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 388.5 | 394.2 | 151 ms | 2.5 ms | 1.0 | 3 | 11,655 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 409.1 | 256.1 | 238 ms | 4.7 ms | 70.2 | 2 | 12,272 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,210.2 | 307.7 | 271 ms | 3.2 ms | 4.0 | 8 | 36,306 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 1,859.6 | 234.4 | 631 ms | 4.3 ms | 8.2 | 8 | 55,784 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 2,891.0 | 187.2 | 1,320 ms | 5.3 ms | 15.8 | 16 | 86,727 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 3,615.9 | 113.2 | 3,411 ms | 8.8 ms | 38.0 | 0 | 108,480 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 5,723.3 | 96.3 | 6,814 ms | 10.4 ms | 64.0 | 0 | 171,712 | 30.0 s |
| Sustained decode | 64K | 128 | Valid | 6,954.8 | 57.5 | 13,585 ms | 17.4 ms | 129.2 | 0 | 208,656 | 30.0 s |
| Sustained decode | 123K | 1 | Valid | 371.3 | 376.9 | 220 ms | 2.7 ms | 1.0 | 3 | 11,042 | 29.7 s |
| Sustained decode | 123K | 2 | Valid | 106.3 | 53.3 | 441 ms | 18.8 ms | 128.8 | 0 | 3,190 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 1,035.2 | 278.3 | 595 ms | 3.6 ms | 4.2 | 8 | 31,056 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 1,493.0 | 197.0 | 1,101 ms | 5.1 ms | 9.6 | 8 | 44,792 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 2,228.7 | 148.2 | 1,929 ms | 6.7 ms | 17.2 | 16 | 66,881 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,663.2 | 57.4 | 6,196 ms | 17.4 ms | 48.0 | 0 | 49,888 | 30.0 s |
| Sustained decode | 123K | 64 | Valid | 2,339.4 | 42.5 | 12,765 ms | 23.5 ms | 98.0 | 0 | 70,144 | 30.0 s |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.