Speed test matrix
Ling-3.0-flash-int4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev8719+ge1a24a189 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-flash-int4 --served-model-name ling --trust-remote-code --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 262144 --mem-fraction-static 0.877644 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context20.6K398 ms TTFT
16K context22.4K724 ms TTFT
32K context22.4K1,442 ms TTFT
64K context20.8K3,095 ms TTFT
128K context17.8K7,221 ms TTFT
251K context13.8K18,205 ms TTFT
256K context13.8K18,672 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 197.072 ms TTFT | 332.2113 ms TTFT | 556.2149 ms TTFT | 841.1178 ms TTFT | 1,505.8235 ms TTFT | 2,089.4412 ms TTFT | 3,139.0746 ms TTFT | 3,118.0Requests queued13,369 ms TTFT |
| 16K | 195.794 ms TTFT | 111.9155 ms TTFT | 516.9170 ms TTFT | 833.0232 ms TTFT | 1,478.4383 ms TTFT | 2,007.6725 ms TTFT | 3,064.41,355 ms TTFT | 2,895.7Requests queued12,918 ms TTFT |
| 32K | 194.6121 ms TTFT | 109.2185 ms TTFT | 503.9244 ms TTFT | 921.5449 ms TTFT | 1,432.5800 ms TTFT | 1,844.51,498 ms TTFT | 2,844.92,902 ms TTFT | Over capacity |
| 64K | 190.9183 ms TTFT | 112.2276 ms TTFT | 457.6538 ms TTFT | 863.11,058 ms TTFT | 1,339.81,665 ms TTFT | 1,457.23,290 ms TTFT | Over capacity | Over capacity |
| 128K | 184.9297 ms TTFT | 91.0484 ms TTFT | 355.1818 ms TTFT | 761.81,590 ms TTFT | 1,118.13,169 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 175.5548 ms TTFT | 165.2913 ms TTFT | 464.61,703 ms TTFT | 681.43,300 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.6K | — | 398 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 22.4K | — | 724 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.4K | — | 1,442 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.8K | — | 3,095 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.8K | — | 7,221 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.8K | — | 18,205 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.8K | — | 18,672 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 197.0 | 197.5 | 72 ms | 5.1 ms | 1.0 | 1 | 5,909 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 332.2 | 168.4 | 113 ms | 6.0 ms | 2.2 | 2 | 9,966 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 556.2 | 145.0 | 149 ms | 6.9 ms | 4.9 | 4 | 16,684 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 841.1 | 105.4 | 178 ms | 9.5 ms | 11.3 | 0 | 25,233 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,505.8 | 93.3 | 235 ms | 10.7 ms | 16.0 | 0 | 45,168 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,089.4 | 64.3 | 412 ms | 15.6 ms | 36.3 | 0 | 62,688 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,139.0 | 47.4 | 746 ms | 21.1 ms | 64.0 | 0 | 94,144 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 3,118.0 | 46.6 | 13,369 ms | 21.4 ms | 68.0 | 4 | 93,496 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 195.7 | 196.4 | 94 ms | 5.1 ms | 1.0 | 1 | 5,872 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 111.9 | 56.0 | 155 ms | 17.9 ms | 41.7 | 0 | 3,358 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 516.9 | 128.1 | 170 ms | 7.8 ms | 7.4 | 4 | 15,508 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 833.0 | 104.2 | 232 ms | 9.6 ms | 12.0 | 0 | 24,984 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,478.4 | 91.3 | 383 ms | 11.0 ms | 16.0 | 0 | 44,352 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,007.6 | 61.7 | 725 ms | 16.2 ms | 37.3 | 0 | 60,224 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,064.4 | 45.3 | 1,355 ms | 22.1 ms | 64.0 | 0 | 91,904 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 2,895.7 | 42.8 | 12,918 ms | 23.4 ms | 68.0 | 0 | 86,837 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 194.6 | 195.5 | 121 ms | 5.1 ms | 1.0 | 1 | 5,838 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 109.2 | 54.5 | 185 ms | 18.3 ms | 41.9 | 0 | 3,274 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 503.9 | 116.4 | 244 ms | 8.6 ms | 7.6 | 0 | 15,116 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 921.5 | 115.5 | 449 ms | 8.7 ms | 8.0 | 0 | 27,640 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,432.5 | 90.2 | 800 ms | 11.1 ms | 16.0 | 0 | 42,976 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,844.5 | 56.9 | 1,498 ms | 17.6 ms | 38.9 | 0 | 55,328 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 2,844.9 | 41.4 | 2,902 ms | 24.2 ms | 64.0 | 0 | 85,316 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 190.9 | 192.0 | 183 ms | 5.2 ms | 1.0 | 1 | 5,727 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 112.2 | 54.3 | 276 ms | 18.4 ms | 39.6 | 0 | 3,366 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 457.6 | 107.0 | 538 ms | 9.3 ms | 9.9 | 0 | 13,728 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 863.1 | 109.1 | 1,058 ms | 9.2 ms | 8.0 | 0 | 25,888 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,339.8 | 84.1 | 1,665 ms | 11.9 ms | 16.0 | 0 | 40,192 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,457.2 | 46.3 | 3,290 ms | 21.6 ms | 42.8 | 0 | 43,712 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 184.9 | 186.9 | 297 ms | 5.3 ms | 1.0 | 1 | 5,548 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 91.0 | 45.5 | 484 ms | 22.0 ms | 34.0 | 0 | 2,730 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 355.1 | 84.2 | 818 ms | 11.9 ms | 16.9 | 0 | 10,652 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 761.8 | 95.2 | 1,590 ms | 10.5 ms | 8.8 | 0 | 22,848 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,118.1 | 70.4 | 3,169 ms | 14.2 ms | 17.5 | 0 | 33,536 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 175.5 | 178.8 | 548 ms | 5.6 ms | 1.0 | 1 | 5,265 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 165.2 | 79.8 | 913 ms | 12.5 ms | 13.8 | 0 | 4,956 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 464.6 | 114.7 | 1,703 ms | 8.7 ms | 4.3 | 0 | 13,936 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 681.4 | 88.2 | 3,300 ms | 11.3 ms | 8.0 | 0 | 20,441 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.