Speed test matrix
Ling-2.6-flash on H200 NVL 141GB ×2
sglang · 0.5.6.post3.dev8719+ge1a24a189 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-2.6-flash --served-model-name ling --trust-remote-code --tp 2 --tool-call-parser qwen --context-length 131072 --mem-fraction-static 0.874548 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context25.5K322 ms TTFT
16K context30.4K533 ms TTFT
32K context31.7K1,020 ms TTFT
64K context31.3K2,058 ms TTFT
123K context29.4K4,206 ms TTFT
128K context29.3K4,389 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 207.683 ms TTFT | 329.5105 ms TTFT | 490.4132 ms TTFT | 647.5168 ms TTFT | 1,067.6244 ms TTFT | 1,485.5377 ms TTFT | 2,810.2Requests queued1,366 ms TTFT | 2,891.5Requests queued8,674 ms TTFT |
| 16K | 206.990 ms TTFT | 78.1138 ms TTFT | 166.1172 ms TTFT | 436.1227 ms TTFT | 920.9391 ms TTFT | 1,455.0765 ms TTFT | 2,343.2Requests queued8,918 ms TTFT | 2,270.5Requests queued19,032 ms TTFT |
| 32K | 206.2116 ms TTFT | 94.2192 ms TTFT | 242.0249 ms TTFT | 656.1414 ms TTFT | 975.2798 ms TTFT | 1,286.91,537 ms TTFT | 1,766.4Requests queued11,160 ms TTFT | Over capacity |
| 64K | 204.1163 ms TTFT | 67.9285 ms TTFT | 217.1450 ms TTFT | 543.4842 ms TTFT | 894.01,714 ms TTFT | 1,150.13,201 ms TTFT | Over capacity | Over capacity |
| 123K | 200.1287 ms TTFT | 73.6479 ms TTFT | 162.8821 ms TTFT | 517.11,609 ms TTFT | 772.23,192 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 25.5K | — | 322 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 30.4K | — | 533 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 31.7K | — | 1,020 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 31.3K | — | 2,058 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 29.4K | — | 4,206 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 29.3K | — | 4,389 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 207.6 | 208.3 | 83 ms | 4.8 ms | 1.0 | 1 | 6,226 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 329.5 | 163.3 | 105 ms | 6.1 ms | 2.1 | 2 | 9,884 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 490.4 | 133.9 | 132 ms | 7.5 ms | 5.1 | 4 | 14,712 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 647.5 | 81.3 | 168 ms | 12.3 ms | 12.0 | 0 | 19,432 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,067.6 | 65.5 | 244 ms | 15.3 ms | 20.8 | 0 | 32,032 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,485.5 | 46.8 | 377 ms | 21.4 ms | 48.0 | 0 | 44,544 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 2,810.2 | 41.2 | 1,366 ms | 24.3 ms | 65.0 | 0 | 84,291 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 2,891.5 | 39.5 | 8,674 ms | 25.3 ms | 78.4 | 0 | 86,742 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 206.9 | 207.6 | 90 ms | 4.8 ms | 1.0 | 1 | 6,207 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 78.1 | 39.2 | 138 ms | 25.5 ms | 68.0 | 0 | 2,344 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 166.1 | 41.3 | 172 ms | 24.2 ms | 64.1 | 0 | 4,976 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 436.1 | 52.7 | 227 ms | 19.0 ms | 41.4 | 0 | 13,080 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 920.9 | 56.9 | 391 ms | 17.6 ms | 24.9 | 0 | 27,632 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,455.0 | 45.7 | 765 ms | 21.9 ms | 48.0 | 0 | 43,584 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 2,343.2 | 37.6 | 8,918 ms | 26.6 ms | 68.0 | 0 | 70,302 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 2,270.5 | 35.9 | 19,032 ms | 27.9 ms | 79.0 | 0 | 68,111 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 206.2 | 207.0 | 116 ms | 4.8 ms | 1.0 | 1 | 6,185 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 94.2 | 46.6 | 192 ms | 21.5 ms | 36.1 | 0 | 2,824 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 242.0 | 59.0 | 249 ms | 17.0 ms | 25.7 | 0 | 7,265 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 656.1 | 81.1 | 414 ms | 12.3 ms | 10.7 | 0 | 19,696 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 975.2 | 60.8 | 798 ms | 16.4 ms | 20.0 | 0 | 29,248 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,286.9 | 40.8 | 1,537 ms | 24.5 ms | 48.0 | 0 | 38,592 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,766.4 | 33.7 | 11,160 ms | 29.7 ms | 75.3 | 0 | 52,970 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 204.1 | 205.2 | 163 ms | 4.9 ms | 1.0 | 1 | 6,122 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 67.9 | 33.8 | 285 ms | 29.6 ms | 63.9 | 0 | 2,036 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 217.1 | 53.6 | 450 ms | 18.7 ms | 23.0 | 0 | 6,512 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 543.4 | 65.6 | 842 ms | 15.2 ms | 15.4 | 0 | 16,288 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 894.0 | 57.0 | 1,714 ms | 17.6 ms | 22.4 | 0 | 26,816 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,150.1 | 37.4 | 3,201 ms | 26.8 ms | 48.0 | 0 | 34,464 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 200.1 | 202.2 | 287 ms | 4.9 ms | 1.0 | 1 | 6,002 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 73.6 | 36.4 | 479 ms | 27.5 ms | 41.4 | 0 | 2,206 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 162.8 | 41.0 | 821 ms | 24.4 ms | 37.5 | 0 | 4,880 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 517.1 | 65.7 | 1,609 ms | 15.2 ms | 12.9 | 0 | 15,504 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 772.2 | 50.5 | 3,192 ms | 19.8 ms | 23.5 | 0 | 23,168 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.