Speed test matrix
Ling-2.6-flash-base on H200 NVL 141GB
sglang · 0.5.6.post3.dev8719+ge1a24a189 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-2.6-flash-base --served-model-name ling --trust-remote-code --tp 1 --tool-call-parser qwen --context-length 262144 --mem-fraction-static 0.880548 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context25.7K318 ms TTFT
16K context29.1K558 ms TTFT
32K context30.9K1,045 ms TTFT
64K context29.4K2,189 ms TTFT
128K context26.2K4,914 ms TTFT
251K context21.2K11,913 ms TTFT
256K context21K12,255 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 184.078 ms TTFT | 243.9121 ms TTFT | 352.0131 ms TTFT | 642.2173 ms TTFT | 950.4235 ms TTFT | 1,483.3392 ms TTFT | 1,664.9Requests queued11,536 ms TTFT | 618.3Requests queued41,386 ms TTFT |
| 16K | 183.495 ms TTFT | 94.1167 ms TTFT | 303.0189 ms TTFT | 624.7219 ms TTFT | 902.7389 ms TTFT | 1,471.7699 ms TTFT | 1,563.6Requests queued10,579 ms TTFT | 568.0Requests queued46,801 ms TTFT |
| 32K | 182.4125 ms TTFT | 89.5180 ms TTFT | 307.5276 ms TTFT | 600.0395 ms TTFT | 836.3755 ms TTFT | 1,416.91,970 ms TTFT | 1,540.5Requests queued9,957 ms TTFT | Over capacity |
| 64K | 180.7173 ms TTFT | 81.5263 ms TTFT | 166.8439 ms TTFT | 483.1887 ms TTFT | 745.31,682 ms TTFT | 1,362.46,013 ms TTFT | Over capacity | Over capacity |
| 128K | 176.8283 ms TTFT | 127.2472 ms TTFT | 349.1854 ms TTFT | 586.91,612 ms TTFT | 744.73,110 ms TTFT | Over capacity | Over capacity | Over capacity |
| 251K | 171.2522 ms TTFT | 86.4935 ms TTFT | 240.51,689 ms TTFT | 496.93,452 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 25.7K | — | 318 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.1K | — | 558 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.9K | — | 1,045 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.4K | — | 2,189 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 26.2K | — | 4,914 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 21.2K | — | 11,913 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 21K | — | 12,255 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 184.0 | 184.5 | 78 ms | 5.4 ms | 1.0 | 1 | 5,519 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 243.9 | 130.1 | 121 ms | 7.7 ms | 2.4 | 2 | 7,316 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 352.0 | 88.2 | 131 ms | 11.3 ms | 6.0 | 0 | 10,560 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 642.2 | 77.8 | 173 ms | 12.9 ms | 9.4 | 0 | 19,264 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 950.4 | 58.8 | 235 ms | 17.0 ms | 20.8 | 0 | 28,512 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,483.3 | 45.2 | 392 ms | 22.1 ms | 32.0 | 0 | 44,481 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,664.9 | 43.2 | 11,536 ms | 23.2 ms | 39.0 | 0 | 49,920 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 618.3 | 40.7 | 41,386 ms | 24.6 ms | 38.8 | 0 | 18,543 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 183.4 | 184.0 | 95 ms | 5.4 ms | 1.0 | 1 | 5,501 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 94.1 | 47.0 | 167 ms | 21.3 ms | 32.0 | 0 | 2,822 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 303.0 | 71.8 | 189 ms | 13.9 ms | 11.6 | 0 | 9,088 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 624.7 | 77.1 | 219 ms | 13.0 ms | 10.0 | 0 | 18,736 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 902.7 | 56.1 | 389 ms | 17.8 ms | 21.1 | 0 | 27,072 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,471.7 | 43.7 | 699 ms | 22.9 ms | 32.0 | 0 | 44,128 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,563.6 | 40.6 | 10,579 ms | 24.6 ms | 38.7 | 0 | 46,878 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 568.0 | 41.2 | 46,801 ms | 24.3 ms | 39.0 | 0 | 17,038 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 182.4 | 183.2 | 125 ms | 5.5 ms | 1.0 | 1 | 5,472 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 89.5 | 44.7 | 180 ms | 22.4 ms | 32.0 | 0 | 2,684 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 307.5 | 72.7 | 276 ms | 13.8 ms | 11.7 | 0 | 9,224 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 600.0 | 74.5 | 395 ms | 13.4 ms | 10.0 | 0 | 17,992 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 836.3 | 52.2 | 755 ms | 19.2 ms | 22.1 | 0 | 25,088 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,416.9 | 41.8 | 1,970 ms | 23.9 ms | 32.0 | 0 | 42,496 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,540.5 | 40.6 | 9,957 ms | 24.6 ms | 38.9 | 0 | 46,203 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 180.7 | 181.8 | 173 ms | 5.5 ms | 1.0 | 1 | 5,419 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 81.5 | 40.9 | 263 ms | 24.4 ms | 36.0 | 0 | 2,444 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 166.8 | 41.4 | 439 ms | 24.2 ms | 31.0 | 0 | 5,004 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 483.1 | 58.1 | 887 ms | 17.2 ms | 17.4 | 0 | 14,488 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 745.3 | 46.9 | 1,682 ms | 21.3 ms | 24.2 | 0 | 22,352 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,362.4 | 39.1 | 6,013 ms | 25.6 ms | 32.0 | 0 | 40,864 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 176.8 | 178.6 | 283 ms | 5.6 ms | 1.0 | 1 | 5,305 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 127.2 | 60.3 | 472 ms | 16.6 ms | 13.3 | 0 | 3,817 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 349.1 | 83.9 | 854 ms | 11.9 ms | 5.6 | 0 | 10,473 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 586.9 | 73.4 | 1,612 ms | 13.6 ms | 9.4 | 0 | 17,608 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 744.7 | 48.1 | 3,110 ms | 20.8 ms | 22.8 | 0 | 22,336 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 171.2 | 174.4 | 522 ms | 5.7 ms | 1.0 | 1 | 5,137 | 30.0 s |
| Sustained decode | 251K | 2 | Valid | 86.4 | 43.4 | 935 ms | 23.0 ms | 18.0 | 0 | 2,592 | 30.0 s |
| Sustained decode | 251K | 4 | Valid | 240.5 | 58.3 | 1,689 ms | 17.1 ms | 12.9 | 0 | 7,216 | 30.0 s |
| Sustained decode | 251K | 8 | Valid | 496.9 | 63.6 | 3,452 ms | 15.7 ms | 11.4 | 0 | 14,904 | 30.0 s |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.