Speed test matrix
Ling-2.6-flash-fp8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev8719+ge1a24a189 · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-2.6-flash-fp8 --served-model-name ling --trust-remote-code --tp 1 --tool-call-parser qwen --context-length 131072 --mem-fraction-static 0.880815 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context25.7K319 ms TTFT
16K context29.4K551 ms TTFT
32K context30.8K1,049 ms TTFT
64K context29.3K2,200 ms TTFT
123K context26.1K4,728 ms TTFT
128K context26.1K4,927 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 185.373 ms TTFT | 250.4116 ms TTFT | 365.1150 ms TTFT | 693.7162 ms TTFT | 965.8239 ms TTFT | 1,615.3398 ms TTFT | 1,696.9Requests queued9,061 ms TTFT | 839.0Requests queued78,525 ms TTFT |
| 16K | 184.695 ms TTFT | 95.5149 ms TTFT | 297.6170 ms TTFT | 633.9234 ms TTFT | 886.7415 ms TTFT | 1,456.1743 ms TTFT | 1,597.5Requests queued9,599 ms TTFT | 601.7Requests queued46,243 ms TTFT |
| 32K | 183.7117 ms TTFT | 90.4202 ms TTFT | 304.7245 ms TTFT | 608.3403 ms TTFT | 863.5767 ms TTFT | 1,423.41,561 ms TTFT | 1,496.2Requests queued10,991 ms TTFT | Over capacity |
| 64K | 181.9171 ms TTFT | 82.0286 ms TTFT | 174.6439 ms TTFT | 490.7818 ms TTFT | 764.81,649 ms TTFT | 1,314.43,649 ms TTFT | Over capacity | Over capacity |
| 123K | 179.0263 ms TTFT | 121.3555 ms TTFT | 332.7790 ms TTFT | 568.61,537 ms TTFT | 694.63,036 ms TTFT | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 25.7K | — | 319 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 29.4K | — | 551 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 30.8K | — | 1,049 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.3K | — | 2,200 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 26.1K | — | 4,728 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 26.1K | — | 4,927 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 185.3 | 185.8 | 73 ms | 5.4 ms | 1.0 | 1 | 5,559 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 250.4 | 131.2 | 116 ms | 7.7 ms | 2.4 | 2 | 7,512 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 365.1 | 91.4 | 150 ms | 10.9 ms | 6.0 | 0 | 10,952 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 693.7 | 84.4 | 162 ms | 11.8 ms | 8.8 | 0 | 20,808 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 965.8 | 59.8 | 239 ms | 16.7 ms | 19.7 | 0 | 28,976 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,615.3 | 48.0 | 398 ms | 20.9 ms | 32.0 | 0 | 48,448 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,696.9 | 44.8 | 9,061 ms | 22.3 ms | 39.0 | 0 | 50,910 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 839.0 | 46.5 | 78,525 ms | 21.5 ms | 38.4 | 0 | 25,159 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 184.6 | 185.2 | 95 ms | 5.4 ms | 1.0 | 1 | 5,539 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 95.5 | 47.7 | 149 ms | 21.0 ms | 32.0 | 0 | 2,866 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 297.6 | 70.6 | 170 ms | 14.2 ms | 11.7 | 0 | 8,928 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 633.9 | 78.1 | 234 ms | 12.8 ms | 10.1 | 0 | 19,016 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 886.7 | 55.4 | 415 ms | 18.1 ms | 21.1 | 0 | 26,592 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,456.1 | 43.6 | 743 ms | 22.9 ms | 32.0 | 0 | 43,680 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,597.5 | 41.0 | 9,599 ms | 24.4 ms | 39.0 | 0 | 47,931 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 601.7 | 41.3 | 46,243 ms | 24.2 ms | 39.0 | 0 | 18,053 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 183.7 | 184.5 | 117 ms | 5.4 ms | 1.0 | 1 | 5,461 | 29.7 s |
| Sustained decode | 32K | 2 | Valid | 90.4 | 45.2 | 202 ms | 22.1 ms | 32.0 | 0 | 2,712 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 304.7 | 72.1 | 245 ms | 13.9 ms | 10.2 | 0 | 9,140 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 608.3 | 75.3 | 403 ms | 13.3 ms | 10.1 | 0 | 18,249 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 863.5 | 54.0 | 767 ms | 18.5 ms | 21.9 | 0 | 25,904 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,423.4 | 42.4 | 1,561 ms | 23.6 ms | 32.0 | 0 | 42,689 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,496.2 | 39.7 | 10,991 ms | 25.2 ms | 38.9 | 0 | 44,877 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 181.9 | 183.0 | 171 ms | 5.5 ms | 1.0 | 1 | 5,456 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 82.0 | 41.1 | 286 ms | 24.3 ms | 36.0 | 0 | 2,460 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 174.6 | 43.4 | 439 ms | 23.0 ms | 31.0 | 0 | 5,236 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 490.7 | 59.3 | 818 ms | 16.9 ms | 16.1 | 0 | 14,721 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 764.8 | 48.1 | 1,649 ms | 20.8 ms | 24.1 | 0 | 22,944 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,314.4 | 38.4 | 3,649 ms | 26.1 ms | 32.0 | 0 | 39,424 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 179.0 | 180.6 | 263 ms | 5.5 ms | 1.0 | 1 | 5,370 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 121.3 | 57.6 | 555 ms | 17.3 ms | 14.2 | 0 | 3,640 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 332.7 | 80.2 | 790 ms | 12.5 ms | 6.6 | 0 | 9,980 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 568.6 | 71.4 | 1,537 ms | 14.0 ms | 9.7 | 0 | 17,056 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 694.6 | 45.0 | 3,036 ms | 22.2 ms | 23.6 | 0 | 20,832 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.