Speed test matrix
Ling-2.6-flash-int4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev8719+ge1a24a189 · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-2.6-flash-int4 --served-model-name ling --trust-remote-code --tp 1 --tool-call-parser qwen --context-length 131072 --mem-fraction-static 0.877194 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context16.8K489 ms TTFT
16K context18.1K894 ms TTFT
32K context18.2K1,770 ms TTFT
64K context17.8K3,611 ms TTFT
123K context16.4K7,520 ms TTFT
128K context16.5K7,796 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 181.569 ms TTFT | 281.199 ms TTFT | 401.7123 ms TTFT | 833.2157 ms TTFT | 1,256.0221 ms TTFT | 1,704.6393 ms TTFT | 2,202.7787 ms TTFT | 1,633.7Requests queued85,715 ms TTFT |
| 16K | 181.281 ms TTFT | 69.6144 ms TTFT | 152.8650 ms TTFT | 630.5248 ms TTFT | 1,136.0395 ms TTFT | 1,595.7805 ms TTFT | 2,117.71,504 ms TTFT | 1,238.3Requests queued32,657 ms TTFT |
| 32K | 180.3114 ms TTFT | 64.6184 ms TTFT | 146.1227 ms TTFT | 541.3420 ms TTFT | 959.6856 ms TTFT | 1,391.21,464 ms TTFT | 1,922.43,005 ms TTFT | 1,112.6Requests queued41,135 ms TTFT |
| 64K | 178.6173 ms TTFT | 57.4291 ms TTFT | 131.9460 ms TTFT | 409.8897 ms TTFT | 827.21,608 ms TTFT | 1,214.93,271 ms TTFT | 1,633.56,328 ms TTFT | Over capacity |
| 123K | 175.5273 ms TTFT | 50.2455 ms TTFT | 113.1820 ms TTFT | 302.21,532 ms TTFT | 665.83,129 ms TTFT | 1,000.45,970 ms TTFT | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.8K | — | 489 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18.1K | — | 894 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 18.2K | — | 1,770 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.8K | — | 3,611 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 16.4K | — | 7,520 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.5K | — | 7,796 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 181.5 | 182.0 | 69 ms | 5.5 ms | 1.0 | 1 | 5,445 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 281.1 | 143.5 | 99 ms | 7.0 ms | 2.4 | 2 | 8,430 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 401.7 | 100.2 | 123 ms | 10.0 ms | 5.8 | 0 | 12,052 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 833.2 | 102.2 | 157 ms | 9.8 ms | 8.0 | 0 | 24,992 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,256.0 | 76.9 | 221 ms | 13.0 ms | 16.3 | 0 | 37,680 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,704.6 | 53.1 | 393 ms | 18.8 ms | 43.2 | 0 | 51,136 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,202.7 | 34.5 | 787 ms | 29.0 ms | 96.0 | 0 | 66,036 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,633.7 | 34.5 | 85,715 ms | 29.0 ms | 100.9 | 0 | 49,003 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 181.2 | 181.7 | 81 ms | 5.5 ms | 1.0 | 1 | 5,436 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 69.6 | 34.6 | 144 ms | 28.9 ms | 90.5 | 0 | 2,086 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 152.8 | 38.2 | 650 ms | 26.2 ms | 70.0 | 0 | 4,580 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 630.5 | 74.1 | 248 ms | 13.5 ms | 13.1 | 0 | 18,912 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,136.0 | 70.5 | 395 ms | 14.2 ms | 20.3 | 0 | 34,080 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,595.7 | 49.9 | 805 ms | 20.0 ms | 45.9 | 0 | 47,843 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,117.7 | 33.1 | 1,504 ms | 30.2 ms | 96.0 | 0 | 63,488 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,238.3 | 31.8 | 32,657 ms | 31.5 ms | 103.0 | 0 | 37,128 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 180.3 | 181.0 | 114 ms | 5.5 ms | 1.0 | 1 | 5,408 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 64.6 | 32.3 | 184 ms | 31.0 ms | 96.9 | 0 | 1,938 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 146.1 | 36.5 | 227 ms | 27.4 ms | 70.0 | 0 | 4,380 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 541.3 | 63.7 | 420 ms | 15.7 ms | 26.3 | 0 | 16,240 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 959.6 | 59.5 | 856 ms | 16.8 ms | 23.1 | 0 | 28,784 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,391.2 | 44.0 | 1,464 ms | 22.7 ms | 48.0 | 0 | 41,728 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,922.4 | 30.5 | 3,005 ms | 32.8 ms | 96.0 | 0 | 57,664 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 1,112.6 | 28.5 | 41,135 ms | 35.0 ms | 103.0 | 0 | 33,346 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 178.6 | 179.6 | 173 ms | 5.6 ms | 1.0 | 1 | 5,357 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 57.4 | 28.8 | 291 ms | 34.7 ms | 98.0 | 0 | 1,722 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 131.9 | 32.5 | 460 ms | 30.8 ms | 71.5 | 0 | 3,952 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 409.8 | 49.1 | 897 ms | 20.4 ms | 42.7 | 0 | 12,288 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 827.2 | 52.1 | 1,608 ms | 19.2 ms | 26.3 | 0 | 24,815 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,214.9 | 39.1 | 3,271 ms | 25.6 ms | 48.8 | 0 | 36,447 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 1,633.5 | 26.4 | 6,328 ms | 37.9 ms | 96.0 | 0 | 48,960 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 175.5 | 177.2 | 273 ms | 5.6 ms | 1.0 | 1 | 5,266 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 50.2 | 25.1 | 455 ms | 39.9 ms | 98.0 | 0 | 1,504 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 113.1 | 28.0 | 820 ms | 35.7 ms | 71.4 | 0 | 3,392 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 302.2 | 37.1 | 1,532 ms | 27.0 ms | 52.7 | 0 | 9,064 | 30.0 s |
| Sustained decode | 123K | 16 | Valid | 665.8 | 43.2 | 3,129 ms | 23.1 ms | 28.7 | 0 | 19,968 | 30.0 s |
| Sustained decode | 123K | 32 | Valid | 1,000.4 | 33.7 | 5,970 ms | 29.7 ms | 54.4 | 0 | 29,984 | 30.0 s |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.