Speed test matrix
Ling-3.0-flash-int4 on H200 NVL 141GB
sglang · 0.5.6.post3.dev8719+ge1a24a189 · INT46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-flash-int4 --served-model-name ling --trust-remote-code --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 262144 --mem-fraction-static 0.862250 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-algorithm DSPARK --speculative-draft-model-path inclusionAI/Ling-3.0-flash-dspark --enable-linear-replayssm-spec --linear-replayssm-cache-len 32
Prefill
prompt tok/s by context length8K context19.6K419 ms TTFT
16K context22.2K732 ms TTFT
32K context21.8K1,483 ms TTFT
64K context20.3K3,167 ms TTFT
128K context17.4K7,372 ms TTFT
251K context13.6K18,473 ms TTFT
256K context13.5K18,970 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 301.375 ms TTFT | 450.8109 ms TTFT | 783.8118 ms TTFT | 1,103.7129 ms TTFT | 1,252.3221 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 291.396 ms TTFT | 404.0149 ms TTFT | 762.9134 ms TTFT | 920.8204 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 291.3128 ms TTFT | 340.6160 ms TTFT | 637.6199 ms TTFT | 741.7415 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 262.2183 ms TTFT | 344.3237 ms TTFT | 511.4441 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 225.8296 ms TTFT | 201.6511 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 172.5567 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.6K | — | 419 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 22.2K | — | 732 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 21.8K | — | 1,483 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.3K | — | 3,167 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.4K | — | 7,372 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 13.6K | — | 18,473 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 13.5K | — | 18,970 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 301.3 | 297.5 | 75 ms | 3.4 ms | 1.0 | 2 | 9,038 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 450.8 | 234.0 | 109 ms | 4.3 ms | 2.3 | 2 | 13,522 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 783.8 | 200.4 | 118 ms | 5.0 ms | 4.1 | 4 | 23,507 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 1,103.7 | 150.4 | 129 ms | 6.6 ms | 8.7 | 7 | 33,099 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,252.3 | 79.8 | 221 ms | 12.5 ms | 22.5 | 0 | 37,539 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 291.3 | 334.7 | 96 ms | 3.0 ms | 1.0 | 2 | 8,736 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 404.0 | 172.3 | 149 ms | 5.8 ms | 6.6 | 2 | 12,118 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 762.9 | 186.7 | 134 ms | 5.4 ms | 4.1 | 4 | 22,885 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 920.8 | 125.0 | 204 ms | 8.0 ms | 9.3 | 4 | 27,611 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 291.3 | 309.9 | 128 ms | 3.2 ms | 1.0 | 2 | 8,738 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 340.6 | 196.2 | 160 ms | 5.3 ms | 5.2 | 2 | 10,215 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 637.6 | 167.5 | 199 ms | 6.0 ms | 4.7 | 4 | 19,126 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 741.7 | 95.0 | 415 ms | 10.5 ms | 11.1 | 0 | 22,235 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 262.2 | 268.2 | 183 ms | 3.7 ms | 0.9 | 2 | 7,864 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 344.3 | 175.6 | 237 ms | 5.7 ms | 3.0 | 2 | 10,326 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 511.4 | 138.7 | 441 ms | 7.2 ms | 4.9 | 3 | 15,338 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 225.8 | 234.5 | 296 ms | 4.3 ms | 1.0 | 1 | 6,771 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 201.6 | 105.1 | 511 ms | 9.5 ms | 4.7 | 0 | 6,046 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 172.5 | 181.7 | 567 ms | 5.5 ms | 1.0 | 1 | 5,175 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.