Speed test matrix
Ling-3.0-flash-fp8 on H200 NVL 141GB
sglang · 0.5.6.post3.dev8719+ge1a24a189 · FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev8719+ge1a24a189
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
e1a24a189fe4
Run
- Served model
- ling
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-inclusionai-sglang/bin/sglang" serve --model-path inclusionAI/Ling-3.0-flash-fp8 --served-model-name ling --trust-remote-code --tp 1 --reasoning-parser ling3 --tool-call-parser ling3 --context-length 262144 --mem-fraction-static 0.881145 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context27.4K300 ms TTFT
16K context33K491 ms TTFT
32K context32.6K991 ms TTFT
64K context29.6K2,173 ms TTFT
128K context24.1K5,328 ms TTFT
251K context17.2K14,659 ms TTFT
256K context17.1K15,009 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 205.285 ms TTFT | 328.2131 ms TTFT | 514.8134 ms TTFT | 816.3Requests queued158 ms TTFT | 800.4Requests queued35,721 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 203.9101 ms TTFT | 354.0115 ms TTFT | 507.2203 ms TTFT | 804.4Requests queued244 ms TTFT | 797.7Requests queued35,797 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 201.8134 ms TTFT | 349.5135 ms TTFT | 490.4240 ms TTFT | 784.7Requests queued413 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 199.1191 ms TTFT | 282.0242 ms TTFT | 453.9458 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 192.8290 ms TTFT | 327.5433 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 181.6559 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 27.4K | — | 300 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 33K | — | 491 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 32.6K | — | 991 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 29.6K | — | 2,173 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 24.1K | — | 5,328 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 17.2K | — | 14,659 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.1K | — | 15,009 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 205.2 | 205.9 | 85 ms | 4.9 ms | 1.0 | 1 | 6,157 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 328.2 | 167.2 | 131 ms | 6.0 ms | 2.3 | 2 | 9,846 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 514.8 | 136.7 | 134 ms | 7.4 ms | 5.0 | 4 | 15,440 | 30.0 s |
| Sustained decode | 8K | 8 | Requests queued | 816.3 | 117.4 | 158 ms | 8.5 ms | 7.0 | 10 | 24,484 | 30.0 s |
| Sustained decode | 8K | 16 | Requests queued | 800.4 | 115.6 | 35,721 ms | 8.7 ms | 7.0 | 13 | 24,011 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 203.9 | 204.7 | 101 ms | 4.9 ms | 1.0 | 1 | 6,115 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 354.0 | 174.0 | 115 ms | 5.7 ms | 2.0 | 2 | 10,618 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 507.2 | 133.0 | 203 ms | 7.5 ms | 4.8 | 4 | 15,212 | 30.0 s |
| Sustained decode | 16K | 8 | Requests queued | 804.4 | 115.9 | 244 ms | 8.6 ms | 6.9 | 10 | 24,132 | 30.0 s |
| Sustained decode | 16K | 16 | Requests queued | 797.7 | 115.3 | 35,797 ms | 8.7 ms | 6.9 | 13 | 23,926 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 201.8 | 202.9 | 134 ms | 4.9 ms | 1.0 | 1 | 6,055 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 349.5 | 172.8 | 135 ms | 5.8 ms | 2.0 | 2 | 10,485 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 490.4 | 128.3 | 240 ms | 7.8 ms | 4.8 | 4 | 14,712 | 30.0 s |
| Sustained decode | 32K | 8 | Requests queued | 784.7 | 111.6 | 413 ms | 9.0 ms | 6.9 | 10 | 23,540 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 199.1 | 200.4 | 191 ms | 5.0 ms | 1.0 | 1 | 5,971 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 282.0 | 155.6 | 242 ms | 6.5 ms | 2.7 | 2 | 8,461 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 453.9 | 112.9 | 458 ms | 8.9 ms | 5.6 | 0 | 13,616 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 192.8 | 194.7 | 290 ms | 5.1 ms | 1.0 | 1 | 5,784 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 327.5 | 166.3 | 433 ms | 6.0 ms | 1.9 | 2 | 9,826 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 181.6 | 185.2 | 559 ms | 5.4 ms | 1.0 | 1 | 5,448 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.