Speed test matrix
GLM-5.3-Flash on H200 SXM 141GB ×4
sglang · 0.5.6.post3.dev10338+g787299797 · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev10338+g787299797
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
787299797931
Run
- Served model
- glm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-glm53flash-dflash2-sglang-pr-37818/bin/sglang" serve --model-path zai-org/GLM-5.3-Flash --served-model-name glm --trust-remote-code --tp 4 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 1048576 --mem-fraction-static 0.841182 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype bfloat16 --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --enable-multimodal --moe-runner-backend triton --speculative-algorithm DFLASH --speculative-draft-model-path incoai/GLM-5.3-Flash-DFlash2 --speculative-draft-attention-backend fa4 --speculative-num-draft-tokens 8
Prefill
prompt tok/s by context length8K context13.5K607 ms TTFT
16K context21.8K745 ms TTFT
32K context22.3K1,447 ms TTFT
64K context18.6K3,466 ms TTFT
128K context22.1K5,832 ms TTFT
256K context20.1K12,845 ms TTFT
512K context19.1K26,919 ms TTFT
1019K context15.8K64,878 ms TTFT
1024K context14.7K69,932 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 183.6192 ms TTFT | 330.5269 ms TTFT | 542.5286 ms TTFT | 884.91,138 ms TTFT | 1,459.4646 ms TTFT | 2,302.96,646 ms TTFT | Not run | Over capacity |
| 16K | 184.4232 ms TTFT | 330.9307 ms TTFT | 552.9309 ms TTFT | 916.0680 ms TTFT | 1,478.11,222 ms TTFT | 2,291.62,428 ms TTFT | Over capacity | Over capacity |
| 32K | 193.8306 ms TTFT | 341.1314 ms TTFT | 551.5458 ms TTFT | 927.21,157 ms TTFT | 1,483.62,471 ms TTFT | 2,293.04,912 ms TTFT | Over capacity | Over capacity |
| 64K | 196.8474 ms TTFT | 345.3605 ms TTFT | 560.7822 ms TTFT | 907.32,450 ms TTFT | 1,487.54,614 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 188.7733 ms TTFT | 331.3Requests queued1,217 ms TTFT | 516.51,836 ms TTFT | 908.44,838 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 190.71,317 ms TTFT | 302.5Requests queued1,930 ms TTFT | 529.4Requests queued3,679 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 184.22,677 ms TTFT | 288.9Requests queued5,096 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 140.46,283 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 13.5K | — | 607 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.8K | — | 745 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.3K | — | 1,447 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 18.6K | — | 3,466 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.1K | — | 5,832 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 20.1K | — | 12,845 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 19.1K | — | 26,919 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 15.8K | — | 64,878 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 14.7K | — | 69,932 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 183.6 | 188.8 | 192 ms | 5.3 ms | 1.0 | 1 | 5,506 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 330.5 | 177.5 | 269 ms | 5.6 ms | 2.0 | 2 | 9,913 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 542.5 | 144.6 | 286 ms | 6.9 ms | 4.0 | 4 | 16,272 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 884.9 | 115.8 | 1,138 ms | 8.6 ms | 8.0 | 3 | 26,538 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,459.4 | 88.7 | 646 ms | 11.3 ms | 16.0 | 0 | 43,776 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,302.9 | 72.2 | 6,646 ms | 13.8 ms | 32.0 | 0 | 69,021 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 184.4 | 190.7 | 232 ms | 5.2 ms | 1.0 | 1 | 5,531 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 330.9 | 174.9 | 307 ms | 5.7 ms | 2.0 | 2 | 9,924 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 552.9 | 144.2 | 309 ms | 6.9 ms | 4.0 | 4 | 16,584 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 916.0 | 116.0 | 680 ms | 8.6 ms | 8.0 | 2 | 27,470 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,478.1 | 95.3 | 1,222 ms | 10.5 ms | 16.0 | 0 | 44,322 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,291.6 | 73.0 | 2,428 ms | 13.7 ms | 32.0 | 0 | 68,675 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 193.8 | 197.0 | 306 ms | 5.1 ms | 1.0 | 1 | 5,811 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 341.1 | 176.7 | 314 ms | 5.7 ms | 1.9 | 2 | 10,231 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 551.5 | 143.1 | 458 ms | 7.0 ms | 4.0 | 4 | 16,536 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 927.2 | 120.9 | 1,157 ms | 8.3 ms | 8.0 | 2 | 27,807 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,483.6 | 95.7 | 2,471 ms | 10.4 ms | 16.0 | 0 | 44,505 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,293.0 | 73.8 | 4,912 ms | 13.6 ms | 32.0 | 0 | 68,731 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 196.8 | 202.6 | 474 ms | 4.9 ms | 1.0 | 1 | 5,899 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 345.3 | 179.4 | 605 ms | 5.6 ms | 1.9 | 2 | 10,358 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 560.7 | 155.5 | 822 ms | 6.4 ms | 4.0 | 4 | 16,817 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 907.3 | 116.4 | 2,450 ms | 8.6 ms | 8.0 | 2 | 27,010 | 29.8 s |
| Sustained decode | 64K | 16 | Valid | 1,487.5 | 98.3 | 4,614 ms | 10.2 ms | 16.0 | 0 | 44,614 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 188.7 | 197.6 | 733 ms | 5.1 ms | 1.0 | 1 | 5,660 | 30.0 s |
| Sustained decode | 128K | 2 | Requests queued | 331.3 | 175.8 | 1,217 ms | 5.7 ms | 1.9 | 2 | 9,937 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 516.5 | 149.5 | 1,836 ms | 6.7 ms | 3.8 | 4 | 15,490 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 908.4 | 129.3 | 4,838 ms | 7.7 ms | 7.9 | 5 | 27,247 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 190.7 | 201.0 | 1,317 ms | 5.0 ms | 0.9 | 1 | 5,719 | 30.0 s |
| Sustained decode | 256K | 2 | Requests queued | 302.5 | 164.9 | 1,930 ms | 6.1 ms | 1.9 | 2 | 9,074 | 30.0 s |
| Sustained decode | 256K | 4 | Requests queued | 529.4 | 159.2 | 3,679 ms | 6.3 ms | 3.8 | 4 | 15,877 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 184.2 | 205.0 | 2,677 ms | 4.9 ms | 1.0 | 1 | 5,525 | 30.0 s |
| Sustained decode | 512K | 2 | Requests queued | 288.9 | 176.9 | 5,096 ms | 5.7 ms | 1.9 | 2 | 8,663 | 30.0 s |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 140.4 | 187.7 | 6,283 ms | 5.4 ms | 0.9 | 1 | 4,212 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.