Speed test matrix
GLM-5.3-NVFP4 on H200 SXM 141GB ×8
sglang · 0.5.6.post3.dev10338+g787299797 · NVFP47 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev10338+g787299797
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
787299797931
Run
- Served model
- glm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-glm53flash-dflash2-sglang-pr-37818/bin/sglang" serve --model-path Inferact/GLM-5.3-NVFP4 --served-model-name glm --tp 8 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 524288 --mem-fraction-static 0.80 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --moe-runner-backend marlin
Prefill
prompt tok/s by context length8K context9,653849 ms TTFT
16K context9,6891,675 ms TTFT
32K context9,4713,412 ms TTFT
64K context9,1067,084 ms TTFT
128K context8,40115,341 ms TTFT
256K context7,17535,904 ms TTFT
507K context5,59291,217 ms TTFT
512K context5,56392,586 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 114.264 ms TTFT | 207.4103 ms TTFT | 366.3153 ms TTFT | 622.9183 ms TTFT | 992.0267 ms TTFT | 1,513.7560 ms TTFT | Over capacity | Over capacity |
| 16K | 114.882 ms TTFT | 208.1119 ms TTFT | 366.6189 ms TTFT | 621.3276 ms TTFT | 991.0618 ms TTFT | Over capacity | Over capacity | Over capacity |
| 32K | 114.0121 ms TTFT | 206.8196 ms TTFT | 361.8304 ms TTFT | 617.5601 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 113.5189 ms TTFT | 205.5339 ms TTFT | 360.7632 ms TTFT | 610.21,299 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 112.6322 ms TTFT | 203.4635 ms TTFT | 357.41,171 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 110.9587 ms TTFT | 199.31,163 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 107.41,325 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,653.0 | — | 849 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,689.0 | — | 1,675 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,471.0 | — | 3,412 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,106.0 | — | 7,084 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,401.0 | — | 15,341 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,175.0 | — | 35,904 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 5,592.0 | — | 91,217 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 5,563.0 | — | 92,586 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 114.2 | 114.2 | 64 ms | 8.8 ms | 1.0 | 0 | 3,426 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 207.4 | 103.8 | 103 ms | 9.6 ms | 2.0 | 0 | 6,222 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 366.3 | 91.6 | 153 ms | 10.9 ms | 4.0 | 0 | 10,988 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 622.9 | 78.0 | 183 ms | 12.8 ms | 8.0 | 0 | 18,680 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 992.0 | 62.2 | 267 ms | 16.1 ms | 16.0 | 0 | 29,760 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,513.7 | 47.6 | 560 ms | 21.0 ms | 32.0 | 0 | 45,408 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 114.8 | 114.8 | 82 ms | 8.7 ms | 1.0 | 0 | 3,445 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 208.1 | 104.0 | 119 ms | 9.6 ms | 2.0 | 0 | 6,244 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 366.6 | 91.6 | 189 ms | 10.9 ms | 4.0 | 0 | 10,996 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 621.3 | 77.7 | 276 ms | 12.9 ms | 8.0 | 0 | 18,640 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 991.0 | 62.2 | 618 ms | 16.1 ms | 16.0 | 0 | 29,728 | 30.0 s |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 114.0 | 114.0 | 121 ms | 8.8 ms | 1.0 | 0 | 3,420 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 206.8 | 103.4 | 196 ms | 9.7 ms | 2.0 | 0 | 6,204 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 361.8 | 90.5 | 304 ms | 11.0 ms | 4.0 | 0 | 10,852 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 617.5 | 77.4 | 601 ms | 12.9 ms | 8.0 | 0 | 18,520 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 113.5 | 113.5 | 189 ms | 8.8 ms | 1.0 | 0 | 3,406 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 205.5 | 102.9 | 339 ms | 9.7 ms | 2.0 | 0 | 6,166 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 360.7 | 90.7 | 632 ms | 11.0 ms | 4.0 | 0 | 10,820 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 610.2 | 77.3 | 1,299 ms | 12.9 ms | 8.0 | 0 | 18,304 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 112.6 | 112.6 | 322 ms | 8.9 ms | 1.0 | 0 | 3,378 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 203.4 | 102.1 | 635 ms | 9.8 ms | 2.0 | 0 | 6,102 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 357.4 | 90.2 | 1,171 ms | 11.1 ms | 4.0 | 0 | 10,720 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 110.9 | 110.8 | 587 ms | 9.0 ms | 1.0 | 0 | 3,325 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 199.3 | 100.4 | 1,163 ms | 10.0 ms | 2.0 | 0 | 5,978 | 30.0 s |
| Sustained decode | 256K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 107.4 | 107.4 | 1,325 ms | 9.3 ms | 1.0 | 0 | 3,222 | 30.0 s |
| Sustained decode | 507K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.