Speed test matrix
GLM-5.3-NVFP4 on H200 SXM 141GB ×8
sglang · 0.5.6.post3.dev10338+g787299797 · NVFP47 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev10338+g787299797
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
787299797931
Run
- Served model
- glm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-glm53flash-dflash2-sglang-pr-37818/bin/sglang" serve --model-path incoai/GLM-5.3-NVFP4 --served-model-name glm --tp 8 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 524288 --mem-fraction-static 0.85 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --moe-runner-backend marlin
Prefill
prompt tok/s by context length8K context9,670848 ms TTFT
16K context9,6961,674 ms TTFT
32K context9,4823,408 ms TTFT
64K context9,0947,093 ms TTFT
128K context8,30015,528 ms TTFT
256K context7,09836,294 ms TTFT
507K context5,46393,365 ms TTFT
512K context5,45394,457 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 113.965 ms TTFT | 207.7397 ms TTFT | 362.51,188 ms TTFT | 610.6183 ms TTFT | 986.911,964 ms TTFT | 1,463.2581 ms TTFT | Over capacity | Over capacity |
| 16K | 114.585 ms TTFT | 208.3120 ms TTFT | 362.5206 ms TTFT | 609.4273 ms TTFT | 974.4580 ms TTFT | 1,466.01,131 ms TTFT | Over capacity | Over capacity |
| 32K | 113.7121 ms TTFT | 206.8166 ms TTFT | 359.5285 ms TTFT | 609.0602 ms TTFT | 970.41,163 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 113.2190 ms TTFT | 205.8312 ms TTFT | 359.0612 ms TTFT | 594.71,184 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 112.3324 ms TTFT | 203.6600 ms TTFT | 355.21,159 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 110.6684 ms TTFT | 199.31,250 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 107.21,539 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,670.0 | — | 848 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 9,696.0 | — | 1,674 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,482.0 | — | 3,408 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,094.0 | — | 7,093 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,300.0 | — | 15,528 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,098.0 | — | 36,294 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 5,463.0 | — | 93,365 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 5,453.0 | — | 94,457 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 113.9 | 113.9 | 65 ms | 8.8 ms | 1.0 | 0 | 3,416 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 207.7 | 103.0 | 397 ms | 9.7 ms | 2.0 | 0 | 6,228 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 362.5 | 88.2 | 1,188 ms | 11.3 ms | 4.0 | 0 | 10,876 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 610.6 | 74.4 | 183 ms | 13.4 ms | 8.0 | 0 | 18,312 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 986.9 | 60.0 | 11,964 ms | 16.7 ms | 16.0 | 0 | 29,600 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,463.2 | 45.2 | 581 ms | 22.1 ms | 32.0 | 0 | 43,872 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 114.5 | 114.4 | 85 ms | 8.7 ms | 1.0 | 0 | 3,435 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 208.3 | 104.2 | 120 ms | 9.6 ms | 2.0 | 0 | 6,246 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 362.5 | 90.6 | 206 ms | 11.0 ms | 4.0 | 0 | 10,872 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 609.4 | 76.2 | 273 ms | 13.1 ms | 8.0 | 0 | 18,280 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 974.4 | 61.0 | 580 ms | 16.4 ms | 16.0 | 0 | 29,216 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,466.0 | 46.0 | 1,131 ms | 21.7 ms | 32.0 | 0 | 43,968 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 113.7 | 113.6 | 121 ms | 8.8 ms | 1.0 | 0 | 3,409 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 206.8 | 103.4 | 166 ms | 9.7 ms | 2.0 | 0 | 6,204 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 359.5 | 90.0 | 285 ms | 11.1 ms | 4.0 | 0 | 10,786 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 609.0 | 76.4 | 602 ms | 13.1 ms | 8.0 | 0 | 18,264 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 970.4 | 61.0 | 1,163 ms | 16.4 ms | 16.0 | 0 | 29,104 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 113.2 | 113.2 | 190 ms | 8.8 ms | 1.0 | 0 | 3,397 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 205.8 | 103.0 | 312 ms | 9.7 ms | 2.0 | 0 | 6,174 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 359.0 | 90.0 | 612 ms | 11.1 ms | 4.0 | 0 | 10,770 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 594.7 | 75.2 | 1,184 ms | 13.3 ms | 8.0 | 0 | 17,840 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 112.3 | 112.3 | 324 ms | 8.9 ms | 1.0 | 0 | 3,368 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 203.6 | 102.2 | 600 ms | 9.8 ms | 2.0 | 0 | 6,108 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 355.2 | 89.8 | 1,159 ms | 11.1 ms | 4.0 | 0 | 10,656 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 110.6 | 110.6 | 684 ms | 9.0 ms | 1.0 | 0 | 3,317 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 199.3 | 99.9 | 1,250 ms | 10.0 ms | 2.0 | 0 | 5,980 | 30.0 s |
| Sustained decode | 256K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 107.2 | 107.2 | 1,539 ms | 9.3 ms | 1.0 | 0 | 3,215 | 30.0 s |
| Sustained decode | 507K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.