Speed test matrix
GLM-5.3-NVFP4 on H200 SXM 141GB ×8
sglang · 0.5.6.post3.dev9606+g26fd7fdaa · NVFP4+FP87 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9606+g26fd7fdaa
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
26fd7fdaa273
Run
- Served model
- nvidia/GLM-5.3-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-glm53-sglang-26fd7fd/bin/sglang" serve --model-path nvidia/GLM-5.3-NVFP4 --served-model-name nvidia/GLM-5.3-NVFP4 --trust-remote-code --tp 8 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 524288 --mem-fraction-static 0.81 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --dsa-prefill-backend flashmla_kv --dsa-decode-backend flashmla_kv --disable-shared-experts-fusion --attention-backend dsa --moe-runner-backend marlin --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context7,9161,036 ms TTFT
16K context8,0672,011 ms TTFT
32K context8,0334,023 ms TTFT
64K context7,7738,297 ms TTFT
128K context7,19317,913 ms TTFT
256K context6,23041,342 ms TTFT
507K context4,984102,322 ms TTFT
512K context4,978103,448 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 104.775 ms TTFT | 188.3176 ms TTFT | 335.0469 ms TTFT | 555.1544 ms TTFT | 888.6301 ms TTFT | 1,355.9599 ms TTFT | 1,999.71,142 ms TTFT | Over capacity |
| 16K | 105.1100 ms TTFT | 188.9139 ms TTFT | 334.8209 ms TTFT | 561.6327 ms TTFT | 897.4678 ms TTFT | 1,358.91,290 ms TTFT | Over capacity | Over capacity |
| 32K | 104.4131 ms TTFT | 187.4186 ms TTFT | 334.0294 ms TTFT | 557.4628 ms TTFT | 886.41,207 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 103.9201 ms TTFT | 186.8316 ms TTFT | 331.2639 ms TTFT | 554.91,250 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 103.3329 ms TTFT | 184.9618 ms TTFT | 328.11,136 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 101.7740 ms TTFT | 180.81,492 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 98.81,625 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 7,916.0 | — | 1,036 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,067.0 | — | 2,011 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,033.0 | — | 4,023 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 7,773.0 | — | 8,297 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,193.0 | — | 17,913 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,230.0 | — | 41,342 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 4,984.0 | — | 102,322 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 4,978.0 | — | 103,448 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 104.7 | 104.7 | 75 ms | 9.6 ms | 1.0 | 0 | 3,140 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 188.3 | 94.0 | 176 ms | 10.6 ms | 2.0 | 0 | 5,648 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 335.0 | 83.5 | 469 ms | 12.0 ms | 4.0 | 0 | 10,048 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 555.1 | 69.3 | 544 ms | 14.4 ms | 8.0 | 0 | 16,648 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 888.6 | 55.5 | 301 ms | 18.0 ms | 16.0 | 0 | 26,656 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,355.9 | 42.4 | 599 ms | 23.6 ms | 32.0 | 0 | 40,672 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,999.7 | 31.3 | 1,142 ms | 31.9 ms | 64.0 | 0 | 59,968 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 105.1 | 105.1 | 100 ms | 9.5 ms | 1.0 | 0 | 3,151 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 188.9 | 94.5 | 139 ms | 10.6 ms | 2.0 | 0 | 5,668 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 334.8 | 83.6 | 209 ms | 12.0 ms | 4.0 | 0 | 10,044 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 561.6 | 70.1 | 327 ms | 14.3 ms | 8.0 | 0 | 16,848 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 897.4 | 56.1 | 678 ms | 17.8 ms | 16.0 | 0 | 26,912 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,358.9 | 42.7 | 1,290 ms | 23.4 ms | 32.0 | 0 | 40,768 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 104.4 | 104.4 | 131 ms | 9.6 ms | 1.0 | 0 | 3,132 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 187.4 | 93.7 | 186 ms | 10.7 ms | 2.0 | 0 | 5,622 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 334.0 | 83.4 | 294 ms | 12.0 ms | 4.0 | 0 | 10,020 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 557.4 | 69.9 | 628 ms | 14.3 ms | 8.0 | 0 | 16,723 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 886.4 | 55.7 | 1,207 ms | 18.0 ms | 16.0 | 0 | 26,592 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 103.9 | 104.1 | 201 ms | 9.6 ms | 1.0 | 0 | 3,118 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 186.8 | 93.5 | 316 ms | 10.7 ms | 2.0 | 0 | 5,604 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 331.2 | 83.2 | 639 ms | 12.0 ms | 4.0 | 0 | 9,936 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 554.9 | 70.1 | 1,250 ms | 14.3 ms | 8.0 | 0 | 16,640 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 103.3 | 103.3 | 329 ms | 9.7 ms | 1.0 | 0 | 3,099 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 184.9 | 92.8 | 618 ms | 10.8 ms | 2.0 | 0 | 5,546 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 328.1 | 82.8 | 1,136 ms | 12.1 ms | 4.0 | 0 | 9,840 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 101.7 | 101.7 | 740 ms | 9.8 ms | 1.0 | 0 | 3,052 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 180.8 | 91.3 | 1,492 ms | 11.0 ms | 2.0 | 0 | 5,422 | 30.0 s |
| Sustained decode | 256K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 98.8 | 98.8 | 1,625 ms | 10.1 ms | 1.0 | 0 | 2,963 | 30.0 s |
| Sustained decode | 507K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.