Speed test matrix
GLM-5.3-Flash-NVFP4 on H200 SXM 141GB ×4
sglang · 0.5.6.post3.dev10338+g787299797 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev10338+g787299797
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
787299797931
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-glm53flash-dflash2-sglang-pr-37818/bin/sglang" serve --model-path nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --tp 4 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 1048576 --mem-fraction-static 0.85 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --moe-runner-backend marlin --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context12K685 ms TTFT
16K context17.3K938 ms TTFT
32K context17.5K1,844 ms TTFT
64K context17.6K3,670 ms TTFT
128K context17.4K7,411 ms TTFT
256K context16.5K15,630 ms TTFT
512K context15.3K33,661 ms TTFT
1019K context13.1K78,529 ms TTFT
1024K context12.4K82,848 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 124.1184 ms TTFT | 234.2322 ms TTFT | 427.8342 ms TTFT | 751.8528 ms TTFT | 1,247.0616 ms TTFT | 2,015.31,012 ms TTFT | 3,159.52,016 ms TTFT | 4,782.93,973 ms TTFT |
| 16K | 123.9213 ms TTFT | 233.0345 ms TTFT | 428.2484 ms TTFT | 753.6623 ms TTFT | 1,243.11,038 ms TTFT | 2,007.72,039 ms TTFT | 3,181.54,041 ms TTFT | 4,767.87,871 ms TTFT |
| 32K | 123.4282 ms TTFT | 233.1418 ms TTFT | 427.5622 ms TTFT | 756.51,089 ms TTFT | 1,246.72,138 ms TTFT | 2,009.94,197 ms TTFT | 3,175.48,402 ms TTFT | Over capacity |
| 64K | 122.5422 ms TTFT | 233.1641 ms TTFT | 424.21,111 ms TTFT | 755.02,219 ms TTFT | 1,232.44,374 ms TTFT | 1,979.48,778 ms TTFT | Over capacity | Over capacity |
| 128K | 120.5697 ms TTFT | 230.11,194 ms TTFT | 424.12,240 ms TTFT | 743.34,346 ms TTFT | 1,223.28,885 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 121.01,290 ms TTFT | 226.92,310 ms TTFT | 411.94,600 ms TTFT | 733.69,062 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 117.42,865 ms TTFT | 220.25,282 ms TTFT | 401.010,132 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 111.16,894 ms TTFT | 172.9Requests queued11,787 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12K | — | 685 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 17.3K | — | 938 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 17.5K | — | 1,844 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 17.6K | — | 3,670 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.4K | — | 7,411 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 16.5K | — | 15,630 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 15.3K | — | 33,661 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 13.1K | — | 78,529 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 12.4K | — | 82,848 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 124.1 | 125.4 | 184 ms | 8.0 ms | 1.0 | 1 | 3,722 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 234.2 | 117.2 | 322 ms | 8.5 ms | 2.0 | 0 | 7,026 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 427.8 | 107.0 | 342 ms | 9.3 ms | 4.0 | 0 | 12,832 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 751.8 | 94.2 | 528 ms | 10.6 ms | 8.0 | 0 | 22,554 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,247.0 | 77.9 | 616 ms | 12.8 ms | 16.0 | 0 | 37,408 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,015.3 | 62.9 | 1,012 ms | 15.9 ms | 32.0 | 0 | 60,448 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,159.5 | 49.3 | 2,016 ms | 20.3 ms | 64.0 | 0 | 94,784 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,782.9 | 37.3 | 3,973 ms | 26.8 ms | 128.0 | 0 | 143,488 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 123.9 | 125.3 | 213 ms | 8.0 ms | 1.0 | 1 | 3,716 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 233.0 | 116.5 | 345 ms | 8.6 ms | 2.0 | 0 | 6,988 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 428.2 | 107.2 | 484 ms | 9.3 ms | 4.0 | 0 | 12,844 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 753.6 | 94.0 | 623 ms | 10.6 ms | 8.0 | 0 | 22,608 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,243.1 | 77.7 | 1,038 ms | 12.9 ms | 16.0 | 0 | 37,280 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,007.7 | 62.6 | 2,039 ms | 16.0 ms | 32.0 | 0 | 60,224 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,181.5 | 49.7 | 4,041 ms | 20.1 ms | 64.0 | 0 | 95,424 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,767.8 | 37.3 | 7,871 ms | 26.8 ms | 128.0 | 0 | 143,040 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 123.4 | 124.6 | 282 ms | 8.0 ms | 1.0 | 1 | 3,703 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 233.1 | 116.6 | 418 ms | 8.6 ms | 2.0 | 0 | 6,990 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 427.5 | 106.8 | 622 ms | 9.4 ms | 4.0 | 0 | 12,824 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 756.5 | 94.3 | 1,089 ms | 10.6 ms | 8.0 | 0 | 22,688 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,246.7 | 77.6 | 2,138 ms | 12.9 ms | 16.0 | 0 | 37,392 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,009.9 | 62.7 | 4,197 ms | 16.0 ms | 32.0 | 0 | 60,287 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,175.4 | 49.7 | 8,402 ms | 20.1 ms | 64.0 | 0 | 95,232 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 122.5 | 124.4 | 422 ms | 8.0 ms | 1.0 | 1 | 3,676 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 233.1 | 116.7 | 641 ms | 8.6 ms | 2.0 | 0 | 6,992 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 424.2 | 106.3 | 1,111 ms | 9.4 ms | 4.0 | 0 | 12,724 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 755.0 | 95.3 | 2,219 ms | 10.5 ms | 8.0 | 0 | 22,648 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,232.4 | 79.1 | 4,374 ms | 12.6 ms | 16.0 | 0 | 36,960 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,979.4 | 65.6 | 8,778 ms | 15.3 ms | 32.0 | 0 | 59,360 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 120.5 | 128.0 | 697 ms | 7.8 ms | 1.0 | 1 | 3,615 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 230.1 | 115.6 | 1,194 ms | 8.6 ms | 2.0 | 0 | 6,902 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 424.1 | 107.6 | 2,240 ms | 9.3 ms | 4.0 | 0 | 12,720 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 743.3 | 96.5 | 4,346 ms | 10.4 ms | 8.0 | 0 | 22,296 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,223.2 | 83.6 | 8,885 ms | 12.0 ms | 16.0 | 0 | 36,688 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 121.0 | 120.9 | 1,290 ms | 8.3 ms | 1.0 | 0 | 3,629 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 226.9 | 114.9 | 2,310 ms | 8.7 ms | 2.0 | 0 | 6,806 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 411.9 | 107.3 | 4,600 ms | 9.3 ms | 4.0 | 0 | 12,356 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 733.6 | 100.5 | 9,062 ms | 10.0 ms | 8.0 | 0 | 22,008 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 117.4 | 117.4 | 2,865 ms | 8.5 ms | 1.0 | 0 | 3,523 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 220.2 | 114.2 | 5,282 ms | 8.8 ms | 2.0 | 0 | 6,606 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 401.0 | 110.7 | 10,132 ms | 9.0 ms | 4.0 | 0 | 12,024 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 111.1 | 111.1 | 6,894 ms | 9.0 ms | 1.0 | 0 | 3,332 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 172.9 | 105.1 | 11,787 ms | 9.8 ms | 1.9 | 1 | 6,125 | 35.4 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.