Speed test matrix
GLM-5.3-Flash-NVFP4 on H200 NVL 141GB ×2
sglang · 0.0.0.dev0 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
4a0c6f2fe049
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-glm53flash-dflash2-sglang-4a0c6f2fe/bin/sglang" serve --model-path nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --tp 2 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 1048576 --mem-fraction-static 0.89 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --dsa-prefill-backend tilelang --dsa-decode-backend tilelang --moe-runner-backend marlin --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context10.8K758 ms TTFT
16K context11.3K1,433 ms TTFT
32K context11.4K2,832 ms TTFT
64K context11.4K5,658 ms TTFT
128K context11.3K11,435 ms TTFT
256K context10.9K23,557 ms TTFT
512K context10.3K49,873 ms TTFT
1019K context9,214111,237 ms TTFT
1024K context9,227111,625 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 108.524 ms TTFT | 195.556 ms TTFT | 341.451 ms TTFT | 558.260 ms TTFT | 857.183 ms TTFT | 1,324.5243 ms TTFT | 1,330.3Requests queued250 ms TTFT | Over capacity |
| 16K | 108.053 ms TTFT | 196.084 ms TTFT | 339.6121 ms TTFT | 551.3158 ms TTFT | 862.2235 ms TTFT | 1,323.2379 ms TTFT | Over capacity | Over capacity |
| 32K | 108.442 ms TTFT | 195.367 ms TTFT | 338.6102 ms TTFT | 553.1124 ms TTFT | 858.5182 ms TTFT | 1,319.4314 ms TTFT | Over capacity | Over capacity |
| 64K | 108.277 ms TTFT | 195.2122 ms TTFT | 343.8187 ms TTFT | 555.5292 ms TTFT | 863.1556 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 108.159 ms TTFT | 195.691 ms TTFT | 338.3137 ms TTFT | 556.1194 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 108.260 ms TTFT | 194.893 ms TTFT | 341.2147 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 107.953 ms TTFT | 193.375 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 107.0129 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.8K | — | 758 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11.3K | — | 1,433 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.4K | — | 2,832 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.4K | — | 5,658 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.3K | — | 11,435 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.9K | — | 23,557 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 10.3K | — | 49,873 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,214.0 | — | 111,237 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,227.0 | — | 111,625 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 108.5 | 108.5 | 24 ms | 9.2 ms | 1.0 | 0 | 3,254 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 195.5 | 97.8 | 56 ms | 10.2 ms | 2.0 | 0 | 5,866 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 341.4 | 85.6 | 51 ms | 11.7 ms | 4.0 | 0 | 10,243 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 558.2 | 69.9 | 60 ms | 14.3 ms | 8.0 | 0 | 16,744 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 857.1 | 53.8 | 83 ms | 18.6 ms | 16.0 | 0 | 25,712 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,324.5 | 41.7 | 243 ms | 24.0 ms | 32.0 | 0 | 39,712 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,330.3 | 40.3 | 250 ms | 24.8 ms | 33.0 | 0 | 39,897 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 108.0 | 108.0 | 53 ms | 9.3 ms | 1.0 | 0 | 3,241 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 196.0 | 97.9 | 84 ms | 10.2 ms | 2.0 | 0 | 5,878 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 339.6 | 85.1 | 121 ms | 11.8 ms | 4.0 | 0 | 10,188 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 551.3 | 69.2 | 158 ms | 14.5 ms | 8.0 | 0 | 16,544 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 862.2 | 54.1 | 235 ms | 18.5 ms | 16.0 | 0 | 25,855 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,323.2 | 41.7 | 379 ms | 24.0 ms | 32.0 | 0 | 39,696 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 108.4 | 108.4 | 42 ms | 9.2 ms | 1.0 | 0 | 3,251 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 195.3 | 97.6 | 67 ms | 10.2 ms | 2.0 | 0 | 5,858 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 338.6 | 84.7 | 102 ms | 11.8 ms | 4.0 | 0 | 10,156 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 553.1 | 69.4 | 124 ms | 14.4 ms | 8.0 | 0 | 16,592 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 858.5 | 53.8 | 182 ms | 18.6 ms | 16.0 | 0 | 25,744 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,319.4 | 41.5 | 314 ms | 24.1 ms | 32.0 | 0 | 39,552 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 108.2 | 108.1 | 77 ms | 9.2 ms | 1.0 | 0 | 3,245 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 195.2 | 97.6 | 122 ms | 10.2 ms | 2.0 | 0 | 5,856 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 343.8 | 85.9 | 187 ms | 11.6 ms | 4.0 | 0 | 10,312 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 555.5 | 69.5 | 292 ms | 14.4 ms | 8.0 | 0 | 16,656 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 863.1 | 54.1 | 556 ms | 18.5 ms | 16.0 | 0 | 25,888 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 108.1 | 108.0 | 59 ms | 9.3 ms | 1.0 | 0 | 3,241 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 195.6 | 97.8 | 91 ms | 10.2 ms | 2.0 | 0 | 5,867 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 338.3 | 84.9 | 137 ms | 11.8 ms | 4.0 | 0 | 10,148 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 556.1 | 69.5 | 194 ms | 14.4 ms | 8.0 | 0 | 16,680 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 108.2 | 108.2 | 60 ms | 9.2 ms | 1.0 | 0 | 3,245 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 194.8 | 97.4 | 93 ms | 10.3 ms | 2.0 | 0 | 5,842 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 341.2 | 85.1 | 147 ms | 11.7 ms | 4.0 | 0 | 10,236 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 107.9 | 107.9 | 53 ms | 9.3 ms | 1.0 | 0 | 3,237 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 193.3 | 96.7 | 75 ms | 10.3 ms | 2.0 | 0 | 5,798 | 30.0 s |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 107.0 | 107.0 | 129 ms | 9.3 ms | 1.0 | 0 | 3,210 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.