Speed test matrix
GLM-5.3-Flash-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
sglang · 0.0.0.dev0 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.0.0.dev0
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
4df49edcf00b
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-glm53flash-dflash2-sglang-pr-38430/bin/sglang" serve --model-path nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --trust-remote-code --tp 4 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 1048576 --mem-fraction-static 0.83 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --attention-backend dsa --dsa-prefill-backend flashinfer_sparse_mla --dsa-decode-backend flashinfer_sparse_mla --fp4-gemm-backend flashinfer_cutlass --moe-runner-backend flashinfer_cutlass --speculative-algorithm DFLASH --speculative-draft-model-path incoai/GLM-5.3-Flash-DFlash2 --speculative-draft-attention-backend fa4 --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context11.4K718 ms TTFT
16K context12.7K1,281 ms TTFT
32K context12.8K2,530 ms TTFT
64K context12.8K5,057 ms TTFT
128K context12.4K10,360 ms TTFT
256K context9,11128,276 ms TTFT
512K context11.2K46,031 ms TTFT
1019K context9,479108,126 ms TTFT
1024K context9,455108,938 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 170.434 ms TTFT | 278.757 ms TTFT | 404.5467 ms TTFT | 613.5396 ms TTFT | 925.3596 ms TTFT | 1,335.0446 ms TTFT | Not run | Not run |
| 16K | 177.161 ms TTFT | 285.889 ms TTFT | 395.8118 ms TTFT | 614.5319 ms TTFT | 935.2216 ms TTFT | 1,333.3431 ms TTFT | Not run | Over capacity |
| 32K | 169.651 ms TTFT | 267.374 ms TTFT | 407.799 ms TTFT | 638.5127 ms TTFT | 968.3176 ms TTFT | 1,357.7281 ms TTFT | Over capacity | Over capacity |
| 64K | 179.274 ms TTFT | 262.4107 ms TTFT | 401.0175 ms TTFT | 620.0260 ms TTFT | 954.3446 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 177.765 ms TTFT | 271.796 ms TTFT | 398.0137 ms TTFT | 626.2196 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 174.665 ms TTFT | 287.890 ms TTFT | 389.8137 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 167.054 ms TTFT | 262.374 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 161.5122 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 11.4K | — | 718 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 12.7K | — | 1,281 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.8K | — | 2,530 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.8K | — | 5,057 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.4K | — | 10,360 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 9,111.0 | — | 28,276 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 11.2K | — | 46,031 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,479.0 | — | 108,126 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,455.0 | — | 108,938 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 170.4 | 179.9 | 34 ms | 5.6 ms | 1.0 | 1 | 5,110 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 278.7 | 143.7 | 57 ms | 7.0 ms | 2.0 | 2 | 8,360 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 404.5 | 104.7 | 467 ms | 9.6 ms | 4.0 | 0 | 12,128 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 613.5 | 78.8 | 396 ms | 12.7 ms | 8.0 | 0 | 18,403 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 925.3 | 59.1 | 596 ms | 16.9 ms | 16.0 | 0 | 27,720 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,335.0 | 43.2 | 446 ms | 23.1 ms | 32.0 | 0 | 40,028 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 177.1 | 181.0 | 61 ms | 5.5 ms | 1.0 | 1 | 5,310 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 285.8 | 148.9 | 89 ms | 6.7 ms | 2.0 | 2 | 8,569 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 395.8 | 101.6 | 118 ms | 9.8 ms | 4.0 | 0 | 11,874 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 614.5 | 77.9 | 319 ms | 12.8 ms | 8.0 | 0 | 18,430 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 935.2 | 61.0 | 216 ms | 16.4 ms | 16.0 | 0 | 28,041 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,333.3 | 42.8 | 431 ms | 23.4 ms | 32.0 | 0 | 39,967 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 169.6 | 175.6 | 51 ms | 5.7 ms | 1.0 | 1 | 5,085 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 267.3 | 146.8 | 74 ms | 6.8 ms | 2.0 | 2 | 8,018 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 407.7 | 104.9 | 99 ms | 9.5 ms | 4.0 | 0 | 12,231 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 638.5 | 82.7 | 127 ms | 12.1 ms | 8.0 | 0 | 19,141 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 968.3 | 62.8 | 176 ms | 15.9 ms | 16.0 | 0 | 29,013 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,357.7 | 43.8 | 281 ms | 22.8 ms | 32.0 | 0 | 40,730 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 179.2 | 182.7 | 74 ms | 5.5 ms | 1.0 | 1 | 5,375 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 262.4 | 146.5 | 107 ms | 6.9 ms | 1.9 | 2 | 7,869 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 401.0 | 104.0 | 175 ms | 9.6 ms | 4.0 | 0 | 12,029 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 620.0 | 78.6 | 260 ms | 12.7 ms | 8.0 | 0 | 18,592 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 954.3 | 60.7 | 446 ms | 16.5 ms | 16.0 | 0 | 28,622 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 177.7 | 184.2 | 65 ms | 5.4 ms | 1.0 | 1 | 5,330 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 271.7 | 149.5 | 96 ms | 6.7 ms | 2.0 | 2 | 8,151 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 398.0 | 102.5 | 137 ms | 9.8 ms | 4.0 | 0 | 11,938 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 626.2 | 78.9 | 196 ms | 12.7 ms | 8.0 | 0 | 18,769 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 174.6 | 176.0 | 65 ms | 5.7 ms | 1.0 | 1 | 5,238 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 287.8 | 151.4 | 90 ms | 6.6 ms | 2.0 | 2 | 8,631 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 389.8 | 98.2 | 137 ms | 10.2 ms | 4.0 | 0 | 11,695 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 167.0 | 178.0 | 54 ms | 5.7 ms | 1.0 | 1 | 5,009 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 262.3 | 142.0 | 74 ms | 7.1 ms | 2.0 | 2 | 7,870 | 30.0 s |
| Sustained decode | 512K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 161.5 | 166.1 | 122 ms | 6.0 ms | 1.0 | 1 | 4,845 | 30.0 s |
| Sustained decode | 1019K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.