Speed test matrix
GLM-5.3-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
sglang · 0.5.6.post3.dev11412+g964c45cf3 · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11412+g964c45cf3
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
964c45cf31e8
Run
- Served model
- GLM-5.3-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-incoai-sglang-964c45cf3/bin/sglang" serve --model-path incoai/GLM-5.3-NVFP4 --served-model-name GLM-5.3-NVFP4 --tp 8 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 131072 --mem-fraction-static 0.76 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --chunked-prefill-size 8192 --kv-cache-dtype fp8_e4m3 --dsa-prefill-backend flashinfer_sparse_mla --dsa-decode-backend flashinfer_sparse_mla --attention-backend dsa --fp4-gemm-backend flashinfer_cutlass --moe-runner-backend flashinfer_cutlass --speculative-algorithm DFLASH --speculative-draft-model-path incoai/GLM-5.3-DFlash2 --speculative-draft-attention-backend trtllm_mha --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context4,2751,918 ms TTFT
16K context4,2823,790 ms TTFT
32K context4,2577,592 ms TTFT
64K context4,17715,445 ms TTFT
123K context3,99031,041 ms TTFT
128K context3,96832,476 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 92.569 ms TTFT | 148.2295 ms TTFT | 205.3557 ms TTFT | 280.5165 ms TTFT | 501.81,062 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 92.4114 ms TTFT | 151.9171 ms TTFT | 217.4281 ms TTFT | 290.7317 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 96.0196 ms TTFT | 135.6225 ms TTFT | 215.2347 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 97.2222 ms TTFT | 147.5283 ms TTFT | 207.8491 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 92.0283 ms TTFT | 144.5487 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,275.0 | — | 1,918 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,282.0 | — | 3,790 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,257.0 | — | 7,592 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,177.0 | — | 15,445 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 3,990.0 | — | 31,041 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,968.0 | — | 32,476 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 92.5 | 96.3 | 69 ms | 10.4 ms | 1.0 | 0 | 2,779 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 148.2 | 75.6 | 295 ms | 13.3 ms | 2.0 | 0 | 4,440 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 205.3 | 53.8 | 557 ms | 18.6 ms | 4.0 | 0 | 6,153 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 280.5 | 36.8 | 165 ms | 27.2 ms | 8.0 | 0 | 8,404 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 501.8 | 32.3 | 1,062 ms | 31.0 ms | 16.0 | 0 | 15,037 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 92.4 | 95.7 | 114 ms | 10.4 ms | 1.0 | 0 | 2,773 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 151.9 | 78.6 | 171 ms | 12.7 ms | 2.0 | 0 | 4,562 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 217.4 | 55.5 | 281 ms | 18.0 ms | 4.0 | 0 | 6,524 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 290.7 | 38.4 | 317 ms | 26.1 ms | 8.0 | 0 | 8,721 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 96.0 | 97.8 | 196 ms | 10.2 ms | 1.0 | 0 | 2,877 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 135.6 | 71.0 | 225 ms | 14.1 ms | 2.0 | 0 | 4,062 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 215.2 | 54.8 | 347 ms | 18.2 ms | 4.0 | 0 | 6,452 | 30.0 s |
| Sustained decode | 32K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 97.2 | 99.1 | 222 ms | 10.1 ms | 1.0 | 0 | 2,914 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 147.5 | 76.2 | 283 ms | 13.1 ms | 2.0 | 0 | 4,420 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 207.8 | 55.1 | 491 ms | 18.2 ms | 4.0 | 0 | 6,234 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 92.0 | 94.1 | 283 ms | 10.6 ms | 1.0 | 0 | 2,760 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 144.5 | 75.0 | 487 ms | 13.3 ms | 2.0 | 0 | 4,333 | 30.0 s |
| Sustained decode | 123K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.