Speed test matrix
GLM-5.3-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
sglang · 0.5.6.post3.dev11412+g964c45cf3 · NVFP46 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11412+g964c45cf3
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
964c45cf31e8
Run
- Served model
- GLM-5.3-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-incoai-sglang-964c45cf3/bin/sglang" serve --model-path incoai/GLM-5.3-NVFP4 --served-model-name GLM-5.3-NVFP4 --tp 8 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 262144 --mem-fraction-static 0.75 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --chunked-prefill-size 8192 --kv-cache-dtype fp8_e4m3 --dsa-prefill-backend flashinfer_sparse_mla --dsa-decode-backend flashinfer_sparse_mla --attention-backend dsa --fp4-gemm-backend flashinfer_cutlass --moe-runner-backend flashinfer_cutlass --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context4,2341,936 ms TTFT
16K context4,2373,830 ms TTFT
32K context4,1947,706 ms TTFT
64K context4,11415,682 ms TTFT
128K context3,94332,684 ms TTFT
251K context3,55471,076 ms TTFT
256K context3,43175,091 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 53.496 ms TTFT | 96.8191 ms TTFT | 175.7441 ms TTFT | 297.8748 ms TTFT | 466.6229 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 52.8112 ms TTFT | 96.3140 ms TTFT | 174.2202 ms TTFT | 290.8311 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 52.5153 ms TTFT | 96.0188 ms TTFT | 175.0306 ms TTFT | 294.9473 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 52.5196 ms TTFT | 96.1279 ms TTFT | 173.2445 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 52.2281 ms TTFT | 95.1504 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 50.7544 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,234.0 | — | 1,936 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,237.0 | — | 3,830 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,194.0 | — | 7,706 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,114.0 | — | 15,682 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,943.0 | — | 32,684 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,554.0 | — | 71,076 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,431.0 | — | 75,091 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 53.4 | 53.4 | 96 ms | 18.7 ms | 1.0 | 0 | 1,603 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 96.8 | 48.3 | 191 ms | 20.7 ms | 2.0 | 0 | 2,908 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 175.7 | 44.0 | 441 ms | 22.8 ms | 4.0 | 0 | 5,268 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 297.8 | 37.3 | 748 ms | 26.8 ms | 8.0 | 0 | 8,928 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 466.6 | 29.1 | 229 ms | 34.3 ms | 16.0 | 0 | 13,984 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 52.8 | 52.8 | 112 ms | 18.9 ms | 1.0 | 0 | 1,584 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 96.3 | 48.2 | 140 ms | 20.7 ms | 2.0 | 0 | 2,888 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 174.2 | 43.6 | 202 ms | 22.9 ms | 4.0 | 0 | 5,232 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 290.8 | 36.4 | 311 ms | 27.5 ms | 8.0 | 0 | 8,720 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 52.5 | 52.6 | 153 ms | 19.0 ms | 1.0 | 0 | 1,574 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 96.0 | 48.0 | 188 ms | 20.8 ms | 2.0 | 0 | 2,878 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 175.0 | 43.6 | 306 ms | 22.9 ms | 4.0 | 0 | 5,248 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 294.9 | 36.9 | 473 ms | 27.1 ms | 8.0 | 0 | 8,840 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 52.5 | 52.6 | 196 ms | 19.0 ms | 1.0 | 0 | 1,575 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 96.1 | 48.1 | 279 ms | 20.8 ms | 2.0 | 0 | 2,882 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 173.2 | 43.2 | 445 ms | 23.1 ms | 4.0 | 0 | 5,196 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 52.2 | 52.2 | 281 ms | 19.2 ms | 1.0 | 0 | 1,566 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 95.1 | 47.7 | 504 ms | 21.0 ms | 2.0 | 0 | 2,850 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 50.7 | 50.8 | 544 ms | 19.7 ms | 1.0 | 0 | 1,522 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.