Speed test matrix
GLM-5.3-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8
sglang · 0.5.6.post3.dev9606+g26fd7fdaa · NVFP4+FP86 contexts (8K–251K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 251K, 256K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9606+g26fd7fdaa
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
26fd7fdaa273
Run
- Served model
- nvidia/GLM-5.3-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-glm53-sglang-26fd7fd/bin/sglang" serve --model-path nvidia/GLM-5.3-NVFP4 --served-model-name nvidia/GLM-5.3-NVFP4 --trust-remote-code --tp 8 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 262144 --mem-fraction-static 0.76 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --disable-shared-experts-fusion --attention-backend dsa --fp4-gemm-backend flashinfer_cutlass --moe-runner-backend flashinfer_cutlass --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context4,2741,918 ms TTFT
16K context4,3093,766 ms TTFT
32K context4,2427,620 ms TTFT
64K context4,17115,466 ms TTFT
128K context3,99032,300 ms TTFT
251K context3,59370,290 ms TTFT
256K context3,48673,904 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 56.364 ms TTFT | 101.1561 ms TTFT | 183.6367 ms TTFT | 308.9419 ms TTFT | 476.3230 ms TTFT | Over capacity | Over capacity | Over capacity |
| 16K | 56.0110 ms TTFT | 101.1146 ms TTFT | 182.6199 ms TTFT | 305.5303 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 32K | 55.7157 ms TTFT | 100.0181 ms TTFT | 180.8325 ms TTFT | 305.2539 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 64K | 55.3219 ms TTFT | 99.3289 ms TTFT | 179.5477 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 55.1298 ms TTFT | 98.8495 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 251K | 54.3552 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 4,274.0 | — | 1,918 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 4,309.0 | — | 3,766 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 4,242.0 | — | 7,620 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,171.0 | — | 15,466 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 3,990.0 | — | 32,300 ms | — | — | 1 | — | — |
| Prefill | 251K | 1 | Valid | 3,593.0 | — | 70,290 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 3,486.0 | — | 73,904 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 56.3 | 56.4 | 64 ms | 17.7 ms | 1.0 | 0 | 1,689 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 101.1 | 50.0 | 561 ms | 20.0 ms | 2.0 | 0 | 3,032 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 183.6 | 45.9 | 367 ms | 21.8 ms | 4.0 | 0 | 5,508 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 308.9 | 38.6 | 419 ms | 25.9 ms | 8.0 | 0 | 9,264 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 476.3 | 30.0 | 230 ms | 33.4 ms | 16.0 | 0 | 14,288 | 30.0 s |
| Sustained decode | 8K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 56.0 | 56.0 | 110 ms | 17.9 ms | 1.0 | 0 | 1,678 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 101.1 | 50.6 | 146 ms | 19.8 ms | 2.0 | 0 | 3,032 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 182.6 | 45.7 | 199 ms | 21.9 ms | 4.0 | 0 | 5,476 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 305.5 | 38.3 | 303 ms | 26.1 ms | 8.0 | 0 | 9,160 | 30.0 s |
| Sustained decode | 16K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 55.7 | 55.7 | 157 ms | 18.0 ms | 1.0 | 0 | 1,669 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 100.0 | 50.0 | 181 ms | 20.0 ms | 2.0 | 0 | 3,000 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 180.8 | 45.1 | 325 ms | 22.2 ms | 4.0 | 0 | 5,424 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 305.2 | 38.2 | 539 ms | 26.2 ms | 8.0 | 0 | 9,152 | 30.0 s |
| Sustained decode | 32K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 55.3 | 55.3 | 219 ms | 18.1 ms | 1.0 | 0 | 1,659 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 99.3 | 49.7 | 289 ms | 20.1 ms | 2.0 | 0 | 2,978 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 179.5 | 44.9 | 477 ms | 22.3 ms | 4.0 | 0 | 5,384 | 30.0 s |
| Sustained decode | 64K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 55.1 | 55.1 | 298 ms | 18.2 ms | 1.0 | 0 | 1,652 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 98.8 | 49.5 | 495 ms | 20.2 ms | 2.0 | 0 | 2,962 | 30.0 s |
| Sustained decode | 128K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 1 | Valid | 54.3 | 54.4 | 552 ms | 18.4 ms | 1.0 | 0 | 1,629 | 30.0 s |
| Sustained decode | 251K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 251K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.