Speed test matrix
GLM-5.3-Flash on RTX PRO 6000 Blackwell Server Edition 96GB ×8
sglang · 0.5.6.post3.dev11345+g028ac64f7 · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11345+g028ac64f7
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
028ac64f797f
Run
- Served model
- zai-org/GLM-5.3-Flash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-incoai-sglang-pr-028ac64f7/bin/sglang" serve --model-path zai-org/GLM-5.3-Flash --served-model-name zai-org/GLM-5.3-Flash --trust-remote-code --tp 8 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 1048576 --mem-fraction-static 0.78 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --dsa-prefill-backend flashinfer_sparse_mla --dsa-decode-backend flashinfer_sparse_mla --linear-attn-backend triton --attention-backend dsa --fp8-gemm-backend cutlass --moe-runner-backend triton
Prefill
prompt tok/s by context length8K context9,863831 ms TTFT
16K context11.6K1,402 ms TTFT
32K context11.8K2,746 ms TTFT
64K context11.7K5,528 ms TTFT
128K context11.5K11,197 ms TTFT
256K context8,33030,926 ms TTFT
512K context10.3K49,794 ms TTFT
1019K context8,889115,306 ms TTFT
1024K context8,876116,045 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 118.8118 ms TTFT | 217.1327 ms TTFT | 383.0467 ms TTFT | 639.4455 ms TTFT | 940.1535 ms TTFT | 1,336.6698 ms TTFT | 1,915.4704 ms TTFT | 3,095.71,353 ms TTFT |
| 16K | 117.5131 ms TTFT | 219.2220 ms TTFT | 384.9240 ms TTFT | 638.2331 ms TTFT | 932.4540 ms TTFT | 1,322.2972 ms TTFT | 1,886.51,844 ms TTFT | 3,078.78,742 ms TTFT |
| 32K | 117.2156 ms TTFT | 219.1254 ms TTFT | 384.6364 ms TTFT | 631.5523 ms TTFT | 933.3964 ms TTFT | 1,328.31,873 ms TTFT | 1,886.53,702 ms TTFT | Over capacity |
| 64K | 116.1226 ms TTFT | 217.3301 ms TTFT | 380.0479 ms TTFT | 628.5887 ms TTFT | 924.11,747 ms TTFT | 1,315.93,494 ms TTFT | Over capacity | Over capacity |
| 128K | 115.7308 ms TTFT | 215.5472 ms TTFT | 378.8805 ms TTFT | 625.21,610 ms TTFT | 920.33,204 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 113.0533 ms TTFT | 210.8918 ms TTFT | 370.01,792 ms TTFT | 610.43,586 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 111.01,024 ms TTFT | 204.72,209 ms TTFT | 360.04,027 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 103.22,300 ms TTFT | 191.34,608 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,863.0 | — | 831 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11.6K | — | 1,402 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 11.8K | — | 2,746 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.7K | — | 5,528 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.5K | — | 11,197 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,330.0 | — | 30,926 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 10.3K | — | 49,794 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,889.0 | — | 115,306 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 8,876.0 | — | 116,045 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 118.8 | 118.9 | 118 ms | 8.4 ms | 1.0 | 0 | 3,565 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 217.1 | 108.0 | 327 ms | 9.3 ms | 2.0 | 0 | 6,514 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 383.0 | 95.9 | 467 ms | 10.4 ms | 4.0 | 0 | 11,488 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 639.4 | 80.1 | 455 ms | 12.5 ms | 8.0 | 0 | 19,176 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 940.1 | 58.6 | 535 ms | 17.1 ms | 16.0 | 0 | 28,192 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,336.6 | 41.8 | 698 ms | 23.9 ms | 32.0 | 0 | 40,096 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,915.4 | 29.9 | 704 ms | 33.4 ms | 64.0 | 0 | 57,472 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 3,095.7 | 24.2 | 1,353 ms | 41.3 ms | 128.0 | 0 | 92,800 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 117.5 | 117.6 | 131 ms | 8.5 ms | 1.0 | 0 | 3,528 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 219.2 | 109.6 | 220 ms | 9.1 ms | 2.0 | 0 | 6,576 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 384.9 | 96.4 | 240 ms | 10.4 ms | 4.0 | 0 | 11,560 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 638.2 | 80.0 | 331 ms | 12.5 ms | 8.0 | 0 | 19,152 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 932.4 | 58.4 | 540 ms | 17.1 ms | 16.0 | 0 | 27,967 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,322.2 | 41.4 | 972 ms | 24.1 ms | 32.0 | 0 | 39,616 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,886.5 | 29.6 | 1,844 ms | 33.7 ms | 64.0 | 0 | 56,576 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 3,078.7 | 24.4 | 8,742 ms | 41.0 ms | 128.0 | 0 | 92,288 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 117.2 | 117.2 | 156 ms | 8.5 ms | 1.0 | 0 | 3,515 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 219.1 | 109.6 | 254 ms | 9.1 ms | 2.0 | 0 | 6,574 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 384.6 | 96.2 | 364 ms | 10.4 ms | 4.0 | 0 | 11,536 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 631.5 | 78.8 | 523 ms | 12.7 ms | 8.0 | 0 | 18,944 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 933.3 | 58.4 | 964 ms | 17.1 ms | 16.0 | 0 | 27,985 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,328.3 | 41.6 | 1,873 ms | 24.1 ms | 32.0 | 0 | 39,840 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,886.5 | 29.6 | 3,702 ms | 33.8 ms | 64.0 | 0 | 56,576 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 116.1 | 116.2 | 226 ms | 8.6 ms | 1.0 | 0 | 3,484 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 217.3 | 108.7 | 301 ms | 9.2 ms | 2.0 | 0 | 6,520 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 380.0 | 95.0 | 479 ms | 10.5 ms | 4.0 | 0 | 11,400 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 628.5 | 78.6 | 887 ms | 12.7 ms | 8.0 | 0 | 18,840 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 924.1 | 57.9 | 1,747 ms | 17.3 ms | 16.0 | 0 | 27,696 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,315.9 | 41.3 | 3,494 ms | 24.2 ms | 32.0 | 0 | 39,456 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 115.7 | 115.7 | 308 ms | 8.6 ms | 1.0 | 0 | 3,469 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 215.5 | 107.8 | 472 ms | 9.3 ms | 2.0 | 0 | 6,462 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 378.8 | 95.0 | 805 ms | 10.5 ms | 4.0 | 0 | 11,360 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 625.2 | 78.7 | 1,610 ms | 12.7 ms | 8.0 | 0 | 18,752 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 920.3 | 58.5 | 3,204 ms | 17.1 ms | 16.0 | 0 | 27,600 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 113.0 | 113.0 | 533 ms | 8.9 ms | 1.0 | 0 | 3,386 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 210.8 | 105.9 | 918 ms | 9.4 ms | 2.0 | 0 | 6,316 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 370.0 | 93.7 | 1,792 ms | 10.7 ms | 4.0 | 0 | 11,088 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 610.4 | 78.9 | 3,586 ms | 12.7 ms | 8.0 | 0 | 18,312 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 111.0 | 111.1 | 1,024 ms | 9.0 ms | 1.0 | 0 | 3,331 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 204.7 | 103.8 | 2,209 ms | 9.6 ms | 2.0 | 0 | 6,144 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 360.0 | 93.3 | 4,027 ms | 10.7 ms | 4.0 | 0 | 10,804 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 103.2 | 103.4 | 2,300 ms | 9.7 ms | 1.0 | 0 | 3,099 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 191.3 | 98.8 | 4,608 ms | 10.1 ms | 2.0 | 0 | 5,740 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.