Speed test matrix
GLM-5.3-Flash on RTX PRO 6000 Blackwell Server Edition 96GB ×8
sglang · 0.5.6.post3.dev11345+g028ac64f7 · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev11345+g028ac64f7
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
028ac64f797f
Run
- Served model
- zai-org/GLM-5.3-Flash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-incoai-sglang-pr-028ac64f7/bin/sglang" serve --model-path zai-org/GLM-5.3-Flash --served-model-name zai-org/GLM-5.3-Flash --trust-remote-code --tp 8 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 1048576 --mem-fraction-static 0.76 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --dsa-prefill-backend flashinfer_sparse_mla --dsa-decode-backend flashinfer_sparse_mla --linear-attn-backend triton --attention-backend dsa --fp8-gemm-backend cutlass --moe-runner-backend triton --speculative-algorithm DFLASH --speculative-draft-model-path incoai/GLM-5.3-Flash-DFlash2 --speculative-draft-attention-backend fa4
Prefill
prompt tok/s by context length8K context9,232888 ms TTFT
16K context10.6K1,527 ms TTFT
32K context10.7K3,007 ms TTFT
64K context10.7K6,014 ms TTFT
128K context10.6K12,145 ms TTFT
256K context7,89032,653 ms TTFT
512K context9,63453,464 ms TTFT
1019K context8,387122,206 ms TTFT
1024K context7,806131,949 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 190.7122 ms TTFT | 267.4178 ms TTFT | 389.2501 ms TTFT | 593.5536 ms TTFT | 948.2593 ms TTFT | 1,345.8954 ms TTFT | Not run | Not run |
| 16K | 182.8143 ms TTFT | 268.5193 ms TTFT | 394.0260 ms TTFT | 578.6413 ms TTFT | 955.3438 ms TTFT | 1,345.2755 ms TTFT | Not run | Over capacity |
| 32K | 197.5172 ms TTFT | 290.3215 ms TTFT | 412.6357 ms TTFT | 605.5540 ms TTFT | 952.2950 ms TTFT | 1,356.31,854 ms TTFT | Over capacity | Over capacity |
| 64K | 190.0246 ms TTFT | 274.4274 ms TTFT | 405.3527 ms TTFT | 556.7863 ms TTFT | 936.61,809 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 177.2308 ms TTFT | 285.5382 ms TTFT | 398.9789 ms TTFT | 579.81,746 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 184.2539 ms TTFT | 257.9749 ms TTFT | 388.61,805 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 172.01,126 ms TTFT | 249.01,555 ms TTFT | 374.33,790 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 154.52,345 ms TTFT | 224.9Requests queued4,745 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,232.0 | — | 888 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.6K | — | 1,527 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.7K | — | 3,007 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.7K | — | 6,014 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.6K | — | 12,145 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 7,890.0 | — | 32,653 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,634.0 | — | 53,464 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,387.0 | — | 122,206 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 7,806.0 | — | 131,949 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 190.7 | 192.3 | 122 ms | 5.2 ms | 1.0 | 1 | 5,719 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 267.4 | 151.6 | 178 ms | 6.6 ms | 2.0 | 2 | 8,018 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 389.2 | 102.0 | 501 ms | 9.8 ms | 4.0 | 0 | 11,681 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 593.5 | 76.4 | 536 ms | 13.1 ms | 8.0 | 0 | 17,801 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 948.2 | 60.1 | 593 ms | 16.6 ms | 16.0 | 0 | 28,416 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,345.8 | 42.7 | 954 ms | 23.4 ms | 32.0 | 0 | 40,368 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 182.8 | 190.2 | 143 ms | 5.3 ms | 1.0 | 1 | 5,483 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 268.5 | 143.3 | 193 ms | 7.0 ms | 1.9 | 2 | 8,062 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 394.0 | 101.1 | 260 ms | 9.9 ms | 4.0 | 0 | 11,809 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 578.6 | 75.6 | 413 ms | 13.2 ms | 8.0 | 0 | 17,342 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 955.3 | 61.2 | 438 ms | 16.3 ms | 16.0 | 0 | 28,650 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,345.2 | 42.7 | 755 ms | 23.4 ms | 32.0 | 0 | 40,323 | 30.0 s |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 197.5 | 202.9 | 172 ms | 4.9 ms | 1.0 | 1 | 5,924 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 290.3 | 148.2 | 215 ms | 6.8 ms | 2.0 | 2 | 8,709 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 412.6 | 107.5 | 357 ms | 9.3 ms | 4.0 | 0 | 12,377 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 605.5 | 76.1 | 540 ms | 13.2 ms | 8.0 | 0 | 18,164 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 952.2 | 61.4 | 950 ms | 16.3 ms | 16.0 | 0 | 28,565 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,356.3 | 43.8 | 1,854 ms | 22.8 ms | 32.0 | 0 | 40,688 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 190.0 | 198.1 | 246 ms | 5.1 ms | 1.0 | 1 | 5,699 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 274.4 | 148.0 | 274 ms | 6.8 ms | 1.9 | 2 | 8,231 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 405.3 | 104.4 | 527 ms | 9.6 ms | 4.0 | 0 | 12,153 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 556.7 | 73.3 | 863 ms | 13.7 ms | 8.0 | 0 | 16,690 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 936.6 | 59.8 | 1,809 ms | 16.7 ms | 16.0 | 0 | 28,070 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 177.2 | 183.9 | 308 ms | 5.4 ms | 1.0 | 1 | 5,316 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 285.5 | 150.2 | 382 ms | 6.7 ms | 2.0 | 2 | 8,566 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 398.9 | 102.2 | 789 ms | 9.8 ms | 4.0 | 0 | 11,966 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 579.8 | 75.2 | 1,746 ms | 13.3 ms | 8.0 | 0 | 17,372 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 184.2 | 191.2 | 539 ms | 5.2 ms | 1.0 | 1 | 5,525 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 257.9 | 134.2 | 749 ms | 7.5 ms | 1.9 | 2 | 7,738 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 388.6 | 102.6 | 1,805 ms | 9.7 ms | 4.0 | 0 | 11,649 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 172.0 | 181.3 | 1,126 ms | 5.5 ms | 1.0 | 1 | 5,160 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 249.0 | 153.9 | 1,555 ms | 6.6 ms | 1.9 | 2 | 7,469 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 374.3 | 100.4 | 3,790 ms | 10.0 ms | 4.0 | 0 | 11,223 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 154.5 | 169.9 | 2,345 ms | 5.9 ms | 1.0 | 1 | 4,636 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 224.9 | 133.7 | 4,745 ms | 7.5 ms | 1.9 | 2 | 7,152 | 31.8 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.