Speed test matrix
GLM-5.3-Flash-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
sglang · 0.5.6.post3.dev10352+g28379a7c5 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev10352+g28379a7c5
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
28379a7c5568
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-glm53flash-sglang-pr-38430/bin/sglang" serve --model-path nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --tp 4 --reasoning-parser glm45 --tool-call-parser glm47 --context-length 1048576 --mem-fraction-static 0.77 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend dsa --dsa-prefill-backend flashinfer_sparse_mla --dsa-decode-backend flashinfer_sparse_mla --fp4-gemm-backend flashinfer_cutlass --moe-runner-backend flashinfer_cutlass --quantization modelopt_fp4
Prefill
prompt tok/s by context length8K context9,672848 ms TTFT
16K context11.9K1,364 ms TTFT
32K context12K2,701 ms TTFT
64K context11.9K5,410 ms TTFT
128K context11.8K10,933 ms TTFT
256K context8,65429,769 ms TTFT
512K context10.6K48,407 ms TTFT
1019K context9,364109,460 ms TTFT
1024K context9,161112,433 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 109.4123 ms TTFT | 158.7243 ms TTFT | 321.9399 ms TTFT | 521.2413 ms TTFT | 788.5632 ms TTFT | 1,147.8653 ms TTFT | 1,801.01,251 ms TTFT | 1,870.1Requests queued2,443 ms TTFT |
| 16K | 109.3167 ms TTFT | 158.7226 ms TTFT | 320.2329 ms TTFT | 522.6486 ms TTFT | 779.6725 ms TTFT | 1,166.41,408 ms TTFT | 1,785.32,832 ms TTFT | Over capacity |
| 32K | 109.0227 ms TTFT | 158.3249 ms TTFT | 321.3395 ms TTFT | 522.1847 ms TTFT | 776.61,658 ms TTFT | 1,154.43,287 ms TTFT | Over capacity | Over capacity |
| 64K | 108.5321 ms TTFT | 158.0488 ms TTFT | 319.4860 ms TTFT | 513.01,602 ms TTFT | 771.33,178 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 107.4523 ms TTFT | 156.1877 ms TTFT | 317.11,603 ms TTFT | 511.13,312 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 105.4988 ms TTFT | 154.21,827 ms TTFT | 313.63,456 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 102.02,076 ms TTFT | 150.84,036 ms TTFT | 305.77,940 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 95.56,624 ms TTFT | 143.58,934 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,672.0 | — | 848 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 11.9K | — | 1,364 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12K | — | 2,701 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.9K | — | 5,410 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.8K | — | 10,933 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 8,654.0 | — | 29,769 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 10.6K | — | 48,407 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,364.0 | — | 109,460 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,161.0 | — | 112,433 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 109.4 | 109.3 | 123 ms | 9.1 ms | 1.0 | 0 | 3,281 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 158.7 | 79.3 | 243 ms | 12.6 ms | 2.0 | 0 | 4,760 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 321.9 | 80.6 | 399 ms | 12.4 ms | 4.0 | 0 | 9,656 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 521.2 | 65.1 | 413 ms | 15.4 ms | 8.0 | 0 | 15,632 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 788.5 | 49.4 | 632 ms | 20.3 ms | 16.0 | 0 | 23,648 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,147.8 | 35.9 | 653 ms | 27.9 ms | 32.0 | 0 | 34,432 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,801.0 | 28.3 | 1,251 ms | 35.3 ms | 64.0 | 0 | 54,016 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,870.1 | 26.1 | 2,443 ms | 38.4 ms | 74.0 | 0 | 56,093 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 109.3 | 109.3 | 167 ms | 9.2 ms | 1.0 | 0 | 3,279 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 158.7 | 79.4 | 226 ms | 12.6 ms | 2.0 | 0 | 4,760 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 320.2 | 80.2 | 329 ms | 12.5 ms | 4.0 | 0 | 9,604 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 522.6 | 65.2 | 486 ms | 15.3 ms | 8.0 | 0 | 15,672 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 779.6 | 48.7 | 725 ms | 20.5 ms | 16.0 | 0 | 23,376 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,166.4 | 36.6 | 1,408 ms | 27.3 ms | 32.0 | 0 | 34,976 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,785.3 | 28.1 | 2,832 ms | 35.5 ms | 64.0 | 0 | 53,504 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 109.0 | 109.0 | 227 ms | 9.2 ms | 1.0 | 0 | 3,269 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 158.3 | 79.1 | 249 ms | 12.6 ms | 2.0 | 0 | 4,748 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 321.3 | 80.3 | 395 ms | 12.5 ms | 4.0 | 0 | 9,636 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 522.1 | 65.1 | 847 ms | 15.4 ms | 8.0 | 0 | 15,664 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 776.6 | 48.5 | 1,658 ms | 20.6 ms | 16.0 | 0 | 23,296 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,154.4 | 36.3 | 3,287 ms | 27.6 ms | 32.0 | 0 | 34,624 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 108.5 | 108.4 | 321 ms | 9.2 ms | 1.0 | 0 | 3,254 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 158.0 | 79.1 | 488 ms | 12.6 ms | 2.0 | 0 | 4,740 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 319.4 | 80.0 | 860 ms | 12.5 ms | 4.0 | 0 | 9,580 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 513.0 | 64.6 | 1,602 ms | 15.5 ms | 8.0 | 0 | 15,384 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 771.3 | 49.2 | 3,178 ms | 20.3 ms | 16.0 | 0 | 23,136 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 107.4 | 107.4 | 523 ms | 9.3 ms | 1.0 | 0 | 3,222 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 156.1 | 78.5 | 877 ms | 12.7 ms | 2.0 | 0 | 4,682 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 317.1 | 80.1 | 1,603 ms | 12.5 ms | 4.0 | 0 | 9,512 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 511.1 | 65.8 | 3,312 ms | 15.2 ms | 8.0 | 0 | 15,328 | 30.0 s |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 105.4 | 105.5 | 988 ms | 9.5 ms | 1.0 | 0 | 3,162 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 154.2 | 78.3 | 1,827 ms | 12.8 ms | 2.0 | 0 | 4,626 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 313.6 | 80.7 | 3,456 ms | 12.4 ms | 4.0 | 0 | 9,408 | 30.0 s |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 102.0 | 101.9 | 2,076 ms | 9.8 ms | 1.0 | 0 | 3,058 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 150.8 | 78.2 | 4,036 ms | 12.8 ms | 2.0 | 0 | 4,522 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 305.7 | 82.2 | 7,940 ms | 12.2 ms | 4.0 | 0 | 9,168 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 95.5 | 95.5 | 6,624 ms | 10.5 ms | 1.0 | 0 | 2,865 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 143.5 | 77.7 | 8,934 ms | 12.9 ms | 2.0 | 0 | 4,304 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.