Speed test matrix
GLM-5.3-Flash on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.30.1rc1.dev162+g2617fe938 · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev162+g2617fe938
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
2617fe938355
Run
- Served model
- zai-org/GLM-5.3-Flash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_rtxpro6k-glm53-vllm-2617fe938/bin/vllm" serve zai-org/GLM-5.3-Flash --served-model-name zai-org/GLM-5.3-Flash --tensor-parallel-size 8 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.79 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER_MLA_SPARSE_SM120 --linear-backend deep_gemm --moe-backend deep_gemm
Prefill
prompt tok/s by context length8K context9,835833 ms TTFT
16K context10.7K1,514 ms TTFT
32K context10.9K2,970 ms TTFT
64K context11.1K5,785 ms TTFT
128K context11.1K11,626 ms TTFT
256K context10.7K23,967 ms TTFT
512K context9,92451,892 ms TTFT
1019K context8,594119,248 ms TTFT
1024K context8,564120,247 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 108.1114 ms TTFT | 188.1164 ms TTFT | 311.6370 ms TTFT | 486.6532 ms TTFT | 731.6907 ms TTFT | 1,049.81,148 ms TTFT | 1,515.42,128 ms TTFT | 2,455.93,539 ms TTFT |
| 16K | 106.6145 ms TTFT | 188.5296 ms TTFT | 313.2460 ms TTFT | 487.4774 ms TTFT | 733.0937 ms TTFT | 1,024.61,866 ms TTFT | 1,502.12,980 ms TTFT | 2,441.05,404 ms TTFT |
| 32K | 106.5127 ms TTFT | 188.6203 ms TTFT | 311.1296 ms TTFT | 487.2433 ms TTFT | 731.7650 ms TTFT | 1,021.3910 ms TTFT | 1,500.01,600 ms TTFT | Over capacity |
| 64K | 106.0192 ms TTFT | 188.0273 ms TTFT | 312.0349 ms TTFT | 488.6542 ms TTFT | 733.4915 ms TTFT | 1,038.41,495 ms TTFT | Over capacity | Over capacity |
| 128K | 105.3394 ms TTFT | 186.7498 ms TTFT | 311.4672 ms TTFT | 485.11,097 ms TTFT | 724.01,948 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 105.3597 ms TTFT | 186.5792 ms TTFT | 310.31,163 ms TTFT | 481.21,976 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 105.71,294 ms TTFT | 183.81,836 ms TTFT | 306.42,767 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 104.52,769 ms TTFT | 179.93,833 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,835.0 | — | 833 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.7K | — | 1,514 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.9K | — | 2,970 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.1K | — | 5,785 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.1K | — | 11,626 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.7K | — | 23,967 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,924.0 | — | 51,892 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,594.0 | — | 119,248 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 8,564.0 | — | 120,247 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 108.1 | 108.2 | 114 ms | 9.2 ms | 1.0 | 0 | 3,244 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 188.1 | 93.8 | 164 ms | 10.7 ms | 2.0 | 0 | 5,640 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 311.6 | 78.1 | 370 ms | 12.8 ms | 4.0 | 0 | 9,348 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 486.6 | 61.0 | 532 ms | 16.4 ms | 8.0 | 0 | 14,592 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 731.6 | 46.0 | 907 ms | 21.7 ms | 16.0 | 0 | 21,936 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,049.8 | 32.6 | 1,148 ms | 30.7 ms | 32.0 | 0 | 31,485 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,515.4 | 23.3 | 2,128 ms | 42.9 ms | 64.0 | 0 | 45,435 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,455.9 | 18.1 | 3,539 ms | 55.1 ms | 128.0 | 0 | 73,596 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 106.6 | 106.6 | 145 ms | 9.4 ms | 1.0 | 0 | 3,199 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 188.5 | 94.1 | 296 ms | 10.6 ms | 2.0 | 0 | 5,652 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 313.2 | 78.5 | 460 ms | 12.7 ms | 4.0 | 0 | 9,392 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 487.4 | 61.1 | 774 ms | 16.4 ms | 8.0 | 0 | 14,617 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 733.0 | 45.4 | 937 ms | 22.0 ms | 16.0 | 0 | 21,984 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,024.6 | 31.5 | 1,866 ms | 31.7 ms | 32.0 | 0 | 30,718 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,502.1 | 22.4 | 2,980 ms | 44.7 ms | 64.0 | 0 | 45,052 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,441.0 | 17.2 | 5,404 ms | 58.3 ms | 128.0 | 0 | 73,212 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 106.5 | 106.5 | 127 ms | 9.4 ms | 1.0 | 0 | 3,195 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 188.6 | 94.3 | 203 ms | 10.6 ms | 2.0 | 0 | 5,659 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 311.1 | 77.7 | 296 ms | 12.9 ms | 4.0 | 0 | 9,328 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 487.2 | 60.9 | 433 ms | 16.4 ms | 8.0 | 0 | 14,616 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 731.7 | 45.8 | 650 ms | 21.9 ms | 16.0 | 0 | 21,934 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,021.3 | 31.8 | 910 ms | 31.4 ms | 32.0 | 0 | 30,621 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,500.0 | 22.4 | 1,600 ms | 44.6 ms | 64.0 | 0 | 44,987 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 106.0 | 106.0 | 192 ms | 9.4 ms | 1.0 | 0 | 3,179 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 188.0 | 94.0 | 273 ms | 10.6 ms | 2.0 | 0 | 5,640 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 312.0 | 77.7 | 349 ms | 12.9 ms | 4.0 | 0 | 9,360 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 488.6 | 60.7 | 542 ms | 16.5 ms | 8.0 | 0 | 14,655 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 733.4 | 45.4 | 915 ms | 22.0 ms | 16.0 | 0 | 21,999 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,038.4 | 31.8 | 1,495 ms | 31.5 ms | 32.0 | 0 | 31,135 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 105.3 | 105.5 | 394 ms | 9.5 ms | 1.0 | 0 | 3,159 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 186.7 | 93.3 | 498 ms | 10.7 ms | 2.0 | 0 | 5,600 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 311.4 | 77.5 | 672 ms | 12.9 ms | 4.0 | 0 | 9,340 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 485.1 | 59.9 | 1,097 ms | 16.7 ms | 8.0 | 0 | 14,550 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 724.0 | 44.2 | 1,948 ms | 22.6 ms | 16.0 | 0 | 21,712 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 105.3 | 105.8 | 597 ms | 9.5 ms | 1.0 | 0 | 3,160 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 186.5 | 93.4 | 792 ms | 10.7 ms | 2.0 | 0 | 5,594 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 310.3 | 77.4 | 1,163 ms | 12.9 ms | 4.0 | 0 | 9,308 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 481.2 | 59.8 | 1,976 ms | 16.7 ms | 8.0 | 0 | 14,432 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 105.7 | 106.2 | 1,294 ms | 9.4 ms | 1.0 | 0 | 3,171 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 183.8 | 92.2 | 1,836 ms | 10.8 ms | 2.0 | 0 | 5,512 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 306.4 | 76.7 | 2,767 ms | 13.0 ms | 4.0 | 0 | 9,188 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 104.5 | 105.2 | 2,769 ms | 9.5 ms | 1.0 | 0 | 3,133 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 179.9 | 90.5 | 3,833 ms | 11.0 ms | 2.0 | 0 | 5,396 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.