Speed test matrix
GLM-5.3-Flash on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.30.1rc1.dev452+g73a583112 · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- zai-org/GLM-5.3-Flash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 "$HOME/env_rtxpro6k-incoai-vllm-73a583112/bin/vllm" serve zai-org/GLM-5.3-Flash --served-model-name zai-org/GLM-5.3-Flash --tensor-parallel-size 8 --reasoning-parser glm47 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.79 --host 0.0.0.0 --port 8000 --api-key <redacted> --no-enable-flashinfer-autotune --attention-backend FLASHINFER_MLA_SPARSE_SM120 --linear-backend b12x --moe-backend deep_gemm
Prefill
prompt tok/s by context length8K context9,733842 ms TTFT
16K context6,8842,357 ms TTFT
32K context8,7523,692 ms TTFT
64K context11.2K5,758 ms TTFT
128K context10.4K12,356 ms TTFT
256K context10.7K23,990 ms TTFT
512K context9,90352,003 ms TTFT
1019K context8,587119,336 ms TTFT
1024K context8,485121,368 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 107.9126 ms TTFT | 188.2183 ms TTFT | 314.0306 ms TTFT | 492.0475 ms TTFT | 739.3856 ms TTFT | 1,048.6929 ms TTFT | 1,509.21,734 ms TTFT | 2,462.64,058 ms TTFT |
| 16K | 106.8151 ms TTFT | 189.4243 ms TTFT | 315.4418 ms TTFT | 487.7732 ms TTFT | 741.5898 ms TTFT | 1,047.61,614 ms TTFT | 1,530.12,990 ms TTFT | 2,442.15,143 ms TTFT |
| 32K | 106.6133 ms TTFT | 189.6248 ms TTFT | 311.9298 ms TTFT | 475.8390 ms TTFT | 729.9528 ms TTFT | 1,002.6963 ms TTFT | 1,432.53,235 ms TTFT | Over capacity |
| 64K | 106.6199 ms TTFT | 189.0283 ms TTFT | 310.5403 ms TTFT | 481.6563 ms TTFT | 726.0873 ms TTFT | 1,026.81,524 ms TTFT | Over capacity | Over capacity |
| 128K | 106.2366 ms TTFT | 186.9511 ms TTFT | 309.6713 ms TTFT | 486.51,151 ms TTFT | 717.32,007 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 106.2607 ms TTFT | 186.0864 ms TTFT | 309.81,239 ms TTFT | 480.02,058 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 106.11,323 ms TTFT | 185.61,863 ms TTFT | 307.42,883 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 104.92,761 ms TTFT | 181.63,864 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,733.0 | — | 842 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,884.0 | — | 2,357 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,752.0 | — | 3,692 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 11.2K | — | 5,758 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.4K | — | 12,356 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.7K | — | 23,990 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,903.0 | — | 52,003 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,587.0 | — | 119,336 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 8,485.0 | — | 121,368 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 107.9 | 108.0 | 126 ms | 9.3 ms | 1.0 | 0 | 3,237 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 188.2 | 94.0 | 183 ms | 10.6 ms | 2.0 | 0 | 5,646 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 314.0 | 78.6 | 306 ms | 12.7 ms | 4.0 | 0 | 9,421 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 492.0 | 61.6 | 475 ms | 16.2 ms | 8.0 | 0 | 14,760 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 739.3 | 46.4 | 856 ms | 21.5 ms | 16.0 | 0 | 22,174 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,048.6 | 32.3 | 929 ms | 30.9 ms | 32.0 | 0 | 31,456 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,509.2 | 22.4 | 1,734 ms | 44.6 ms | 64.0 | 0 | 45,278 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,462.6 | 17.7 | 4,058 ms | 56.5 ms | 128.0 | 0 | 73,850 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 106.8 | 106.8 | 151 ms | 9.4 ms | 1.0 | 0 | 3,204 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 189.4 | 94.6 | 243 ms | 10.6 ms | 2.0 | 0 | 5,682 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 315.4 | 79.0 | 418 ms | 12.7 ms | 4.0 | 0 | 9,460 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 487.7 | 61.1 | 732 ms | 16.4 ms | 8.0 | 0 | 14,631 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 741.5 | 45.9 | 898 ms | 21.8 ms | 16.0 | 0 | 22,239 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,047.6 | 32.1 | 1,614 ms | 31.2 ms | 32.0 | 0 | 31,424 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,530.1 | 22.8 | 2,990 ms | 43.8 ms | 64.0 | 0 | 45,885 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 2,442.1 | 17.0 | 5,143 ms | 58.7 ms | 128.0 | 0 | 73,212 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 106.6 | 106.6 | 133 ms | 9.4 ms | 1.0 | 0 | 3,198 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 189.6 | 94.7 | 248 ms | 10.6 ms | 2.0 | 0 | 5,688 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 311.9 | 78.0 | 298 ms | 12.8 ms | 4.0 | 0 | 9,352 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 475.8 | 59.4 | 390 ms | 16.8 ms | 8.0 | 0 | 14,272 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 729.9 | 45.3 | 528 ms | 22.1 ms | 16.0 | 0 | 21,888 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,002.6 | 31.1 | 963 ms | 32.2 ms | 32.0 | 0 | 30,048 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,432.5 | 22.4 | 3,235 ms | 44.6 ms | 64.0 | 0 | 42,940 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 106.6 | 106.6 | 199 ms | 9.4 ms | 1.0 | 0 | 3,198 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 189.0 | 94.4 | 283 ms | 10.6 ms | 2.0 | 0 | 5,668 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 310.5 | 77.8 | 403 ms | 12.9 ms | 4.0 | 0 | 9,313 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 481.6 | 60.0 | 563 ms | 16.7 ms | 8.0 | 0 | 14,440 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 726.0 | 44.9 | 873 ms | 22.3 ms | 16.0 | 0 | 21,776 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,026.8 | 31.4 | 1,524 ms | 31.8 ms | 32.0 | 0 | 30,783 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 106.2 | 106.4 | 366 ms | 9.4 ms | 1.0 | 0 | 3,185 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 186.9 | 93.4 | 511 ms | 10.7 ms | 2.0 | 0 | 5,606 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 309.6 | 77.1 | 713 ms | 13.0 ms | 4.0 | 0 | 9,288 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 486.5 | 60.0 | 1,151 ms | 16.7 ms | 8.0 | 0 | 14,593 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 717.3 | 43.8 | 2,007 ms | 22.8 ms | 16.0 | 0 | 21,504 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 106.2 | 106.6 | 607 ms | 9.4 ms | 1.0 | 0 | 3,185 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 186.0 | 93.2 | 864 ms | 10.7 ms | 2.0 | 0 | 5,580 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 309.8 | 77.5 | 1,239 ms | 12.9 ms | 4.0 | 0 | 9,292 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 480.0 | 59.7 | 2,058 ms | 16.7 ms | 8.0 | 0 | 14,400 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 106.1 | 106.6 | 1,323 ms | 9.4 ms | 1.0 | 0 | 3,181 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 185.6 | 93.1 | 1,863 ms | 10.7 ms | 2.0 | 0 | 5,568 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 307.4 | 76.9 | 2,883 ms | 13.0 ms | 4.0 | 0 | 9,222 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 104.9 | 105.7 | 2,761 ms | 9.5 ms | 1.0 | 0 | 3,146 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 181.6 | 91.4 | 3,864 ms | 10.9 ms | 2.0 | 0 | 5,446 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.