Speed test matrix
GLM-5.3-Flash on RTX PRO 6000 Blackwell Server Edition 96GB ×8
vllm · 0.30.1rc1.dev162+g2617fe938 · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 8
- VRAM
- 768 GB (8 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev162+g2617fe938
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
2617fe938355
Run
- Served model
- zai-org/GLM-5.3-Flash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_rtxpro6k-glm53-vllm-2617fe938/bin/vllm" serve zai-org/GLM-5.3-Flash --served-model-name zai-org/GLM-5.3-Flash --tensor-parallel-size 8 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.78 --host 0.0.0.0 --port 8000 --api-key <redacted> --attention-backend FLASHINFER_MLA_SPARSE_SM120 --linear-backend deep_gemm --moe-backend deep_gemm --speculative-config '{"method":"mtp","num_speculative_tokens":5}'Prefill
prompt tok/s by context length8K context9,565857 ms TTFT
16K context10.3K1,571 ms TTFT
32K context10.6K3,039 ms TTFT
64K context10.7K6,019 ms TTFT
128K context10.7K12,092 ms TTFT
256K context10.3K25,002 ms TTFT
512K context9,48054,322 ms TTFT
1019K context8,157125,631 ms TTFT
1024K context8,138126,541 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 156.2211 ms TTFT | 248.9235 ms TTFT | 371.52,342 ms TTFT | 520.7876 ms TTFT | 744.11,535 ms TTFT | 1,261.72,876 ms TTFT | 1,853.95,562 ms TTFT | 1,964.9Requests queued9,058 ms TTFT |
| 16K | 148.3246 ms TTFT | 246.9283 ms TTFT | 382.5853 ms TTFT | 525.71,288 ms TTFT | 747.31,989 ms TTFT | 1,268.23,731 ms TTFT | 1,825.76,882 ms TTFT | 1,966.4Requests queued10,871 ms TTFT |
| 32K | 157.2228 ms TTFT | 251.6267 ms TTFT | 364.7696 ms TTFT | 542.3988 ms TTFT | 757.21,905 ms TTFT | 1,249.72,676 ms TTFT | 1,839.25,320 ms TTFT | Over capacity |
| 64K | 154.8296 ms TTFT | 239.6343 ms TTFT | 383.7662 ms TTFT | 532.01,203 ms TTFT | 740.11,810 ms TTFT | 1,234.03,284 ms TTFT | Over capacity | Over capacity |
| 128K | 160.3446 ms TTFT | 247.4470 ms TTFT | 361.1943 ms TTFT | 519.31,431 ms TTFT | 728.72,407 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 158.7680 ms TTFT | 239.1754 ms TTFT | 382.41,273 ms TTFT | 514.22,070 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 148.41,383 ms TTFT | 229.11,918 ms TTFT | 334.52,845 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 132.32,865 ms TTFT | 213.94,011 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 9,565.0 | — | 857 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.3K | — | 1,571 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 10.6K | — | 3,039 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 10.7K | — | 6,019 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.7K | — | 12,092 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 10.3K | — | 25,002 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 9,480.0 | — | 54,322 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 8,157.0 | — | 125,631 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 8,138.0 | — | 126,541 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 156.2 | 168.5 | 211 ms | 6.0 ms | 1.0 | 1 | 4,684 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 248.9 | 126.4 | 235 ms | 7.9 ms | 2.0 | 1 | 7,466 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 371.5 | 88.9 | 2,342 ms | 11.2 ms | 4.0 | 0 | 11,145 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 520.7 | 62.7 | 876 ms | 15.9 ms | 8.0 | 0 | 15,604 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 744.1 | 44.0 | 1,535 ms | 22.7 ms | 16.0 | 0 | 22,325 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,261.7 | 38.2 | 2,876 ms | 26.2 ms | 32.0 | 0 | 37,803 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,853.9 | 25.9 | 5,562 ms | 38.5 ms | 64.0 | 0 | 55,614 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,964.9 | 18.3 | 9,058 ms | 54.7 ms | 99.0 | 0 | 58,944 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 148.3 | 158.6 | 246 ms | 6.3 ms | 1.0 | 1 | 4,450 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 246.9 | 127.0 | 283 ms | 7.9 ms | 2.0 | 2 | 7,403 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 382.5 | 94.1 | 853 ms | 10.6 ms | 4.0 | 0 | 11,464 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 525.7 | 65.8 | 1,288 ms | 15.2 ms | 8.0 | 0 | 15,768 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 747.3 | 46.2 | 1,989 ms | 21.6 ms | 16.0 | 0 | 22,414 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,268.2 | 37.6 | 3,731 ms | 26.6 ms | 32.0 | 0 | 38,028 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,825.7 | 24.9 | 6,882 ms | 40.1 ms | 64.0 | 0 | 54,769 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,966.4 | 18.7 | 10,871 ms | 53.4 ms | 98.0 | 0 | 58,880 | 29.9 s |
| Sustained decode | 32K | 1 | Valid | 157.2 | 165.7 | 228 ms | 6.1 ms | 1.0 | 1 | 4,715 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 251.6 | 132.8 | 267 ms | 7.5 ms | 2.0 | 1 | 7,543 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 364.7 | 91.6 | 696 ms | 10.9 ms | 4.0 | 0 | 10,940 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 542.3 | 67.6 | 988 ms | 14.8 ms | 8.0 | 0 | 16,250 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 757.2 | 45.9 | 1,905 ms | 21.8 ms | 16.0 | 0 | 22,682 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,249.7 | 37.3 | 2,676 ms | 26.8 ms | 32.0 | 0 | 37,415 | 29.9 s |
| Sustained decode | 32K | 64 | Valid | 1,839.2 | 26.1 | 5,320 ms | 38.3 ms | 64.0 | 0 | 55,071 | 29.9 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 154.8 | 158.0 | 296 ms | 6.3 ms | 1.0 | 1 | 4,644 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 239.6 | 126.9 | 343 ms | 7.9 ms | 2.0 | 1 | 7,189 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 383.7 | 96.0 | 662 ms | 10.4 ms | 4.0 | 0 | 11,501 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 532.0 | 67.2 | 1,203 ms | 14.9 ms | 8.0 | 0 | 15,957 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 740.1 | 45.7 | 1,810 ms | 21.9 ms | 16.0 | 0 | 22,181 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,234.0 | 37.6 | 3,284 ms | 26.6 ms | 32.0 | 0 | 37,019 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 160.3 | 162.9 | 446 ms | 6.1 ms | 1.0 | 1 | 4,808 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 247.4 | 127.5 | 470 ms | 7.8 ms | 2.0 | 2 | 7,420 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 361.1 | 91.6 | 943 ms | 10.9 ms | 4.0 | 0 | 10,832 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 519.3 | 66.7 | 1,431 ms | 15.0 ms | 8.0 | 0 | 15,575 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 728.7 | 44.8 | 2,407 ms | 22.3 ms | 16.0 | 0 | 21,846 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 158.7 | 164.0 | 680 ms | 6.1 ms | 1.0 | 1 | 4,758 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 239.1 | 126.2 | 754 ms | 7.9 ms | 2.0 | 2 | 7,173 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 382.4 | 96.9 | 1,273 ms | 10.3 ms | 4.0 | 0 | 11,470 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 514.2 | 64.1 | 2,070 ms | 15.6 ms | 8.0 | 0 | 15,414 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 148.4 | 160.9 | 1,383 ms | 6.2 ms | 1.0 | 1 | 4,449 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 229.1 | 118.6 | 1,918 ms | 8.4 ms | 2.0 | 0 | 6,870 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 334.5 | 86.1 | 2,845 ms | 11.6 ms | 4.0 | 0 | 10,026 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 132.3 | 154.6 | 2,865 ms | 6.5 ms | 0.9 | 1 | 3,968 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 213.9 | 110.0 | 4,011 ms | 9.1 ms | 2.0 | 0 | 6,413 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.