Speed test matrix
GLM-5.3-Flash-BF16 on H200 SXM 141GB ×8
vllm · 0.28.1rc1.dev359+g98ed0856f · BF168 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev359+g98ed0856f
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
98ed0856f31f
Run
- Served model
- glm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53flash-vllm-pr-53906/bin/vllm" serve RedHatAI/GLM-5.3-Flash-BF16 --served-model-name glm --tensor-parallel-size 8 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype bfloat16 --no-enable-flashinfer-autotune
Prefill
prompt tok/s by context length8K context20.1K408 ms TTFT
16K context23.7K685 ms TTFT
32K context27.6K1,172 ms TTFT
64K context24.9K2,591 ms TTFT
128K context29.7K4,337 ms TTFT
256K context27.6K9,320 ms TTFT
512K context24.1K21,396 ms TTFT
1019K context18.8K54,647 ms TTFT
1024K context18.5K55,720 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 140.5171 ms TTFT | 249.0245 ms TTFT | 433.3930 ms TTFT | 662.8370 ms TTFT | 1,185.2493 ms TTFT | 1,815.5582 ms TTFT | 3,070.41,102 ms TTFT | 5,063.61,799 ms TTFT |
| 16K | 140.590 ms TTFT | 248.9127 ms TTFT | 429.9165 ms TTFT | 655.6253 ms TTFT | 1,180.0463 ms TTFT | 1,802.0853 ms TTFT | 3,105.51,231 ms TTFT | 5,058.01,981 ms TTFT |
| 32K | 141.8121 ms TTFT | 251.8168 ms TTFT | 435.9226 ms TTFT | 664.7454 ms TTFT | 1,183.6688 ms TTFT | 1,800.51,102 ms TTFT | 3,074.51,940 ms TTFT | Over capacity |
| 64K | 141.1191 ms TTFT | 249.1293 ms TTFT | 431.8510 ms TTFT | 651.5807 ms TTFT | 1,175.61,208 ms TTFT | 1,767.72,053 ms TTFT | Over capacity | Over capacity |
| 128K | 138.9349 ms TTFT | 243.8411 ms TTFT | 416.3714 ms TTFT | 619.41,219 ms TTFT | 1,144.32,201 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 136.2777 ms TTFT | 243.01,046 ms TTFT | 409.11,591 ms TTFT | 600.52,732 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 131.61,718 ms TTFT | 238.72,375 ms TTFT | 406.03,658 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 137.33,660 ms TTFT | 231.15,364 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 20.1K | — | 408 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 23.7K | — | 685 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 27.6K | — | 1,172 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 24.9K | — | 2,591 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 29.7K | — | 4,337 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 27.6K | — | 9,320 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 24.1K | — | 21,396 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 18.8K | — | 54,647 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 18.5K | — | 55,720 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 140.5 | 141.3 | 171 ms | 7.1 ms | 1.0 | 1 | 4,213 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 249.0 | 121.0 | 245 ms | 8.3 ms | 2.0 | 2 | 7,468 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 433.3 | 108.4 | 930 ms | 9.2 ms | 4.0 | 0 | 12,996 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 662.8 | 83.0 | 370 ms | 12.1 ms | 8.0 | 0 | 19,880 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,185.2 | 74.5 | 493 ms | 13.4 ms | 16.0 | 0 | 35,551 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,815.5 | 56.5 | 582 ms | 17.7 ms | 32.0 | 0 | 54,463 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,070.4 | 47.0 | 1,102 ms | 21.3 ms | 64.0 | 0 | 92,097 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 5,063.6 | 39.0 | 1,799 ms | 25.7 ms | 128.0 | 0 | 151,926 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 140.5 | 141.0 | 90 ms | 7.1 ms | 1.0 | 1 | 4,215 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 248.9 | 124.9 | 127 ms | 8.0 ms | 2.0 | 2 | 7,468 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 429.9 | 107.4 | 165 ms | 9.3 ms | 4.0 | 0 | 12,896 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 655.6 | 81.8 | 253 ms | 12.2 ms | 8.0 | 0 | 19,664 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,180.0 | 73.3 | 463 ms | 13.6 ms | 16.0 | 0 | 35,391 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,802.0 | 56.1 | 853 ms | 17.8 ms | 32.0 | 0 | 54,042 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,105.5 | 48.0 | 1,231 ms | 20.8 ms | 64.0 | 0 | 93,117 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 5,058.0 | 38.6 | 1,981 ms | 25.9 ms | 128.0 | 0 | 151,677 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 141.8 | 141.9 | 121 ms | 7.0 ms | 1.0 | 1 | 4,253 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 251.8 | 124.6 | 168 ms | 8.0 ms | 2.0 | 2 | 7,552 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 435.9 | 108.6 | 226 ms | 9.2 ms | 4.0 | 0 | 13,073 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 664.7 | 82.7 | 454 ms | 12.1 ms | 8.0 | 0 | 19,936 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,183.6 | 73.5 | 688 ms | 13.6 ms | 16.0 | 0 | 35,502 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,800.5 | 55.3 | 1,102 ms | 18.1 ms | 32.0 | 0 | 54,016 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,074.5 | 46.6 | 1,940 ms | 21.5 ms | 64.0 | 0 | 92,223 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 141.1 | 142.1 | 191 ms | 7.0 ms | 1.0 | 1 | 4,234 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 249.1 | 125.7 | 293 ms | 8.0 ms | 2.0 | 2 | 7,472 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 431.8 | 107.3 | 510 ms | 9.3 ms | 4.0 | 0 | 12,952 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 651.5 | 80.8 | 807 ms | 12.4 ms | 8.0 | 0 | 19,544 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,175.6 | 71.9 | 1,208 ms | 13.9 ms | 16.0 | 0 | 35,263 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,767.7 | 52.8 | 2,053 ms | 18.9 ms | 32.0 | 0 | 53,022 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 138.9 | 141.3 | 349 ms | 7.1 ms | 1.0 | 1 | 4,166 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 243.8 | 124.2 | 411 ms | 8.1 ms | 1.9 | 2 | 7,314 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 416.3 | 104.0 | 714 ms | 9.6 ms | 4.0 | 0 | 12,490 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 619.4 | 77.2 | 1,219 ms | 13.0 ms | 8.0 | 0 | 18,577 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,144.3 | 70.8 | 2,201 ms | 14.1 ms | 16.0 | 0 | 34,319 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 136.2 | 140.9 | 777 ms | 7.1 ms | 1.0 | 1 | 4,087 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 243.0 | 122.3 | 1,046 ms | 8.2 ms | 2.0 | 1 | 7,289 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 409.1 | 102.5 | 1,591 ms | 9.8 ms | 4.0 | 0 | 12,274 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 600.5 | 75.1 | 2,732 ms | 13.3 ms | 8.0 | 0 | 18,014 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 131.6 | 143.1 | 1,718 ms | 7.0 ms | 1.0 | 1 | 3,946 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 238.7 | 121.9 | 2,375 ms | 8.2 ms | 2.0 | 2 | 7,138 | 29.9 s |
| Sustained decode | 512K | 4 | Valid | 406.0 | 101.5 | 3,658 ms | 9.9 ms | 4.0 | 0 | 12,176 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 137.3 | 139.0 | 3,660 ms | 7.2 ms | 0.9 | 1 | 3,629 | 26.4 s |
| Sustained decode | 1019K | 2 | Valid | 231.1 | 122.0 | 5,364 ms | 8.2 ms | 1.9 | 1 | 6,932 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.