Speed test matrix
GLM-5.3-Flash on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev359+g98ed0856f · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev359+g98ed0856f
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
98ed0856f31f
Run
- Served model
- zai-org/GLM-5.3-Flash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53flash-vllm-pr-53906/bin/vllm" serve zai-org/GLM-5.3-Flash --served-model-name zai-org/GLM-5.3-Flash --tensor-parallel-size 4 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype bfloat16 --no-enable-flashinfer-autotune --speculative-config '{"method":"mtp","num_speculative_tokens":5}'Prefill
prompt tok/s by context length8K context17.3K473 ms TTFT
16K context20.3K799 ms TTFT
32K context22.2K1,458 ms TTFT
64K context21.9K2,946 ms TTFT
128K context22.3K5,785 ms TTFT
256K context21.6K11,953 ms TTFT
512K context19.2K26,815 ms TTFT
1019K context15.4K66,349 ms TTFT
1024K context15.3K67,155 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 171.6325 ms TTFT | 298.1353 ms TTFT | 500.0230 ms TTFT | 852.5609 ms TTFT | 1,367.8876 ms TTFT | 2,234.91,233 ms TTFT | 3,456.82,244 ms TTFT | 3,676.1Requests queued3,037 ms TTFT |
| 16K | 168.1495 ms TTFT | 309.3535 ms TTFT | 483.0342 ms TTFT | 858.0579 ms TTFT | 1,336.4813 ms TTFT | 2,234.61,180 ms TTFT | 3,457.72,193 ms TTFT | Over capacity |
| 32K | 164.6840 ms TTFT | 291.5893 ms TTFT | 475.3441 ms TTFT | 863.0541 ms TTFT | 1,340.3923 ms TTFT | 2,175.41,474 ms TTFT | Over capacity | Over capacity |
| 64K | 164.21,695 ms TTFT | 290.61,758 ms TTFT | 490.7702 ms TTFT | 855.21,158 ms TTFT | 1,312.31,762 ms TTFT | Over capacity | Over capacity | Over capacity |
| 128K | 153.33,054 ms TTFT | 276.93,147 ms TTFT | 474.5684 ms TTFT | 803.41,244 ms TTFT | 1,311.72,176 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 150.66,362 ms TTFT | 270.76,515 ms TTFT | 453.01,461 ms TTFT | 817.92,383 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 139.614,263 ms TTFT | 285.314,768 ms TTFT | 460.93,378 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 143.534,887 ms TTFT | Failed | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Failed: requests to the server failed during this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.3K | — | 473 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 20.3K | — | 799 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.2K | — | 1,458 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 21.9K | — | 2,946 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 22.3K | — | 5,785 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 21.6K | — | 11,953 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 19.2K | — | 26,815 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 15.4K | — | 66,349 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 15.3K | — | 67,155 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 171.6 | 174.0 | 325 ms | 5.8 ms | 1.0 | 1 | 5,146 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 298.1 | 157.5 | 353 ms | 6.4 ms | 2.0 | 2 | 8,938 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 500.0 | 128.4 | 230 ms | 7.8 ms | 4.0 | 3 | 14,992 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 852.5 | 108.4 | 609 ms | 9.2 ms | 8.0 | 1 | 25,569 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,367.8 | 85.7 | 876 ms | 11.7 ms | 16.0 | 0 | 41,026 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,234.9 | 69.7 | 1,233 ms | 14.4 ms | 32.0 | 0 | 67,021 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,456.8 | 52.9 | 2,244 ms | 18.9 ms | 64.0 | 0 | 103,668 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 3,676.1 | 30.8 | 3,037 ms | 32.5 ms | 83.4 | 0 | 110,188 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 168.1 | 172.4 | 495 ms | 5.8 ms | 1.0 | 1 | 5,041 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 309.3 | 156.4 | 535 ms | 6.4 ms | 2.0 | 2 | 9,273 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 483.0 | 129.3 | 342 ms | 7.8 ms | 4.0 | 2 | 14,480 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 858.0 | 109.1 | 579 ms | 9.2 ms | 8.0 | 0 | 25,728 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,336.4 | 86.0 | 813 ms | 11.6 ms | 16.0 | 0 | 40,077 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,234.6 | 70.2 | 1,180 ms | 14.3 ms | 32.0 | 0 | 67,039 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,457.7 | 53.1 | 2,193 ms | 18.8 ms | 64.0 | 0 | 103,711 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 164.6 | 169.5 | 840 ms | 5.9 ms | 1.0 | 1 | 4,937 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 291.5 | 153.9 | 893 ms | 6.5 ms | 2.0 | 2 | 8,746 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 475.3 | 124.2 | 441 ms | 8.0 ms | 4.0 | 3 | 14,250 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 863.0 | 109.0 | 541 ms | 9.2 ms | 8.0 | 0 | 25,873 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,340.3 | 83.0 | 923 ms | 12.0 ms | 16.0 | 0 | 40,187 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,175.4 | 67.4 | 1,474 ms | 14.8 ms | 32.0 | 0 | 65,219 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 164.2 | 174.2 | 1,695 ms | 5.8 ms | 1.0 | 1 | 4,926 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 290.6 | 156.4 | 1,758 ms | 6.4 ms | 2.0 | 2 | 8,716 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 490.7 | 123.3 | 702 ms | 8.1 ms | 4.0 | 3 | 14,722 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 855.2 | 108.6 | 1,158 ms | 9.2 ms | 8.0 | 0 | 25,656 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,312.3 | 81.7 | 1,762 ms | 12.2 ms | 16.0 | 0 | 39,329 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 153.3 | 157.8 | 3,054 ms | 6.3 ms | 1.0 | 1 | 4,598 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 276.9 | 149.3 | 3,147 ms | 6.7 ms | 2.0 | 2 | 8,305 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 474.5 | 122.1 | 684 ms | 8.2 ms | 4.0 | 3 | 14,225 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 803.4 | 101.2 | 1,244 ms | 9.9 ms | 8.0 | 0 | 24,084 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,311.7 | 81.2 | 2,176 ms | 12.3 ms | 16.0 | 0 | 39,352 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 150.6 | 159.2 | 6,362 ms | 6.3 ms | 1.0 | 1 | 4,516 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 270.7 | 146.6 | 6,515 ms | 6.8 ms | 2.0 | 2 | 8,121 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 453.0 | 113.0 | 1,461 ms | 8.8 ms | 4.0 | 0 | 13,587 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 817.9 | 102.8 | 2,383 ms | 9.7 ms | 8.0 | 0 | 24,518 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 139.6 | 163.0 | 14,263 ms | 6.2 ms | 1.0 | 1 | 4,187 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 285.3 | 154.4 | 14,768 ms | 6.5 ms | 1.9 | 2 | 8,555 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 460.9 | 116.5 | 3,378 ms | 8.6 ms | 4.0 | 1 | 13,806 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 143.5 | 171.2 | 34,887 ms | 5.9 ms | 0.9 | 1 | 4,302 | 30.0 s |
| Sustained decode | 1019K | 2 | Failed | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Failed: requests to the server failed during this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.