Speed test matrix
GLM-5.3-Flash on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev359+g98ed0856f · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev359+g98ed0856f
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
98ed0856f31f
Run
- Served model
- zai-org/GLM-5.3-Flash
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53flash-vllm-pr-53906/bin/vllm" serve zai-org/GLM-5.3-Flash --served-model-name zai-org/GLM-5.3-Flash --tensor-parallel-size 4 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.87 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype bfloat16 --no-enable-flashinfer-autotune
Prefill
prompt tok/s by context length8K context19.2K427 ms TTFT
16K context23.4K692 ms TTFT
32K context24.3K1,329 ms TTFT
64K context24.4K2,649 ms TTFT
128K context23.5K5,491 ms TTFT
256K context22.3K11,540 ms TTFT
512K context20.1K25,654 ms TTFT
1019K context16.2K63,121 ms TTFT
1024K context16.1K63,990 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 134.074 ms TTFT | 236.8121 ms TTFT | 417.6148 ms TTFT | 644.2387 ms TTFT | 1,152.5629 ms TTFT | 1,840.5570 ms TTFT | 3,083.21,290 ms TTFT | 4,507.01,362 ms TTFT |
| 16K | 133.590 ms TTFT | 235.4149 ms TTFT | 413.8163 ms TTFT | 637.5236 ms TTFT | 1,148.7363 ms TTFT | 1,823.7563 ms TTFT | 3,068.9945 ms TTFT | 4,490.71,779 ms TTFT |
| 32K | 134.6120 ms TTFT | 238.5190 ms TTFT | 417.9222 ms TTFT | 648.2352 ms TTFT | 1,165.9593 ms TTFT | 1,829.81,018 ms TTFT | 3,066.81,748 ms TTFT | Over capacity |
| 64K | 134.1191 ms TTFT | 237.7283 ms TTFT | 414.0363 ms TTFT | 632.2604 ms TTFT | 1,145.21,026 ms TTFT | 1,800.81,965 ms TTFT | Over capacity | Over capacity |
| 128K | 131.4469 ms TTFT | 231.9640 ms TTFT | 398.0896 ms TTFT | 605.11,426 ms TTFT | 1,121.22,513 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 128.7878 ms TTFT | 229.61,219 ms TTFT | 393.41,814 ms TTFT | 584.32,975 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 124.81,767 ms TTFT | 228.32,505 ms TTFT | 390.23,915 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 130.84,060 ms TTFT | 223.45,684 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.2K | — | 427 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 23.4K | — | 692 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 24.3K | — | 1,329 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 24.4K | — | 2,649 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.5K | — | 5,491 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 22.3K | — | 11,540 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 20.1K | — | 25,654 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 16.2K | — | 63,121 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 16.1K | — | 63,990 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 134.0 | 134.3 | 74 ms | 7.4 ms | 1.0 | 1 | 4,019 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 236.8 | 118.4 | 121 ms | 8.4 ms | 2.0 | 0 | 7,104 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 417.6 | 104.5 | 148 ms | 9.6 ms | 4.0 | 0 | 12,528 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 644.2 | 80.4 | 387 ms | 12.4 ms | 8.0 | 0 | 19,320 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,152.5 | 72.4 | 629 ms | 13.8 ms | 16.0 | 0 | 34,576 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,840.5 | 57.3 | 570 ms | 17.5 ms | 32.0 | 0 | 55,198 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,083.2 | 48.2 | 1,290 ms | 20.7 ms | 64.0 | 0 | 92,473 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,507.0 | 34.9 | 1,362 ms | 28.7 ms | 128.0 | 0 | 135,288 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 133.5 | 134.0 | 90 ms | 7.5 ms | 1.0 | 1 | 4,004 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 235.4 | 117.7 | 149 ms | 8.5 ms | 2.0 | 0 | 7,062 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 413.8 | 103.4 | 163 ms | 9.7 ms | 4.0 | 0 | 12,412 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 637.5 | 79.7 | 236 ms | 12.5 ms | 8.0 | 0 | 19,122 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,148.7 | 71.8 | 363 ms | 13.9 ms | 16.0 | 0 | 34,448 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,823.7 | 56.8 | 563 ms | 17.6 ms | 32.0 | 0 | 54,684 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,068.9 | 47.5 | 945 ms | 21.0 ms | 64.0 | 0 | 92,029 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,490.7 | 34.5 | 1,779 ms | 29.0 ms | 128.0 | 0 | 134,775 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 134.6 | 134.7 | 120 ms | 7.4 ms | 1.0 | 1 | 4,039 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 238.5 | 119.1 | 190 ms | 8.4 ms | 2.0 | 0 | 7,154 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 417.9 | 104.3 | 222 ms | 9.6 ms | 4.0 | 0 | 12,538 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 648.2 | 80.5 | 352 ms | 12.4 ms | 8.0 | 0 | 19,440 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,165.9 | 72.3 | 593 ms | 13.8 ms | 16.0 | 0 | 34,975 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,829.8 | 56.5 | 1,018 ms | 17.7 ms | 32.0 | 0 | 54,878 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,066.8 | 46.7 | 1,748 ms | 21.4 ms | 64.0 | 0 | 91,967 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 134.1 | 135.4 | 191 ms | 7.4 ms | 1.0 | 1 | 4,024 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 237.7 | 118.8 | 283 ms | 8.4 ms | 2.0 | 0 | 7,131 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 414.0 | 103.4 | 363 ms | 9.7 ms | 4.0 | 0 | 12,420 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 632.2 | 78.7 | 604 ms | 12.7 ms | 8.0 | 0 | 18,960 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,145.2 | 71.0 | 1,026 ms | 14.1 ms | 16.0 | 0 | 34,352 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,800.8 | 55.6 | 1,965 ms | 18.0 ms | 32.0 | 0 | 54,018 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 131.4 | 135.0 | 469 ms | 7.4 ms | 1.0 | 1 | 3,942 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 231.9 | 116.0 | 640 ms | 8.6 ms | 2.0 | 0 | 6,956 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 398.0 | 99.1 | 896 ms | 10.1 ms | 4.0 | 0 | 11,940 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 605.1 | 74.5 | 1,426 ms | 13.4 ms | 8.0 | 0 | 18,152 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,121.2 | 67.5 | 2,513 ms | 14.8 ms | 16.0 | 0 | 33,631 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 128.7 | 135.4 | 878 ms | 7.4 ms | 1.0 | 1 | 3,862 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 229.6 | 114.9 | 1,219 ms | 8.7 ms | 2.0 | 0 | 6,886 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 393.4 | 98.0 | 1,814 ms | 10.2 ms | 4.0 | 0 | 11,800 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 584.3 | 72.3 | 2,975 ms | 13.8 ms | 8.0 | 0 | 17,528 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 124.8 | 143.2 | 1,767 ms | 7.0 ms | 0.9 | 1 | 3,744 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 228.3 | 114.9 | 2,505 ms | 8.7 ms | 2.0 | 0 | 6,848 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 390.2 | 98.2 | 3,915 ms | 10.2 ms | 4.0 | 0 | 11,704 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 130.8 | 132.3 | 4,060 ms | 7.6 ms | 0.9 | 1 | 3,588 | 27.4 s |
| Sustained decode | 1019K | 2 | Valid | 223.4 | 114.4 | 5,684 ms | 8.7 ms | 2.0 | 1 | 6,701 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.