Speed test matrix
GLM-5.3-Flash on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev359+g98ed0856f · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev359+g98ed0856f
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
98ed0856f31f
Run
- Served model
- glm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53flash-vllm-pr-53906/bin/vllm" serve RedHatAI/GLM-5.3-Flash --served-model-name glm --tensor-parallel-size 4 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype bfloat16 --no-enable-flashinfer-autotune
Prefill
prompt tok/s by context length8K context19.4K422 ms TTFT
16K context23.4K693 ms TTFT
32K context24.3K1,328 ms TTFT
64K context24.4K2,645 ms TTFT
128K context23.6K5,465 ms TTFT
256K context22.4K11,481 ms TTFT
512K context20.1K25,608 ms TTFT
1019K context16.3K63,029 ms TTFT
1024K context16K64,210 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 132.274 ms TTFT | 234.9132 ms TTFT | 413.4158 ms TTFT | 638.6392 ms TTFT | 1,150.8643 ms TTFT | 1,833.3694 ms TTFT | 3,071.1997 ms TTFT | 4,497.71,456 ms TTFT |
| 16K | 131.990 ms TTFT | 233.9147 ms TTFT | 410.0166 ms TTFT | 638.0254 ms TTFT | 1,138.2372 ms TTFT | 1,821.1572 ms TTFT | 3,069.8947 ms TTFT | 4,509.91,671 ms TTFT |
| 32K | 132.9120 ms TTFT | 236.3187 ms TTFT | 414.6227 ms TTFT | 640.3361 ms TTFT | 1,140.8592 ms TTFT | 1,815.9956 ms TTFT | 3,054.81,720 ms TTFT | Over capacity |
| 64K | 132.6187 ms TTFT | 235.4272 ms TTFT | 411.4365 ms TTFT | 632.6593 ms TTFT | 1,134.61,045 ms TTFT | 1,808.91,967 ms TTFT | Over capacity | Over capacity |
| 128K | 130.0445 ms TTFT | 229.9637 ms TTFT | 396.3882 ms TTFT | 603.01,427 ms TTFT | 1,113.92,336 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 127.3830 ms TTFT | 228.21,120 ms TTFT | 389.41,628 ms TTFT | 581.12,710 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 123.91,646 ms TTFT | 226.62,310 ms TTFT | 386.43,618 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 129.63,925 ms TTFT | 224.15,344 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 19.4K | — | 422 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 23.4K | — | 693 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 24.3K | — | 1,328 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 24.4K | — | 2,645 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 23.6K | — | 5,465 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 22.4K | — | 11,481 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 20.1K | — | 25,608 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 16.3K | — | 63,029 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 16K | — | 64,210 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 132.2 | 132.5 | 74 ms | 7.5 ms | 1.0 | 1 | 3,964 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 234.9 | 117.4 | 132 ms | 8.5 ms | 2.0 | 0 | 7,047 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 413.4 | 103.4 | 158 ms | 9.7 ms | 4.0 | 0 | 12,401 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 638.6 | 79.7 | 392 ms | 12.5 ms | 8.0 | 0 | 19,152 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,150.8 | 72.2 | 643 ms | 13.9 ms | 16.0 | 0 | 34,511 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,833.3 | 56.9 | 694 ms | 17.6 ms | 32.0 | 0 | 54,976 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,071.1 | 48.0 | 997 ms | 20.8 ms | 64.0 | 0 | 92,093 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,497.7 | 34.9 | 1,456 ms | 28.6 ms | 128.0 | 0 | 134,907 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 131.9 | 132.6 | 90 ms | 7.5 ms | 1.0 | 1 | 3,955 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 233.9 | 116.9 | 147 ms | 8.6 ms | 2.0 | 0 | 7,016 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 410.0 | 102.4 | 166 ms | 9.8 ms | 4.0 | 0 | 12,297 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 638.0 | 79.7 | 254 ms | 12.6 ms | 8.0 | 0 | 19,137 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,138.2 | 71.1 | 372 ms | 14.1 ms | 16.0 | 0 | 34,144 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,821.1 | 56.7 | 572 ms | 17.6 ms | 32.0 | 0 | 54,623 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,069.8 | 47.6 | 947 ms | 21.0 ms | 64.0 | 0 | 92,097 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,509.9 | 34.5 | 1,671 ms | 29.0 ms | 128.0 | 0 | 135,294 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 132.9 | 133.1 | 120 ms | 7.5 ms | 1.0 | 1 | 3,988 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 236.3 | 118.0 | 187 ms | 8.5 ms | 2.0 | 0 | 7,090 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 414.6 | 103.4 | 227 ms | 9.7 ms | 4.0 | 0 | 12,436 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 640.3 | 79.8 | 361 ms | 12.5 ms | 8.0 | 0 | 19,209 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,140.8 | 71.1 | 592 ms | 14.1 ms | 16.0 | 0 | 34,226 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,815.9 | 56.1 | 956 ms | 17.8 ms | 32.0 | 0 | 54,460 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,054.8 | 46.6 | 1,720 ms | 21.5 ms | 64.0 | 0 | 91,647 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 132.6 | 133.9 | 187 ms | 7.5 ms | 1.0 | 1 | 3,977 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 235.4 | 117.7 | 272 ms | 8.5 ms | 2.0 | 0 | 7,060 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 411.4 | 102.7 | 365 ms | 9.7 ms | 4.0 | 0 | 12,340 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 632.6 | 78.8 | 593 ms | 12.7 ms | 8.0 | 0 | 18,978 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,134.6 | 70.3 | 1,045 ms | 14.2 ms | 16.0 | 0 | 34,033 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,808.9 | 55.6 | 1,967 ms | 18.0 ms | 32.0 | 0 | 54,242 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 130.0 | 133.8 | 445 ms | 7.5 ms | 1.0 | 1 | 3,901 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 229.9 | 115.0 | 637 ms | 8.7 ms | 2.0 | 0 | 6,898 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 396.3 | 98.5 | 882 ms | 10.2 ms | 4.0 | 0 | 11,885 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 603.0 | 74.2 | 1,427 ms | 13.5 ms | 8.0 | 0 | 18,088 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,113.9 | 67.0 | 2,336 ms | 14.9 ms | 16.0 | 0 | 33,408 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 127.3 | 134.3 | 830 ms | 7.4 ms | 1.0 | 1 | 3,819 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 228.2 | 114.1 | 1,120 ms | 8.8 ms | 2.0 | 0 | 6,845 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 389.4 | 96.9 | 1,628 ms | 10.3 ms | 4.0 | 0 | 11,679 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 581.1 | 71.6 | 2,710 ms | 14.0 ms | 8.0 | 0 | 17,432 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 123.9 | 240.6 | 1,646 ms | 5.2 ms | 0.9 | 1 | 3,716 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 226.6 | 114.1 | 2,310 ms | 8.8 ms | 2.0 | 0 | 6,799 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 386.4 | 97.1 | 3,618 ms | 10.3 ms | 4.0 | 0 | 11,591 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 129.6 | 131.1 | 3,925 ms | 7.6 ms | 0.9 | 1 | 3,574 | 27.6 s |
| Sustained decode | 1019K | 2 | Valid | 224.1 | 113.2 | 5,344 ms | 8.8 ms | 2.0 | 1 | 6,724 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.