Speed test matrix
GLM-5.3-MXFP4 on H200 SXM 141GB ×8
vllm · 0.29.0 · MXFP47 contexts (8K–507K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 507K, 512K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 8
- VRAM
- 1128 GB (8 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- MXFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- glm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-glm53-vllm-v0290/bin/vllm" serve RedHatAI/GLM-5.3-MXFP4 --served-model-name glm --tensor-parallel-size 8 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 524288 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype bfloat16 --no-enable-flashinfer-autotune
Prefill
prompt tok/s by context length8K context8,942917 ms TTFT
16K context8,9001,823 ms TTFT
32K context8,7323,701 ms TTFT
64K context8,3697,709 ms TTFT
128K context7,70016,739 ms TTFT
256K context6,54539,364 ms TTFT
507K context5,037101,265 ms TTFT
512K context5,026102,486 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 95.450 ms TTFT | 180.370 ms TTFT | 326.288 ms TTFT | 559.6130 ms TTFT | 938.8208 ms TTFT | 1,459.2325 ms TTFT | Over capacity | Over capacity |
| 16K | 95.272 ms TTFT | 180.4102 ms TTFT | 326.0147 ms TTFT | 558.5216 ms TTFT | 934.6319 ms TTFT | 1,445.7543 ms TTFT | Over capacity | Over capacity |
| 32K | 94.9113 ms TTFT | 179.4153 ms TTFT | 323.8201 ms TTFT | 560.1322 ms TTFT | 924.0559 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 94.4193 ms TTFT | 178.7259 ms TTFT | 321.7360 ms TTFT | 553.3593 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 128K | 93.5371 ms TTFT | 176.4489 ms TTFT | 317.6708 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 256K | 91.9738 ms TTFT | 173.1997 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 507K | 89.11,769 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 8,942.0 | — | 917 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,900.0 | — | 1,823 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 8,732.0 | — | 3,701 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 8,369.0 | — | 7,709 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 7,700.0 | — | 16,739 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 6,545.0 | — | 39,364 ms | — | — | 1 | — | — |
| Prefill | 507K | 1 | Valid | 5,037.0 | — | 101,265 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 5,026.0 | — | 102,486 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 95.4 | 95.3 | 50 ms | 10.5 ms | 1.0 | 0 | 2,861 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 180.3 | 90.2 | 70 ms | 11.1 ms | 2.0 | 0 | 5,410 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 326.2 | 81.6 | 88 ms | 12.2 ms | 4.0 | 0 | 9,784 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 559.6 | 70.1 | 130 ms | 14.3 ms | 8.0 | 0 | 16,784 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 938.8 | 58.8 | 208 ms | 17.0 ms | 16.0 | 0 | 28,156 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,459.2 | 45.7 | 325 ms | 21.9 ms | 32.0 | 0 | 43,773 | 30.0 s |
| Sustained decode | 8K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 95.2 | 95.2 | 72 ms | 10.5 ms | 1.0 | 0 | 2,854 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 180.4 | 90.2 | 102 ms | 11.1 ms | 2.0 | 0 | 5,412 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 326.0 | 81.5 | 147 ms | 12.3 ms | 4.0 | 0 | 9,776 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 558.5 | 70.0 | 216 ms | 14.3 ms | 8.0 | 0 | 16,751 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 934.6 | 58.4 | 319 ms | 17.1 ms | 16.0 | 0 | 28,032 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,445.7 | 45.1 | 543 ms | 22.2 ms | 32.0 | 0 | 43,360 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 94.9 | 95.0 | 113 ms | 10.5 ms | 1.0 | 0 | 2,847 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 179.4 | 89.7 | 153 ms | 11.1 ms | 2.0 | 0 | 5,382 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 323.8 | 81.0 | 201 ms | 12.4 ms | 4.0 | 0 | 9,712 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 560.1 | 69.9 | 322 ms | 14.3 ms | 8.0 | 0 | 16,800 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 924.0 | 57.6 | 559 ms | 17.4 ms | 16.0 | 0 | 27,712 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 94.4 | 94.5 | 193 ms | 10.6 ms | 1.0 | 0 | 2,833 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 178.7 | 89.4 | 259 ms | 11.2 ms | 2.0 | 0 | 5,359 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 321.7 | 80.4 | 360 ms | 12.4 ms | 4.0 | 0 | 9,651 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 553.3 | 69.0 | 593 ms | 14.5 ms | 8.0 | 0 | 16,600 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 93.5 | 93.6 | 371 ms | 10.7 ms | 1.0 | 0 | 2,804 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 176.4 | 88.3 | 489 ms | 11.3 ms | 2.0 | 0 | 5,290 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 317.6 | 79.4 | 708 ms | 12.6 ms | 4.0 | 0 | 9,529 | 30.0 s |
| Sustained decode | 128K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 91.9 | 92.2 | 738 ms | 10.8 ms | 1.0 | 0 | 2,758 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 173.1 | 86.7 | 997 ms | 11.5 ms | 2.0 | 0 | 5,192 | 30.0 s |
| Sustained decode | 256K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 1 | Valid | 89.1 | 89.6 | 1,769 ms | 11.2 ms | 1.0 | 0 | 2,673 | 30.0 s |
| Sustained decode | 507K | 2 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 507K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.