Speed test matrix
GLM-5.3-Flash on H200 SXM 141GB ×4
vllm · FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
374e367e2b87
Run
- Served model
- glm-5.3-flash-dflash2
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53flash-dflash2-vllm-pr-55423/bin/vllm" serve zai-org/GLM-5.3-Flash --served-model-name glm-5.3-flash-dflash2 --trust-remote-code --tensor-parallel-size 4 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.883543 --host 0.0.0.0 --port 8000 --api-key <redacted> --dtype bfloat16 --kv-cache-dtype bfloat16 --no-enable-flashinfer-autotune --speculative-config '{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7,"disable_eagle_block_drop":true}'Prefill
prompt tok/s by context length8K context18.9K433 ms TTFT
16K context21.2K764 ms TTFT
32K context22.8K1,417 ms TTFT
64K context4,76413,540 ms TTFT
128K context16.9K7,647 ms TTFT
256K context22.2K11,579 ms TTFT
512K context20.1K25,572 ms TTFT
1019K context16.3K62,712 ms TTFT
1024K context16.3K63,320 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 175.8109 ms TTFT | 345.6126 ms TTFT | 511.7176 ms TTFT | 915.611,091 ms TTFT | 1,412.4514 ms TTFT | 2,270.411,511 ms TTFT | 2,793.5Requests queued11,897 ms TTFT | 1,583.7Requests queued1,236 ms TTFT |
| 16K | 169.2127 ms TTFT | 307.5161 ms TTFT | 540.7198 ms TTFT | 883.9304 ms TTFT | 1,438.7523 ms TTFT | 2,247.3758 ms TTFT | 1,505.1Requests queued1,193 ms TTFT | Over capacity |
| 32K | 168.7185 ms TTFT | 319.2194 ms TTFT | 543.0237 ms TTFT | 869.8421 ms TTFT | 1,394.2695 ms TTFT | 2,265.51,086 ms TTFT | 2,291.4Requests queued2,033 ms TTFT | Over capacity |
| 64K | 173.4366 ms TTFT | 337.3394 ms TTFT | 476.6473 ms TTFT | 923.1969 ms TTFT | 1,419.31,388 ms TTFT | 2,231.32,053 ms TTFT | Over capacity | Over capacity |
| 128K | 172.8506 ms TTFT | 329.8523 ms TTFT | 492.8549 ms TTFT | 903.11,166 ms TTFT | 1,383.42,040 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 158.1847 ms TTFT | 302.0977 ms TTFT | 470.31,072 ms TTFT | 892.71,973 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 144.91,556 ms TTFT | 309.21,624 ms TTFT | 449.32,656 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 138.33,458 ms TTFT | 265.4Requests queued4,621 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 18.9K | — | 433 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 21.2K | — | 764 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 22.8K | — | 1,417 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 4,764.0 | — | 13,540 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 16.9K | — | 7,647 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 22.2K | — | 11,579 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 20.1K | — | 25,572 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 16.3K | — | 62,712 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 16.3K | — | 63,320 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 175.8 | 177.8 | 109 ms | 5.6 ms | 1.0 | 1 | 5,272 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 345.6 | 166.2 | 126 ms | 6.0 ms | 2.0 | 2 | 10,365 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 511.7 | 134.5 | 176 ms | 7.4 ms | 4.0 | 4 | 15,351 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 915.6 | 114.9 | 11,091 ms | 8.7 ms | 8.0 | 2 | 27,451 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,412.4 | 85.0 | 514 ms | 11.8 ms | 16.0 | 0 | 42,382 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,270.4 | 71.3 | 11,511 ms | 14.0 ms | 32.0 | 0 | 68,078 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 2,793.5 | 43.5 | 11,897 ms | 23.0 ms | 56.4 | 0 | 83,773 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,583.7 | 44.4 | 1,236 ms | 22.5 ms | 54.5 | 9 | 47,337 | 29.9 s |
| Sustained decode | 16K | 1 | Valid | 169.2 | 173.4 | 127 ms | 5.8 ms | 1.0 | 1 | 5,077 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 307.5 | 163.0 | 161 ms | 6.2 ms | 2.0 | 2 | 9,224 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 540.7 | 140.9 | 198 ms | 7.1 ms | 4.0 | 4 | 16,215 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 883.9 | 113.0 | 304 ms | 8.9 ms | 8.0 | 1 | 26,507 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,438.7 | 91.3 | 523 ms | 11.0 ms | 16.0 | 0 | 43,163 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,247.3 | 71.9 | 758 ms | 13.9 ms | 32.0 | 0 | 67,413 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,505.1 | 44.0 | 1,193 ms | 22.7 ms | 54.3 | 8 | 45,046 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 168.7 | 172.4 | 185 ms | 5.8 ms | 1.0 | 1 | 5,061 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 319.2 | 163.7 | 194 ms | 6.1 ms | 2.0 | 2 | 9,591 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 543.0 | 141.3 | 237 ms | 7.1 ms | 4.0 | 4 | 16,288 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 869.8 | 114.3 | 421 ms | 8.8 ms | 8.0 | 0 | 26,086 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,394.2 | 88.3 | 695 ms | 11.3 ms | 16.0 | 0 | 41,817 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,265.5 | 72.3 | 1,086 ms | 13.8 ms | 32.0 | 0 | 67,896 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 2,291.4 | 51.1 | 2,033 ms | 19.6 ms | 53.4 | 47 | 68,691 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 173.4 | 191.1 | 366 ms | 5.4 ms | 1.0 | 1 | 5,200 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 337.3 | 173.6 | 394 ms | 5.8 ms | 2.0 | 2 | 10,118 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 476.6 | 130.4 | 473 ms | 7.7 ms | 4.0 | 2 | 14,296 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 923.1 | 116.2 | 969 ms | 8.6 ms | 8.0 | 1 | 27,690 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,419.3 | 89.0 | 1,388 ms | 11.2 ms | 16.0 | 0 | 42,577 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,231.3 | 69.1 | 2,053 ms | 14.5 ms | 32.0 | 0 | 66,902 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 172.8 | 181.6 | 506 ms | 5.5 ms | 1.0 | 1 | 5,186 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 329.8 | 170.9 | 523 ms | 5.9 ms | 2.0 | 2 | 9,904 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 492.8 | 130.9 | 549 ms | 7.6 ms | 4.0 | 4 | 14,782 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 903.1 | 115.0 | 1,166 ms | 8.7 ms | 8.0 | 1 | 27,095 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,383.4 | 85.3 | 2,040 ms | 11.7 ms | 16.0 | 0 | 41,505 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 158.1 | 163.3 | 847 ms | 6.1 ms | 1.0 | 1 | 4,743 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 302.0 | 161.8 | 977 ms | 6.2 ms | 1.9 | 2 | 9,062 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 470.3 | 125.8 | 1,072 ms | 8.0 ms | 4.0 | 2 | 14,103 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 892.7 | 113.6 | 1,973 ms | 8.8 ms | 8.0 | 3 | 26,766 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 144.9 | 170.5 | 1,556 ms | 5.9 ms | 1.0 | 1 | 4,348 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 309.2 | 171.7 | 1,624 ms | 5.8 ms | 1.9 | 2 | 9,274 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 449.3 | 119.6 | 2,656 ms | 8.4 ms | 3.9 | 2 | 13,472 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 138.3 | 177.3 | 3,458 ms | 5.7 ms | 0.9 | 1 | 4,150 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 265.4 | 183.2 | 4,621 ms | 5.5 ms | 1.8 | 2 | 7,960 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.