Speed test matrix
GLM-5.3-Flash-NVFP4 on H200 SXM 141GB ×4
vllm · 0.28.1rc1.dev359+g98ed0856f · NVFP4+FP88 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.28.1rc1.dev359+g98ed0856f
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
98ed0856f31f
Run
- Served model
- glm
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-glm53flash-vllm-pr-53906/bin/vllm" serve RedHatAI/GLM-5.3-Flash-NVFP4 --served-model-name glm --tensor-parallel-size 4 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --no-enable-flashinfer-autotune
Prefill
prompt tok/s by context length8K context17.8K462 ms TTFT
16K context19.1K849 ms TTFT
32K context19.8K1,637 ms TTFT
64K context20.1K3,213 ms TTFT
128K context19.4K6,631 ms TTFT
256K context18.6K13,860 ms TTFT
512K context17K30,246 ms TTFT
1019K context14.2K72,222 ms TTFT
1024K context14.1K73,039 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 129.676 ms TTFT | 233.8109 ms TTFT | 418.6148 ms TTFT | 706.6377 ms TTFT | 1,213.5610 ms TTFT | 1,885.8565 ms TTFT | 3,062.61,047 ms TTFT | 4,635.31,338 ms TTFT |
| 16K | 129.292 ms TTFT | 231.9147 ms TTFT | 413.4168 ms TTFT | 700.7239 ms TTFT | 1,206.7368 ms TTFT | 1,882.5571 ms TTFT | 3,064.9963 ms TTFT | 4,622.81,698 ms TTFT |
| 32K | 130.5122 ms TTFT | 234.8220 ms TTFT | 421.1231 ms TTFT | 707.2311 ms TTFT | 1,208.9572 ms TTFT | 1,869.5947 ms TTFT | 3,031.91,686 ms TTFT | 4,596.83,235 ms TTFT |
| 64K | 129.8164 ms TTFT | 233.3265 ms TTFT | 418.0339 ms TTFT | 698.5512 ms TTFT | 1,201.2901 ms TTFT | 1,851.81,682 ms TTFT | 3,005.23,235 ms TTFT | Over capacity |
| 128K | 127.3449 ms TTFT | 228.7610 ms TTFT | 400.9839 ms TTFT | 656.31,319 ms TTFT | 1,169.12,360 ms TTFT | 1,774.04,443 ms TTFT | Over capacity | Over capacity |
| 256K | 125.1816 ms TTFT | 225.91,122 ms TTFT | 393.11,647 ms TTFT | 635.32,730 ms TTFT | 1,092.14,823 ms TTFT | Over capacity | Over capacity | Over capacity |
| 512K | 127.81,615 ms TTFT | 225.52,336 ms TTFT | 391.73,623 ms TTFT | 625.06,108 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 126.93,942 ms TTFT | 222.65,672 ms TTFT | 384.38,412 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 17.8K | — | 462 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 19.1K | — | 849 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 19.8K | — | 1,637 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 20.1K | — | 3,213 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 19.4K | — | 6,631 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 18.6K | — | 13,860 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 17K | — | 30,246 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 14.2K | — | 72,222 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 14.1K | — | 73,039 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 129.6 | 130.0 | 76 ms | 7.7 ms | 1.0 | 1 | 3,887 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 233.8 | 116.8 | 109 ms | 8.6 ms | 2.0 | 0 | 7,013 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 418.6 | 104.8 | 148 ms | 9.5 ms | 4.0 | 0 | 12,555 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 706.6 | 88.4 | 377 ms | 11.3 ms | 8.0 | 0 | 21,192 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,213.5 | 76.1 | 610 ms | 13.1 ms | 16.0 | 0 | 36,401 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,885.8 | 57.9 | 565 ms | 17.3 ms | 32.0 | 0 | 56,573 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,062.6 | 47.8 | 1,047 ms | 20.9 ms | 64.0 | 0 | 91,835 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,635.3 | 35.9 | 1,338 ms | 27.9 ms | 128.0 | 0 | 139,132 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 129.2 | 129.8 | 92 ms | 7.7 ms | 1.0 | 1 | 3,875 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 231.9 | 115.9 | 147 ms | 8.6 ms | 2.0 | 0 | 6,955 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 413.4 | 103.4 | 168 ms | 9.7 ms | 4.0 | 0 | 12,399 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 700.7 | 87.6 | 239 ms | 11.4 ms | 8.0 | 0 | 21,015 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,206.7 | 75.3 | 368 ms | 13.3 ms | 16.0 | 0 | 36,192 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,882.5 | 58.6 | 571 ms | 17.1 ms | 32.0 | 0 | 56,476 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 3,064.9 | 47.4 | 963 ms | 21.1 ms | 64.0 | 0 | 91,901 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 4,622.8 | 35.4 | 1,698 ms | 28.2 ms | 128.0 | 0 | 138,751 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 130.5 | 130.6 | 122 ms | 7.7 ms | 1.0 | 1 | 3,914 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 234.8 | 117.3 | 220 ms | 8.5 ms | 2.0 | 0 | 7,042 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 421.1 | 104.9 | 231 ms | 9.5 ms | 4.0 | 0 | 12,632 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 707.2 | 88.1 | 311 ms | 11.4 ms | 8.0 | 0 | 21,208 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,208.9 | 75.1 | 572 ms | 13.3 ms | 16.0 | 0 | 36,255 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,869.5 | 57.7 | 947 ms | 17.3 ms | 32.0 | 0 | 56,063 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 3,031.9 | 46.4 | 1,686 ms | 21.6 ms | 64.0 | 0 | 90,943 | 30.0 s |
| Sustained decode | 32K | 128 | Valid | 4,596.8 | 34.5 | 3,235 ms | 28.9 ms | 128.0 | 0 | 137,855 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 129.8 | 131.0 | 164 ms | 7.6 ms | 1.0 | 1 | 3,894 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 233.3 | 116.8 | 265 ms | 8.6 ms | 2.0 | 0 | 7,000 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 418.0 | 104.4 | 339 ms | 9.6 ms | 4.0 | 0 | 12,536 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 698.5 | 87.3 | 512 ms | 11.5 ms | 8.0 | 0 | 20,954 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,201.2 | 75.0 | 901 ms | 13.3 ms | 16.0 | 0 | 36,033 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,851.8 | 57.8 | 1,682 ms | 17.3 ms | 32.0 | 0 | 55,551 | 30.0 s |
| Sustained decode | 64K | 64 | Valid | 3,005.2 | 47.2 | 3,235 ms | 21.2 ms | 64.0 | 0 | 90,109 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 127.3 | 131.9 | 449 ms | 7.6 ms | 1.0 | 1 | 3,818 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 228.7 | 114.3 | 610 ms | 8.8 ms | 2.0 | 0 | 6,860 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 400.9 | 99.8 | 839 ms | 10.0 ms | 4.0 | 0 | 12,027 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 656.3 | 81.0 | 1,319 ms | 12.3 ms | 8.0 | 0 | 19,688 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,169.1 | 70.8 | 2,360 ms | 14.1 ms | 16.0 | 0 | 35,073 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,774.0 | 52.5 | 4,443 ms | 19.0 ms | 32.0 | 0 | 53,215 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 125.1 | 133.1 | 816 ms | 7.5 ms | 1.0 | 1 | 3,752 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 225.9 | 113.1 | 1,122 ms | 8.8 ms | 2.0 | 0 | 6,776 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 393.1 | 97.9 | 1,647 ms | 10.2 ms | 4.0 | 0 | 11,791 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 635.3 | 78.5 | 2,730 ms | 12.7 ms | 8.0 | 0 | 19,055 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,092.1 | 66.3 | 4,823 ms | 15.1 ms | 16.0 | 0 | 32,752 | 30.0 s |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 127.8 | 128.5 | 1,615 ms | 7.8 ms | 1.0 | 1 | 3,683 | 28.8 s |
| Sustained decode | 512K | 2 | Valid | 225.5 | 113.5 | 2,336 ms | 8.8 ms | 2.0 | 0 | 6,766 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 391.7 | 98.4 | 3,623 ms | 10.2 ms | 4.0 | 0 | 11,752 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 625.0 | 78.5 | 6,108 ms | 12.7 ms | 8.0 | 0 | 18,744 | 30.0 s |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 126.9 | 128.4 | 3,942 ms | 7.8 ms | 0.9 | 1 | 3,586 | 28.3 s |
| Sustained decode | 1019K | 2 | Valid | 222.6 | 112.0 | 5,672 ms | 8.9 ms | 2.0 | 1 | 6,678 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 384.3 | 97.1 | 8,412 ms | 10.3 ms | 4.0 | 1 | 11,528 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.