Speed test matrix
GLM-5.3-Flash-NVFP4 on H200 SXM 141GB ×4
vllm · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
24b07cd3b099
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS=fp4_gemm,trtllm::fused_moe::gemm1,trtllm::fused_moe::gemm2 "$HOME/env_h200-nvidia-vllm-pr-55222/bin/vllm" serve nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --tensor-parallel-size 4 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.88 --host 0.0.0.0 --port 8000 --api-key <redacted> --data-parallel-size 1 --enable-expert-parallel --enable-ep-weight-filter --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --max-num-batched-tokens 8192 --enable-chunked-prefill --max-num-seqs 32Prefill
prompt tok/s by context length8K context14K588 ms TTFT
16K context13.7K1,182 ms TTFT
32K context14.3K2,265 ms TTFT
64K context3,90416,525 ms TTFT
128K context6,97518,478 ms TTFT
256K context11.2K23,061 ms TTFT
512K context13K39,599 ms TTFT
1019K context11.4K90,266 ms TTFT
1024K context9,733105,831 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 122.8177 ms TTFT | 224.98,027 ms TTFT | 398.3540 ms TTFT | 661.1652 ms TTFT | 1,152.61,091 ms TTFT | 1,798.91,989 ms TTFT | 1,588.7Requests queued40,481 ms TTFT | 1,592.0Requests queued40,632 ms TTFT |
| 16K | 122.3187 ms TTFT | 224.4294 ms TTFT | 398.212,599 ms TTFT | 654.2704 ms TTFT | 1,143.0986 ms TTFT | 1,782.41,528 ms TTFT | 1,111.4Requests queued43,988 ms TTFT | 1,656.9Requests queued39,427 ms TTFT |
| 32K | 123.5231 ms TTFT | 226.7360 ms TTFT | 399.2668 ms TTFT | 665.8740 ms TTFT | 1,146.41,341 ms TTFT | 1,773.82,306 ms TTFT | 1,560.7Requests queued40,908 ms TTFT | 1,566.3Requests queued41,122 ms TTFT |
| 64K | 119.9321 ms TTFT | 226.0435 ms TTFT | 399.9619 ms TTFT | 654.4956 ms TTFT | 1,130.31,416 ms TTFT | 1,746.32,463 ms TTFT | 1,736.9Requests queued41,431 ms TTFT | 1,722.9Requests queued41,573 ms TTFT |
| 128K | 113.5519 ms TTFT | 222.1684 ms TTFT | 385.9881 ms TTFT | 619.71,388 ms TTFT | 1,115.72,449 ms TTFT | 1,723.94,587 ms TTFT | 1,725.4Requests queued83,136 ms TTFT | Over capacity |
| 256K | 113.8825 ms TTFT | 219.81,152 ms TTFT | 380.61,617 ms TTFT | 601.62,659 ms TTFT | 1,038.34,748 ms TTFT | 1,542.78,853 ms TTFT | Over capacity | Over capacity |
| 512K | 121.31,711 ms TTFT | 218.92,673 ms TTFT | 378.63,719 ms TTFT | 590.86,621 ms TTFT | 1,018.211,713 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | 120.53,859 ms TTFT | 215.85,684 ms TTFT | 371.78,565 ms TTFT | 577.214,722 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14K | — | 588 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.7K | — | 1,182 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.3K | — | 2,265 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,904.0 | — | 16,525 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 6,975.0 | — | 18,478 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 11.2K | — | 23,061 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 13K | — | 39,599 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 11.4K | — | 90,266 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,733.0 | — | 105,831 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 122.8 | 134.3 | 177 ms | 7.5 ms | 1.0 | 1 | 3,684 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 224.9 | 94.7 | 8,027 ms | 10.9 ms | 2.0 | 0 | 6,746 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 398.3 | 99.7 | 540 ms | 10.0 ms | 4.0 | 0 | 11,945 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 661.1 | 81.9 | 652 ms | 12.2 ms | 8.0 | 0 | 19,831 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,152.6 | 70.5 | 1,091 ms | 14.2 ms | 16.0 | 0 | 34,577 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,798.9 | 53.8 | 1,989 ms | 18.6 ms | 32.0 | 0 | 53,951 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,588.7 | 53.7 | 40,481 ms | 18.6 ms | 31.6 | 32 | 47,649 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,592.0 | 53.7 | 40,632 ms | 18.6 ms | 31.6 | 32 | 47,742 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 122.3 | 122.4 | 187 ms | 8.2 ms | 1.0 | 0 | 3,668 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 224.4 | 112.0 | 294 ms | 8.9 ms | 2.0 | 0 | 6,732 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 398.2 | 99.6 | 12,599 ms | 10.0 ms | 4.0 | 0 | 11,944 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 654.2 | 81.9 | 704 ms | 12.2 ms | 8.0 | 0 | 19,624 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,143.0 | 71.1 | 986 ms | 14.1 ms | 16.0 | 0 | 34,288 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,782.4 | 54.7 | 1,528 ms | 18.3 ms | 32.0 | 0 | 53,469 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,111.4 | 48.6 | 43,988 ms | 20.6 ms | 31.7 | 32 | 33,345 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,656.9 | 54.6 | 39,427 ms | 18.3 ms | 31.7 | 32 | 49,693 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 123.5 | 123.4 | 231 ms | 8.1 ms | 1.0 | 1 | 3,689 | 29.9 s |
| Sustained decode | 32K | 2 | Valid | 226.7 | 113.1 | 360 ms | 8.8 ms | 2.0 | 0 | 6,802 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 399.2 | 99.9 | 668 ms | 10.0 ms | 4.0 | 0 | 11,975 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 665.8 | 82.0 | 740 ms | 12.2 ms | 8.0 | 0 | 19,968 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,146.4 | 70.1 | 1,341 ms | 14.3 ms | 16.0 | 0 | 34,384 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,773.8 | 52.8 | 2,306 ms | 18.9 ms | 32.0 | 0 | 53,215 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,560.7 | 53.3 | 40,908 ms | 18.8 ms | 31.6 | 32 | 46,820 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 1,566.3 | 53.3 | 41,122 ms | 18.8 ms | 31.6 | 32 | 47,008 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 119.9 | 120.0 | 321 ms | 8.3 ms | 1.0 | 0 | 3,596 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 226.0 | 112.8 | 435 ms | 8.9 ms | 2.0 | 0 | 6,778 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 399.9 | 99.6 | 619 ms | 10.0 ms | 4.0 | 0 | 11,996 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 654.4 | 81.2 | 956 ms | 12.3 ms | 8.0 | 0 | 19,632 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,130.3 | 69.3 | 1,416 ms | 14.4 ms | 16.0 | 0 | 33,907 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,746.3 | 52.7 | 2,463 ms | 19.0 ms | 32.0 | 0 | 52,385 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 1,736.9 | 52.7 | 41,431 ms | 19.0 ms | 32.0 | 32 | 52,098 | 30.0 s |
| Sustained decode | 64K | 128 | Requests queued | 1,722.9 | 52.7 | 41,573 ms | 19.0 ms | 32.0 | 32 | 51,682 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 113.5 | 113.5 | 519 ms | 8.8 ms | 1.0 | 0 | 3,403 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 222.1 | 111.1 | 684 ms | 9.0 ms | 2.0 | 0 | 6,662 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 385.9 | 96.0 | 881 ms | 10.4 ms | 4.0 | 0 | 11,576 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 619.7 | 76.4 | 1,388 ms | 13.1 ms | 8.0 | 0 | 18,584 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 1,115.7 | 67.6 | 2,449 ms | 14.8 ms | 16.0 | 0 | 33,471 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,723.9 | 37.3 | 4,587 ms | 26.8 ms | 32.0 | 0 | 51,712 | 30.0 s |
| Sustained decode | 128K | 64 | Requests queued | 1,725.4 | 49.5 | 83,136 ms | 20.2 ms | 32.0 | 64 | 51,741 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 113.8 | 115.0 | 825 ms | 8.7 ms | 1.0 | 0 | 3,414 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 219.8 | 110.0 | 1,152 ms | 9.1 ms | 2.0 | 0 | 6,592 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 380.6 | 94.7 | 1,617 ms | 10.6 ms | 4.0 | 0 | 11,416 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 601.6 | 74.1 | 2,659 ms | 13.5 ms | 8.0 | 0 | 18,048 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,038.3 | 62.9 | 4,748 ms | 15.9 ms | 16.0 | 0 | 31,137 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,542.7 | 46.0 | 8,853 ms | 21.7 ms | 32.0 | 0 | 46,271 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 121.3 | 121.9 | 1,711 ms | 8.2 ms | 1.0 | 1 | 3,608 | 29.8 s |
| Sustained decode | 512K | 2 | Valid | 218.9 | 109.2 | 2,673 ms | 9.2 ms | 2.0 | 0 | 6,566 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 378.6 | 94.5 | 3,719 ms | 10.6 ms | 4.0 | 0 | 11,358 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 590.8 | 73.8 | 6,621 ms | 13.6 ms | 8.0 | 0 | 17,720 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,018.2 | 62.4 | 11,713 ms | 16.0 ms | 16.0 | 0 | 30,544 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 120.5 | 121.8 | 3,859 ms | 8.2 ms | 1.0 | 0 | 3,613 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 215.8 | 109.1 | 5,684 ms | 9.2 ms | 2.0 | 0 | 6,474 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 371.7 | 93.6 | 8,565 ms | 10.7 ms | 4.0 | 0 | 11,148 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 577.2 | 72.5 | 14,722 ms | 13.8 ms | 8.0 | 1 | 17,315 | 30.0 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.