Speed test matrix
GLM-5.3-Flash-NVFP4 on H200 SXM 141GB ×4
vllm · 0.1.dev1+g3b5189828 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 SXM
- GPU count
- 4
- VRAM
- 564 GB (4 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g3b5189828
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
3b518982806c
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-nvidia-glm53flash-dflash2-vllm-3b5189828/bin/vllm" serve nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --tensor-parallel-size 4 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.93 --host 0.0.0.0 --port 8000 --api-key <redacted> --data-parallel-size 1 --enable-expert-parallel --enable-ep-weight-filter --kv-cache-dtype fp8 --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --max-num-batched-tokens 8192 --enable-chunked-prefill --max-num-seqs 32 --attention-backend FLASHINFER_MLA_SPARSE_SM90 --linear-backend humming --moe-backend humming --speculative-config '{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7,"kv_cache_dtype":"auto"}'Prefill
prompt tok/s by context length8K context16.7K492 ms TTFT
16K context18K900 ms TTFT
32K context18.4K1,757 ms TTFT
64K context18.3K3,534 ms TTFT
128K context17.9K7,191 ms TTFT
256K context17.2K14,973 ms TTFT
512K context15.8K32,619 ms TTFT
1019K context13.3K77,018 ms TTFT
1024K context13.2K77,912 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 224.91,222 ms TTFT | 365.8332 ms TTFT | 543.4347 ms TTFT | 903.51,178 ms TTFT | 1,433.52,116 ms TTFT | 2,214.43,706 ms TTFT | 1,755.1Requests queued34,586 ms TTFT | 1,735.2Requests queued35,163 ms TTFT |
| 16K | 216.6282 ms TTFT | 376.9244 ms TTFT | 634.1243 ms TTFT | 915.0993 ms TTFT | 1,453.21,784 ms TTFT | 2,163.02,797 ms TTFT | 1,884.0Requests queued30,288 ms TTFT | 1,884.2Requests queued33,094 ms TTFT |
| 32K | 215.7241 ms TTFT | 374.3277 ms TTFT | 594.8275 ms TTFT | 918.2808 ms TTFT | 1,415.71,793 ms TTFT | 2,174.72,831 ms TTFT | 1,892.7Requests queued32,655 ms TTFT | 1,949.7Requests queued31,898 ms TTFT |
| 64K | 222.2424 ms TTFT | 382.5450 ms TTFT | 569.6454 ms TTFT | 935.61,467 ms TTFT | 1,382.62,637 ms TTFT | 2,166.64,681 ms TTFT | 1,681.4Requests queued69,217 ms TTFT | 1,639.3Requests queued72,078 ms TTFT |
| 128K | 214.4561 ms TTFT | 350.9589 ms TTFT | 518.2598 ms TTFT | 896.61,770 ms TTFT | 1,389.12,927 ms TTFT | 2,053.35,134 ms TTFT | 1,927.3Requests queued72,203 ms TTFT | Over capacity |
| 256K | 202.9866 ms TTFT | 359.2938 ms TTFT | 550.2989 ms TTFT | 908.92,621 ms TTFT | 1,357.14,401 ms TTFT | 1,990.88,562 ms TTFT | Over capacity | Over capacity |
| 512K | 213.81,666 ms TTFT | 356.62,159 ms TTFT | 513.6Requests queued1,872 ms TTFT | 808.45,622 ms TTFT | 1,233.510,011 ms TTFT | Over capacity | Over capacity | Over capacity |
| 1019K | 183.93,351 ms TTFT | 315.1Requests queued3,489 ms TTFT | 459.8Requests queued5,872 ms TTFT | 754.78,390 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.7K | — | 492 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 18K | — | 900 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 18.4K | — | 1,757 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 18.3K | — | 3,534 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 17.9K | — | 7,191 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 17.2K | — | 14,973 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 15.8K | — | 32,619 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 13.3K | — | 77,018 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 13.2K | — | 77,912 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 224.9 | 228.6 | 1,222 ms | 4.4 ms | 1.0 | 1 | 6,747 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 365.8 | 193.2 | 332 ms | 5.2 ms | 2.0 | 2 | 10,972 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 543.4 | 148.7 | 347 ms | 6.7 ms | 4.0 | 4 | 16,294 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 903.5 | 114.1 | 1,178 ms | 8.8 ms | 8.0 | 0 | 27,094 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,433.5 | 87.6 | 2,116 ms | 11.4 ms | 16.0 | 0 | 42,998 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,214.4 | 64.8 | 3,706 ms | 15.4 ms | 32.0 | 0 | 66,423 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,755.1 | 61.6 | 34,586 ms | 16.2 ms | 31.9 | 32 | 52,602 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,735.2 | 61.0 | 35,163 ms | 16.4 ms | 32.0 | 32 | 52,019 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 216.6 | 221.4 | 282 ms | 4.5 ms | 1.0 | 1 | 6,497 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 376.9 | 190.1 | 244 ms | 5.3 ms | 2.0 | 2 | 11,307 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 634.1 | 162.3 | 243 ms | 6.2 ms | 4.0 | 4 | 19,016 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 915.0 | 115.3 | 993 ms | 8.7 ms | 8.0 | 2 | 27,433 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,453.2 | 90.3 | 1,784 ms | 11.1 ms | 16.0 | 0 | 43,568 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 2,163.0 | 66.1 | 2,797 ms | 15.1 ms | 32.0 | 0 | 64,854 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,884.0 | 64.1 | 30,288 ms | 15.6 ms | 31.9 | 32 | 56,519 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,884.2 | 63.5 | 33,094 ms | 15.8 ms | 31.9 | 32 | 56,500 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 215.7 | 223.6 | 241 ms | 4.5 ms | 1.0 | 1 | 6,470 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 374.3 | 193.3 | 277 ms | 5.2 ms | 2.0 | 2 | 11,226 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 594.8 | 158.4 | 275 ms | 6.3 ms | 4.0 | 4 | 17,836 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 918.2 | 117.5 | 808 ms | 8.5 ms | 8.0 | 2 | 27,537 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,415.7 | 88.9 | 1,793 ms | 11.3 ms | 16.0 | 0 | 42,454 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 2,174.7 | 66.6 | 2,831 ms | 15.0 ms | 32.0 | 0 | 65,210 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,892.7 | 65.0 | 32,655 ms | 15.4 ms | 31.9 | 32 | 56,782 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 1,949.7 | 64.9 | 31,898 ms | 15.4 ms | 31.9 | 32 | 58,478 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 222.2 | 228.7 | 424 ms | 4.4 ms | 1.0 | 1 | 6,664 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 382.5 | 199.7 | 450 ms | 5.0 ms | 2.0 | 2 | 11,472 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 569.6 | 150.3 | 454 ms | 6.7 ms | 4.0 | 4 | 17,079 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 935.6 | 117.0 | 1,467 ms | 8.5 ms | 8.0 | 2 | 28,053 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 1,382.6 | 84.9 | 2,637 ms | 11.8 ms | 16.0 | 0 | 41,454 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 2,166.6 | 63.3 | 4,681 ms | 15.8 ms | 32.0 | 0 | 64,964 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 1,681.4 | 57.7 | 69,217 ms | 17.3 ms | 32.0 | 54 | 49,992 | 29.7 s |
| Sustained decode | 64K | 128 | Requests queued | 1,639.3 | 57.2 | 72,078 ms | 17.5 ms | 32.0 | 56 | 49,168 | 30.0 s |
| Sustained decode | 128K | 1 | Valid | 214.4 | 223.3 | 561 ms | 4.5 ms | 1.0 | 1 | 6,432 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 350.9 | 185.2 | 589 ms | 5.4 ms | 2.0 | 2 | 10,526 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 518.2 | 139.0 | 598 ms | 7.2 ms | 4.0 | 3 | 15,549 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 896.6 | 111.7 | 1,770 ms | 9.0 ms | 8.0 | 2 | 26,797 | 29.9 s |
| Sustained decode | 128K | 16 | Valid | 1,389.1 | 84.5 | 2,927 ms | 11.8 ms | 16.0 | 0 | 41,660 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 2,053.3 | 60.1 | 5,134 ms | 16.6 ms | 32.0 | 0 | 61,531 | 30.0 s |
| Sustained decode | 128K | 64 | Requests queued | 1,927.3 | 57.5 | 72,203 ms | 17.4 ms | 32.0 | 65 | 57,787 | 30.0 s |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 202.9 | 214.4 | 866 ms | 4.7 ms | 1.0 | 1 | 6,086 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 359.2 | 185.7 | 938 ms | 5.4 ms | 2.0 | 2 | 10,775 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 550.2 | 147.5 | 989 ms | 6.8 ms | 3.9 | 4 | 16,498 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 908.9 | 113.4 | 2,621 ms | 8.8 ms | 8.0 | 2 | 27,250 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 1,357.1 | 82.6 | 4,401 ms | 12.1 ms | 16.0 | 0 | 40,674 | 30.0 s |
| Sustained decode | 256K | 32 | Valid | 1,990.8 | 58.3 | 8,562 ms | 17.2 ms | 32.0 | 0 | 59,651 | 30.0 s |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 213.8 | 231.7 | 1,666 ms | 4.3 ms | 1.0 | 1 | 6,413 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 356.6 | 194.9 | 2,159 ms | 5.1 ms | 1.9 | 2 | 10,697 | 30.0 s |
| Sustained decode | 512K | 4 | Requests queued | 513.6 | 143.2 | 1,872 ms | 7.0 ms | 3.8 | 4 | 15,399 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 808.4 | 103.1 | 5,622 ms | 9.7 ms | 8.0 | 1 | 24,253 | 30.0 s |
| Sustained decode | 512K | 16 | Valid | 1,233.5 | 78.6 | 10,011 ms | 12.7 ms | 16.0 | 0 | 36,974 | 30.0 s |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 183.9 | 208.3 | 3,351 ms | 4.8 ms | 0.9 | 1 | 5,515 | 30.0 s |
| Sustained decode | 1019K | 2 | Requests queued | 315.1 | 186.6 | 3,489 ms | 5.4 ms | 1.8 | 2 | 9,449 | 30.0 s |
| Sustained decode | 1019K | 4 | Requests queued | 459.8 | 145.9 | 5,872 ms | 6.9 ms | 3.6 | 4 | 13,794 | 30.0 s |
| Sustained decode | 1019K | 8 | Valid | 754.7 | 102.8 | 8,390 ms | 9.7 ms | 7.7 | 5 | 22,582 | 29.9 s |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.