Speed test matrix
GLM-5.3-Flash-NVFP4 on H200 NVL 141GB ×2
vllm · 0.1.dev1+g3b5189828 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g3b5189828
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
3b518982806c
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-nvidia-glm53flash-dflash2-vllm-3b5189828/bin/vllm" serve nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --tensor-parallel-size 2 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.94 --host 0.0.0.0 --port 8000 --api-key <redacted> --data-parallel-size 1 --enable-expert-parallel --enable-ep-weight-filter --kv-cache-dtype fp8 --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --max-num-batched-tokens 8192 --enable-chunked-prefill --max-num-seqs 32 --attention-backend FLASHINFER_MLA_SPARSE_SM90 --linear-backend humming --moe-backend hummingPrefill
prompt tok/s by context length8K context12.4K662 ms TTFT
16K context13K1,245 ms TTFT
32K context12.9K2,512 ms TTFT
64K context12.7K5,086 ms TTFT
128K context12.5K10,351 ms TTFT
256K context12K21,412 ms TTFT
512K context11.3K45,657 ms TTFT
1019K context9,839104,172 ms TTFT
1024K context9,820104,894 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 99.9330 ms TTFT | 177.6494 ms TTFT | 317.2816 ms TTFT | 526.01,476 ms TTFT | 805.02,816 ms TTFT | 1,234.25,533 ms TTFT | 1,197.1Requests queued5,568 ms TTFT | 1,204.6Requests queued5,581 ms TTFT |
| 16K | 99.8283 ms TTFT | 177.3436 ms TTFT | 313.2679 ms TTFT | 528.81,221 ms TTFT | 800.02,322 ms TTFT | 1,231.74,569 ms TTFT | 1,207.0Requests queued4,613 ms TTFT | 1,224.1Requests queued4,642 ms TTFT |
| 32K | 99.5234 ms TTFT | 176.1355 ms TTFT | 313.4510 ms TTFT | 528.7885 ms TTFT | 807.61,667 ms TTFT | 1,226.83,281 ms TTFT | 1,194.8Requests queued3,306 ms TTFT | 1,186.0Requests queued3,382 ms TTFT |
| 64K | 99.3415 ms TTFT | 176.7624 ms TTFT | 310.3888 ms TTFT | 525.81,536 ms TTFT | 798.22,854 ms TTFT | 1,217.15,557 ms TTFT | 905.9Requests queued64,582 ms TTFT | Over capacity |
| 128K | 99.4439 ms TTFT | 175.4611 ms TTFT | 312.3827 ms TTFT | 523.91,344 ms TTFT | 803.72,356 ms TTFT | 1,207.24,486 ms TTFT | Over capacity | Over capacity |
| 256K | 99.1577 ms TTFT | 174.4798 ms TTFT | 310.41,092 ms TTFT | 520.31,770 ms TTFT | 793.73,081 ms TTFT | Over capacity | Over capacity | Over capacity |
| 512K | 98.61,242 ms TTFT | 174.51,709 ms TTFT | 309.72,328 ms TTFT | 511.23,905 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 97.72,357 ms TTFT | 173.33,333 ms TTFT | 307.45,109 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12.4K | — | 662 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13K | — | 1,245 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.9K | — | 2,512 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.7K | — | 5,086 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.5K | — | 10,351 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12K | — | 21,412 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 11.3K | — | 45,657 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,839.0 | — | 104,172 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,820.0 | — | 104,894 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 99.9 | 99.9 | 330 ms | 10.0 ms | 1.0 | 0 | 2,997 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 177.6 | 88.3 | 494 ms | 11.3 ms | 2.0 | 0 | 5,325 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 317.2 | 78.1 | 816 ms | 12.8 ms | 4.0 | 0 | 9,516 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 526.0 | 63.8 | 1,476 ms | 15.7 ms | 8.0 | 0 | 15,776 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 805.0 | 47.2 | 2,816 ms | 21.2 ms | 16.0 | 0 | 24,145 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,234.2 | 34.1 | 5,533 ms | 29.4 ms | 32.0 | 0 | 37,020 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,197.1 | 35.8 | 5,568 ms | 28.0 ms | 32.0 | 0 | 35,904 | 30.0 s |
| Sustained decode | 8K | 128 | Requests queued | 1,204.6 | 36.1 | 5,581 ms | 27.7 ms | 32.0 | 0 | 36,125 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 99.8 | 99.8 | 283 ms | 10.0 ms | 1.0 | 0 | 2,994 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 177.3 | 88.5 | 436 ms | 11.3 ms | 2.0 | 0 | 5,318 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 313.2 | 77.5 | 679 ms | 12.9 ms | 4.0 | 0 | 9,392 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 528.8 | 64.5 | 1,221 ms | 15.5 ms | 8.0 | 0 | 15,856 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 800.0 | 47.5 | 2,322 ms | 21.1 ms | 16.0 | 0 | 24,000 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,231.7 | 34.8 | 4,569 ms | 28.8 ms | 32.0 | 0 | 36,924 | 30.0 s |
| Sustained decode | 16K | 64 | Requests queued | 1,207.0 | 36.3 | 4,613 ms | 27.5 ms | 32.0 | 0 | 36,191 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,224.1 | 36.7 | 4,642 ms | 27.3 ms | 32.0 | 0 | 36,703 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 99.5 | 99.5 | 234 ms | 10.0 ms | 1.0 | 0 | 2,984 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 176.1 | 87.9 | 355 ms | 11.4 ms | 2.0 | 0 | 5,284 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 313.4 | 77.9 | 510 ms | 12.8 ms | 4.0 | 0 | 9,400 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 528.7 | 65.0 | 885 ms | 15.4 ms | 8.0 | 0 | 15,859 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 807.6 | 48.7 | 1,667 ms | 20.5 ms | 16.0 | 0 | 24,223 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,226.8 | 35.8 | 3,281 ms | 27.9 ms | 32.0 | 0 | 36,800 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 1,194.8 | 36.9 | 3,306 ms | 27.1 ms | 32.0 | 0 | 35,839 | 30.0 s |
| Sustained decode | 32K | 128 | Requests queued | 1,186.0 | 36.6 | 3,382 ms | 27.3 ms | 32.0 | 0 | 35,554 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 99.3 | 99.3 | 415 ms | 10.1 ms | 1.0 | 0 | 2,978 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 176.7 | 88.0 | 624 ms | 11.4 ms | 2.0 | 0 | 5,300 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 310.3 | 76.6 | 888 ms | 13.1 ms | 4.0 | 0 | 9,306 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 525.8 | 63.8 | 1,536 ms | 15.7 ms | 8.0 | 0 | 15,768 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 798.2 | 46.9 | 2,854 ms | 21.3 ms | 16.0 | 0 | 23,936 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,217.1 | 33.7 | 5,557 ms | 29.7 ms | 32.0 | 0 | 36,512 | 30.0 s |
| Sustained decode | 64K | 64 | Requests queued | 905.9 | 34.8 | 64,582 ms | 28.8 ms | 32.0 | 32 | 27,174 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 99.4 | 99.5 | 439 ms | 10.0 ms | 1.0 | 0 | 2,983 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 175.4 | 87.6 | 611 ms | 11.4 ms | 2.0 | 0 | 5,262 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 312.3 | 77.3 | 827 ms | 12.9 ms | 4.0 | 0 | 9,368 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 523.9 | 64.0 | 1,344 ms | 15.6 ms | 8.0 | 0 | 15,711 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 803.7 | 47.9 | 2,356 ms | 20.9 ms | 16.0 | 0 | 24,111 | 30.0 s |
| Sustained decode | 128K | 32 | Valid | 1,207.2 | 34.4 | 4,486 ms | 29.0 ms | 32.0 | 0 | 36,191 | 30.0 s |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 99.1 | 99.3 | 577 ms | 10.1 ms | 1.0 | 0 | 2,972 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 174.4 | 87.2 | 798 ms | 11.5 ms | 2.0 | 0 | 5,230 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 310.4 | 77.3 | 1,092 ms | 12.9 ms | 4.0 | 0 | 9,312 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 520.3 | 64.2 | 1,770 ms | 15.6 ms | 8.0 | 0 | 15,608 | 30.0 s |
| Sustained decode | 256K | 16 | Valid | 793.7 | 48.4 | 3,081 ms | 20.6 ms | 16.0 | 0 | 23,807 | 30.0 s |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 98.6 | 99.0 | 1,242 ms | 10.1 ms | 1.0 | 0 | 2,957 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 174.5 | 87.4 | 1,709 ms | 11.4 ms | 2.0 | 0 | 5,233 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 309.7 | 77.0 | 2,328 ms | 13.0 ms | 4.0 | 0 | 9,292 | 30.0 s |
| Sustained decode | 512K | 8 | Valid | 511.2 | 63.0 | 3,905 ms | 15.9 ms | 8.0 | 0 | 15,337 | 30.0 s |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 97.7 | 98.5 | 2,357 ms | 10.2 ms | 1.0 | 0 | 2,929 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 173.3 | 87.0 | 3,333 ms | 11.5 ms | 2.0 | 0 | 5,196 | 30.0 s |
| Sustained decode | 1019K | 4 | Valid | 307.4 | 76.2 | 5,109 ms | 13.1 ms | 4.0 | 0 | 9,220 | 30.0 s |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.