Speed test matrix
GLM-5.3-Flash-NVFP4 on H200 NVL 141GB ×2
vllm · 0.1.dev1+g3b5189828 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- GPU count
- 2
- VRAM
- 282 GB (2 × 141 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.1.dev1+g3b5189828
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
3b518982806c
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 "$HOME/env_h200-nvidia-glm53flash-dflash2-vllm-3b5189828/bin/vllm" serve nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --tensor-parallel-size 2 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.94 --host 0.0.0.0 --port 8000 --api-key <redacted> --data-parallel-size 1 --enable-expert-parallel --enable-ep-weight-filter --kv-cache-dtype fp8 --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --max-num-batched-tokens 8192 --enable-chunked-prefill --max-num-seqs 32 --attention-backend FLASHINFER_MLA_SPARSE_SM90 --linear-backend humming --moe-backend humming --speculative-config '{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7,"kv_cache_dtype":"auto"}'Prefill
prompt tok/s by context length8K context12.2K671 ms TTFT
16K context13.2K1,232 ms TTFT
32K context12.7K2,544 ms TTFT
64K context12.6K5,122 ms TTFT
128K context12.4K10,392 ms TTFT
256K context12K21,477 ms TTFT
512K context11.2K45,834 ms TTFT
1019K context9,819104,385 ms TTFT
1024K context9,809105,005 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 153.9657 ms TTFT | 247.4692 ms TTFT | 406.91,645 ms TTFT | 608.72,959 ms TTFT | 893.75,628 ms TTFT | 865.7Requests queued8,671 ms TTFT | 864.5Requests queued8,693 ms TTFT | 823.1Requests queued8,719 ms TTFT |
| 16K | 158.1601 ms TTFT | 254.8894 ms TTFT | 383.21,739 ms TTFT | 626.72,775 ms TTFT | 893.14,873 ms TTFT | 948.1Requests queued7,261 ms TTFT | 932.7Requests queued7,281 ms TTFT | 942.7Requests queued7,338 ms TTFT |
| 32K | 156.0446 ms TTFT | 260.9488 ms TTFT | 406.81,394 ms TTFT | 616.62,160 ms TTFT | 898.73,708 ms TTFT | 959.4Requests queued5,477 ms TTFT | 879.4Requests queued5,513 ms TTFT | Over capacity |
| 64K | 156.5828 ms TTFT | 244.81,188 ms TTFT | 367.21,971 ms TTFT | 599.53,515 ms TTFT | 881.36,600 ms TTFT | 251.3Requests queued9,370 ms TTFT | Over capacity | Over capacity |
| 128K | 156.7947 ms TTFT | 258.51,000 ms TTFT | 399.12,127 ms TTFT | 608.73,673 ms TTFT | 866.46,817 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 157.01,202 ms TTFT | 262.61,277 ms TTFT | 389.52,426 ms TTFT | 583.84,047 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 144.31,808 ms TTFT | 254.71,909 ms TTFT | 380.53,131 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 149.12,952 ms TTFT | 235.43,091 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12.2K | — | 671 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 13.2K | — | 1,232 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 12.7K | — | 2,544 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.6K | — | 5,122 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 12.4K | — | 10,392 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12K | — | 21,477 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 11.2K | — | 45,834 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,819.0 | — | 104,385 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,809.0 | — | 105,005 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 153.9 | 163.8 | 657 ms | 6.1 ms | 1.0 | 1 | 4,616 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 247.4 | 131.7 | 692 ms | 7.6 ms | 2.0 | 2 | 7,418 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 406.9 | 102.2 | 1,645 ms | 9.8 ms | 4.0 | 0 | 12,205 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 608.7 | 72.4 | 2,959 ms | 13.8 ms | 8.0 | 0 | 18,246 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 893.7 | 50.7 | 5,628 ms | 19.7 ms | 16.0 | 0 | 26,801 | 30.0 s |
| Sustained decode | 8K | 32 | Requests queued | 865.7 | 37.6 | 8,671 ms | 26.6 ms | 23.1 | 0 | 25,950 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 864.5 | 37.7 | 8,693 ms | 26.5 ms | 23.1 | 0 | 25,679 | 29.7 s |
| Sustained decode | 8K | 128 | Requests queued | 823.1 | 35.3 | 8,719 ms | 28.3 ms | 23.1 | 0 | 24,439 | 29.7 s |
| Sustained decode | 16K | 1 | Valid | 158.1 | 162.7 | 601 ms | 6.1 ms | 1.0 | 1 | 4,742 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 254.8 | 133.0 | 894 ms | 7.5 ms | 2.0 | 2 | 7,643 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 383.2 | 94.2 | 1,739 ms | 10.6 ms | 4.0 | 0 | 11,488 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 626.7 | 76.4 | 2,775 ms | 13.1 ms | 8.0 | 0 | 18,796 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 893.1 | 51.5 | 4,873 ms | 19.4 ms | 16.0 | 0 | 26,755 | 30.0 s |
| Sustained decode | 16K | 32 | Requests queued | 948.1 | 39.0 | 7,261 ms | 25.6 ms | 23.5 | 0 | 28,393 | 29.9 s |
| Sustained decode | 16K | 64 | Requests queued | 932.7 | 39.7 | 7,281 ms | 25.2 ms | 23.3 | 0 | 27,959 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 942.7 | 38.0 | 7,338 ms | 26.3 ms | 23.4 | 0 | 28,255 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 156.0 | 166.4 | 446 ms | 6.0 ms | 1.0 | 1 | 4,680 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 260.9 | 134.8 | 488 ms | 7.4 ms | 2.0 | 2 | 7,824 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 406.8 | 102.8 | 1,394 ms | 9.7 ms | 4.0 | 0 | 12,198 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 616.6 | 75.8 | 2,160 ms | 13.2 ms | 8.0 | 0 | 18,480 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 898.7 | 53.0 | 3,708 ms | 18.9 ms | 16.0 | 0 | 26,952 | 30.0 s |
| Sustained decode | 32K | 32 | Requests queued | 959.4 | 38.9 | 5,477 ms | 25.7 ms | 25.0 | 0 | 28,749 | 30.0 s |
| Sustained decode | 32K | 64 | Requests queued | 879.4 | 37.3 | 5,513 ms | 26.8 ms | 25.0 | 0 | 26,335 | 29.9 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 156.5 | 171.7 | 828 ms | 5.9 ms | 1.0 | 1 | 4,693 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 244.8 | 129.5 | 1,188 ms | 7.7 ms | 2.0 | 2 | 7,343 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 367.2 | 91.4 | 1,971 ms | 10.9 ms | 4.0 | 0 | 11,015 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 599.5 | 71.6 | 3,515 ms | 14.0 ms | 8.0 | 0 | 17,967 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 881.3 | 48.7 | 6,600 ms | 20.5 ms | 16.0 | 0 | 26,412 | 30.0 s |
| Sustained decode | 64K | 32 | Requests queued | 251.3 | 22.0 | 9,370 ms | 45.4 ms | 23.8 | 0 | 7,476 | 29.8 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 156.7 | 166.9 | 947 ms | 6.0 ms | 1.0 | 1 | 4,700 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 258.5 | 141.5 | 1,000 ms | 7.1 ms | 2.0 | 2 | 7,752 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 399.1 | 100.2 | 2,127 ms | 10.0 ms | 4.0 | 0 | 11,963 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 608.7 | 71.9 | 3,673 ms | 13.9 ms | 8.0 | 0 | 18,254 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 866.4 | 49.9 | 6,817 ms | 20.0 ms | 16.0 | 0 | 25,982 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 157.0 | 166.1 | 1,202 ms | 6.0 ms | 1.0 | 1 | 4,710 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 262.6 | 137.7 | 1,277 ms | 7.3 ms | 2.0 | 2 | 7,876 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 389.5 | 96.6 | 2,426 ms | 10.4 ms | 4.0 | 0 | 11,682 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 583.8 | 68.7 | 4,047 ms | 14.6 ms | 8.0 | 0 | 17,498 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 144.3 | 171.6 | 1,808 ms | 5.9 ms | 1.0 | 1 | 4,329 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 254.7 | 138.7 | 1,909 ms | 7.2 ms | 2.0 | 1 | 7,638 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 380.5 | 94.7 | 3,131 ms | 10.6 ms | 4.0 | 0 | 11,414 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 149.1 | 169.6 | 2,952 ms | 5.9 ms | 0.9 | 1 | 4,472 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 235.4 | 139.0 | 3,091 ms | 7.2 ms | 1.9 | 1 | 7,057 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.