Speed test matrix
GLM-5.3-Flash-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4
vllm · 0.32.1rc1.dev96+g8c60cf07b · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- GPU count
- 4
- VRAM
- 384 GB (4 × 96 GB)
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.32.1rc1.dev96+g8c60cf07b
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
8c60cf07b3df
Run
- Served model
- nvidia/GLM-5.3-Flash-NVFP4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS=fp4_gemm,trtllm::fused_moe::gemm1,trtllm::fused_moe::gemm2 "$HOME/env_rtxpro6k-nvidia-glm53flash-dflash2-vllm-8c60cf07b/bin/vllm" serve nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --tensor-parallel-size 4 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.92 --host 0.0.0.0 --port 8000 --api-key <redacted> --data-parallel-size 1 --enable-expert-parallel --enable-ep-weight-filter --kv-cache-dtype fp8 --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --max-num-batched-tokens 8192 --enable-chunked-prefill --max-num-seqs 32 --attention-backend FLASHINFER_MLA_SPARSE_SM120 --linear-backend flashinfer_cutlass --moe-backend flashinfer_cutlass --speculative-config '{"method":"dflash","model":"incoai/GLM-5.3-Flash-DFlash2","num_speculative_tokens":7,"kv_cache_dtype":"auto"}'Prefill
prompt tok/s by context length8K context12K682 ms TTFT
16K context8,1741,985 ms TTFT
32K context13.2K2,442 ms TTFT
64K context13.3K4,845 ms TTFT
128K context13.1K9,807 ms TTFT
256K context12.6K20,395 ms TTFT
512K context11.5K44,966 ms TTFT
1019K context9,641106,315 ms TTFT
1024K context9,596107,335 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 179.7746 ms TTFT | 242.6346 ms TTFT | 397.41,669 ms TTFT | 599.91,497 ms TTFT | 946.92,766 ms TTFT | 1,335.74,917 ms TTFT | 1,217.8Requests queued4,940 ms TTFT | 1,207.7Requests queued5,009 ms TTFT |
| 16K | 159.1276 ms TTFT | 247.1280 ms TTFT | 389.0823 ms TTFT | 606.21,397 ms TTFT | 975.32,357 ms TTFT | 1,330.73,716 ms TTFT | 1,224.5Requests queued3,773 ms TTFT | 1,240.8Requests queued3,828 ms TTFT |
| 32K | 176.1268 ms TTFT | 249.6300 ms TTFT | 376.2844 ms TTFT | 619.61,428 ms TTFT | 931.42,390 ms TTFT | 1,333.33,753 ms TTFT | 1,228.7Requests queued4,054 ms TTFT | Over capacity |
| 64K | 155.3465 ms TTFT | 250.0503 ms TTFT | 394.41,364 ms TTFT | 599.32,150 ms TTFT | 937.73,498 ms TTFT | 1,318.56,451 ms TTFT | Over capacity | Over capacity |
| 128K | 129.6581 ms TTFT | 241.8612 ms TTFT | 376.21,338 ms TTFT | 582.82,371 ms TTFT | 893.73,752 ms TTFT | Over capacity | Over capacity | Over capacity |
| 256K | 161.61,272 ms TTFT | 255.81,023 ms TTFT | 380.51,913 ms TTFT | 566.83,075 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 512K | 136.61,383 ms TTFT | 240.91,798 ms TTFT | 368.13,388 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
| 1019K | 148.52,739 ms TTFT | 229.84,257 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 12K | — | 682 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 8,174.0 | — | 1,985 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 13.2K | — | 2,442 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.3K | — | 4,845 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 13.1K | — | 9,807 ms | — | — | 1 | — | — |
| Prefill | 256K | 1 | Valid | 12.6K | — | 20,395 ms | — | — | 1 | — | — |
| Prefill | 512K | 1 | Valid | 11.5K | — | 44,966 ms | — | — | 1 | — | — |
| Prefill | 1019K | 1 | Valid | 9,641.0 | — | 106,315 ms | — | — | 1 | — | — |
| Prefill | 1024K | 1 | Valid | 9,596.0 | — | 107,335 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 179.7 | 184.7 | 746 ms | 5.4 ms | 1.0 | 1 | 5,391 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 242.6 | 134.8 | 346 ms | 7.4 ms | 2.0 | 2 | 7,275 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 397.4 | 100.9 | 1,669 ms | 9.9 ms | 4.0 | 0 | 11,916 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 599.9 | 73.8 | 1,497 ms | 13.6 ms | 8.0 | 0 | 17,993 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 946.9 | 58.5 | 2,766 ms | 17.1 ms | 16.0 | 0 | 28,406 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,335.7 | 38.2 | 4,917 ms | 26.2 ms | 32.0 | 0 | 40,011 | 30.0 s |
| Sustained decode | 8K | 64 | Requests queued | 1,217.8 | 38.9 | 4,940 ms | 25.7 ms | 32.0 | 0 | 36,464 | 29.9 s |
| Sustained decode | 8K | 128 | Requests queued | 1,207.7 | 39.0 | 5,009 ms | 25.7 ms | 32.0 | 0 | 36,186 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 159.1 | 169.0 | 276 ms | 5.9 ms | 1.0 | 1 | 4,771 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 247.1 | 128.3 | 280 ms | 7.8 ms | 2.0 | 1 | 7,412 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 389.0 | 100.5 | 823 ms | 9.9 ms | 4.0 | 0 | 11,661 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 606.2 | 78.4 | 1,397 ms | 12.8 ms | 8.0 | 0 | 18,184 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 975.3 | 60.9 | 2,357 ms | 16.4 ms | 16.0 | 0 | 29,218 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,330.7 | 40.5 | 3,716 ms | 24.7 ms | 32.0 | 0 | 39,844 | 29.9 s |
| Sustained decode | 16K | 64 | Requests queued | 1,224.5 | 39.9 | 3,773 ms | 25.1 ms | 32.0 | 0 | 36,734 | 30.0 s |
| Sustained decode | 16K | 128 | Requests queued | 1,240.8 | 39.6 | 3,828 ms | 25.2 ms | 32.0 | 0 | 37,215 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 176.1 | 183.3 | 268 ms | 5.5 ms | 1.0 | 1 | 5,281 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 249.6 | 145.1 | 300 ms | 7.0 ms | 2.0 | 2 | 7,486 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 376.2 | 96.0 | 844 ms | 10.4 ms | 4.0 | 0 | 11,280 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 619.6 | 79.1 | 1,428 ms | 12.6 ms | 8.0 | 0 | 18,573 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 931.4 | 58.0 | 2,390 ms | 17.2 ms | 16.0 | 0 | 27,904 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,333.3 | 40.4 | 3,753 ms | 24.7 ms | 32.0 | 0 | 39,930 | 29.9 s |
| Sustained decode | 32K | 64 | Requests queued | 1,228.7 | 39.6 | 4,054 ms | 25.2 ms | 32.0 | 1 | 36,830 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 155.3 | 169.6 | 465 ms | 5.9 ms | 1.0 | 1 | 4,659 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 250.0 | 134.0 | 503 ms | 7.5 ms | 2.0 | 2 | 7,497 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 394.4 | 100.3 | 1,364 ms | 10.0 ms | 4.0 | 0 | 11,825 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 599.3 | 73.5 | 2,150 ms | 13.6 ms | 8.0 | 0 | 17,975 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 937.7 | 56.8 | 3,498 ms | 17.6 ms | 16.0 | 0 | 28,096 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 1,318.5 | 37.5 | 6,451 ms | 26.6 ms | 32.0 | 0 | 39,509 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 1 | Valid | 129.6 | 156.7 | 581 ms | 6.5 ms | 1.0 | 1 | 3,888 | 30.0 s |
| Sustained decode | 128K | 2 | Valid | 241.8 | 132.5 | 612 ms | 7.5 ms | 2.0 | 1 | 7,252 | 30.0 s |
| Sustained decode | 128K | 4 | Valid | 376.2 | 96.6 | 1,338 ms | 10.4 ms | 4.0 | 0 | 11,278 | 30.0 s |
| Sustained decode | 128K | 8 | Valid | 582.8 | 71.7 | 2,371 ms | 13.9 ms | 8.0 | 0 | 17,482 | 30.0 s |
| Sustained decode | 128K | 16 | Valid | 893.7 | 52.9 | 3,752 ms | 18.9 ms | 16.0 | 0 | 26,774 | 30.0 s |
| Sustained decode | 128K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 128K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 1 | Valid | 161.6 | 170.9 | 1,272 ms | 5.9 ms | 1.0 | 1 | 4,847 | 30.0 s |
| Sustained decode | 256K | 2 | Valid | 255.8 | 140.5 | 1,023 ms | 7.1 ms | 2.0 | 1 | 7,674 | 30.0 s |
| Sustained decode | 256K | 4 | Valid | 380.5 | 95.9 | 1,913 ms | 10.4 ms | 4.0 | 0 | 11,412 | 30.0 s |
| Sustained decode | 256K | 8 | Valid | 566.8 | 71.4 | 3,075 ms | 14.0 ms | 8.0 | 0 | 16,986 | 30.0 s |
| Sustained decode | 256K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 256K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 1 | Valid | 136.6 | 180.6 | 1,383 ms | 5.8 ms | 1.0 | 1 | 4,096 | 30.0 s |
| Sustained decode | 512K | 2 | Valid | 240.9 | 133.7 | 1,798 ms | 7.5 ms | 1.9 | 2 | 7,223 | 30.0 s |
| Sustained decode | 512K | 4 | Valid | 368.1 | 93.9 | 3,388 ms | 10.7 ms | 4.0 | 0 | 11,038 | 30.0 s |
| Sustained decode | 512K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 512K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 1 | Valid | 148.5 | 168.4 | 2,739 ms | 5.9 ms | 0.9 | 1 | 4,453 | 30.0 s |
| Sustained decode | 1019K | 2 | Valid | 229.8 | 120.3 | 4,257 ms | 8.3 ms | 2.0 | 1 | 6,893 | 30.0 s |
| Sustained decode | 1019K | 4 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 1019K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.