Speed test matrix

GLM-5.3-Flash-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×4

vllm · 0.30.1rc1.dev162+g2617fe938 · NVFP48 contexts (8K–1019K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 128K, 256K, 512K, 1019K, 1024K

by @keennaylmx dev importMeasured Sep 29, 2026, 17:31 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
GPU count: 4
VRAM: 384 GB (4 × 96 GB)
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev162+g2617fe938
Quantization: NVFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 2617fe938355

Run

Served model: nvidia/GLM-5.3-Flash-NVFP4
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env VLLM_USE_V2_MODEL_RUNNER=1 VLLM_ENGINE_READY_TIMEOUT_S=3600 VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS=fp4_gemm,trtllm::fused_moe::gemm1,trtllm::fused_moe::gemm2 "$HOME/env_rtxpro6k-glm53-vllm-2617fe938/bin/vllm" serve nvidia/GLM-5.3-Flash-NVFP4 --served-model-name nvidia/GLM-5.3-Flash-NVFP4 --tensor-parallel-size 4 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 1048576 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --data-parallel-size 1 --enable-expert-parallel --enable-ep-weight-filter --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":128}' --max-num-batched-tokens 8192 --enable-chunked-prefill --max-num-seqs 32 --attention-backend FLASHINFER_MLA_SPARSE_SM120 --linear-backend flashinfer_cutlass --moe-backend flashinfer_cutlass

Prefill

prompt tok/s by context length

8K context11.1K740 ms TTFT

16K context11.9K1,367 ms TTFT

32K context9,5293,392 ms TTFT

64K context12.5K5,176 ms TTFT

128K context12.4K10,436 ms TTFT

256K context11.5K22,459 ms TTFT

512K context10.9K47,373 ms TTFT

1019K context9,301110,212 ms TTFT

1024K context9,187112,119 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	103.5146 ms TTFT	172.3214 ms TTFT	307.81,491 ms TTFT	487.5761 ms TTFT	758.41,359 ms TTFT	1,116.21,968 ms TTFT	1,084.4Requests queued2,026 ms TTFT	1,065.8Requests queued2,083 ms TTFT
16K	103.3107 ms TTFT	171.3202 ms TTFT	308.2228 ms TTFT	491.51,526 ms TTFT	755.6854 ms TTFT	1,121.81,047 ms TTFT	1,078.8Requests queued561 ms TTFT	1,085.0Requests queued665 ms TTFT
32K	103.2139 ms TTFT	172.7204 ms TTFT	308.3275 ms TTFT	488.8358 ms TTFT	755.5502 ms TTFT	1,106.7896 ms TTFT	1,072.5Requests queued844 ms TTFT	Over capacity
64K	103.0121 ms TTFT	171.8176 ms TTFT	306.6253 ms TTFT	490.2423 ms TTFT	752.0782 ms TTFT	1,115.41,541 ms TTFT	Over capacity	Over capacity
128K	102.3429 ms TTFT	170.3568 ms TTFT	303.3788 ms TTFT	483.71,223 ms TTFT	744.72,109 ms TTFT	Over capacity	Over capacity	Over capacity
256K	101.9709 ms TTFT	170.4932 ms TTFT	303.31,357 ms TTFT	482.22,136 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
512K	101.21,401 ms TTFT	168.81,956 ms TTFT	300.22,971 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
1019K	99.72,967 ms TTFT	165.94,114 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	11.1K	—	740 ms	—	—	1	—	—
Prefill	16K	1	Valid	11.9K	—	1,367 ms	—	—	1	—	—
Prefill	32K	1	Valid	9,529.0	—	3,392 ms	—	—	1	—	—
Prefill	64K	1	Valid	12.5K	—	5,176 ms	—	—	1	—	—
Prefill	128K	1	Valid	12.4K	—	10,436 ms	—	—	1	—	—
Prefill	256K	1	Valid	11.5K	—	22,459 ms	—	—	1	—	—
Prefill	512K	1	Valid	10.9K	—	47,373 ms	—	—	1	—	—
Prefill	1019K	1	Valid	9,301.0	—	110,212 ms	—	—	1	—	—
Prefill	1024K	1	Valid	9,187.0	—	112,119 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	103.5	103.5	146 ms	9.7 ms	1.0	0	3,105	30.0 s
Sustained decode	8K	2	Valid	172.3	86.0	214 ms	11.6 ms	2.0	0	5,168	30.0 s
Sustained decode	8K	4	Valid	307.8	77.0	1,491 ms	13.0 ms	4.0	0	9,232	30.0 s
Sustained decode	8K	8	Valid	487.5	60.8	761 ms	16.4 ms	8.0	0	14,625	30.0 s
Sustained decode	8K	16	Valid	758.4	47.2	1,359 ms	21.2 ms	16.0	0	22,753	30.0 s
Sustained decode	8K	32	Valid	1,116.2	33.8	1,968 ms	29.6 ms	32.0	0	33,466	30.0 s
Sustained decode	8K	64	Requests queued	1,084.4	33.8	2,026 ms	29.6 ms	32.0	0	32,513	30.0 s
Sustained decode	8K	128	Requests queued	1,065.8	33.7	2,083 ms	29.7 ms	32.0	0	31,967	30.0 s
Sustained decode	16K	1	Valid	103.3	103.3	107 ms	9.7 ms	1.0	0	3,100	30.0 s
Sustained decode	16K	2	Valid	171.3	85.7	202 ms	11.7 ms	2.0	0	5,138	30.0 s
Sustained decode	16K	4	Valid	308.2	77.2	228 ms	13.0 ms	4.0	0	9,245	30.0 s
Sustained decode	16K	8	Valid	491.5	61.5	1,526 ms	16.3 ms	8.0	0	14,744	30.0 s
Sustained decode	16K	16	Valid	755.6	46.5	854 ms	21.5 ms	16.0	0	22,654	30.0 s
Sustained decode	16K	32	Valid	1,121.8	33.5	1,047 ms	29.8 ms	32.0	0	33,632	30.0 s
Sustained decode	16K	64	Requests queued	1,078.8	34.2	561 ms	29.2 ms	32.0	0	32,353	30.0 s
Sustained decode	16K	128	Requests queued	1,085.0	33.8	665 ms	29.6 ms	32.0	0	32,543	30.0 s
Sustained decode	32K	1	Valid	103.2	103.1	139 ms	9.7 ms	1.0	0	3,094	30.0 s
Sustained decode	32K	2	Valid	172.7	86.3	204 ms	11.6 ms	2.0	0	5,181	30.0 s
Sustained decode	32K	4	Valid	308.3	77.1	275 ms	13.0 ms	4.0	0	9,248	30.0 s
Sustained decode	32K	8	Valid	488.8	61.2	358 ms	16.3 ms	8.0	0	14,663	30.0 s
Sustained decode	32K	16	Valid	755.5	47.1	502 ms	21.3 ms	16.0	0	22,655	30.0 s
Sustained decode	32K	32	Valid	1,106.7	34.4	896 ms	29.1 ms	32.0	0	33,182	30.0 s
Sustained decode	32K	64	Requests queued	1,072.5	33.8	844 ms	29.5 ms	32.0	0	32,164	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	103.0	103.1	121 ms	9.7 ms	1.0	0	3,090	30.0 s
Sustained decode	64K	2	Valid	171.8	85.8	176 ms	11.6 ms	2.0	0	5,152	30.0 s
Sustained decode	64K	4	Valid	306.6	76.7	253 ms	13.0 ms	4.0	0	9,198	30.0 s
Sustained decode	64K	8	Valid	490.2	61.3	423 ms	16.3 ms	8.0	0	14,704	30.0 s
Sustained decode	64K	16	Valid	752.0	47.0	782 ms	21.3 ms	16.0	0	22,560	30.0 s
Sustained decode	64K	32	Valid	1,115.4	34.8	1,541 ms	28.8 ms	32.0	0	33,440	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	1	Valid	102.3	102.4	429 ms	9.8 ms	1.0	0	3,068	30.0 s
Sustained decode	128K	2	Valid	170.3	85.0	568 ms	11.8 ms	2.0	0	5,110	30.0 s
Sustained decode	128K	4	Valid	303.3	75.4	788 ms	13.3 ms	4.0	0	9,095	30.0 s
Sustained decode	128K	8	Valid	483.7	59.4	1,223 ms	16.8 ms	8.0	0	14,505	30.0 s
Sustained decode	128K	16	Valid	744.7	44.9	2,109 ms	22.3 ms	16.0	0	22,337	30.0 s
Sustained decode	128K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	128K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	1	Valid	101.9	102.1	709 ms	9.8 ms	1.0	0	3,057	30.0 s
Sustained decode	256K	2	Valid	170.4	85.1	932 ms	11.7 ms	2.0	0	5,112	30.0 s
Sustained decode	256K	4	Valid	303.3	75.3	1,357 ms	13.3 ms	4.0	0	9,096	30.0 s
Sustained decode	256K	8	Valid	482.2	59.2	2,136 ms	16.9 ms	8.0	0	14,464	30.0 s
Sustained decode	256K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	256K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	1	Valid	101.2	101.6	1,401 ms	9.8 ms	1.0	0	3,036	30.0 s
Sustained decode	512K	2	Valid	168.8	84.6	1,956 ms	11.8 ms	2.0	0	5,062	30.0 s
Sustained decode	512K	4	Valid	300.2	74.9	2,971 ms	13.3 ms	4.0	0	9,004	30.0 s
Sustained decode	512K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	512K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	1	Valid	99.7	100.4	2,967 ms	10.0 ms	1.0	0	2,990	30.0 s
Sustained decode	1019K	2	Valid	165.9	83.5	4,114 ms	12.0 ms	2.0	0	4,975	30.0 s
Sustained decode	1019K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	1019K	128	Over capacity	—	—	—	—	—	—	—	—

Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.