Speed test matrix

GLM-5.3-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB ×8

vllm · 0.30.1rc1.dev167+g417b0b6aa · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 30, 2026, 11:05 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
GPU count: 8
VRAM: 768 GB (8 × 96 GB)
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev167+g417b0b6aa
Quantization: NVFP4
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 417b0b6aa289

Run

Served model: GLM-5.3-NVFP4
Revision: main
Benchmark tool: llm-inference-bench v0.6.2
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_rtxpro6k-incoai-vllm-pr-417b0b6aa/bin/vllm" serve incoai/GLM-5.3-NVFP4 --served-model-name GLM-5.3-NVFP4 --tensor-parallel-size 8 --reasoning-parser glm45 --enable-auto-tool-choice --tool-call-parser glm47 --max-model-len 131072 --gpu-memory-utilization 0.82 --host 0.0.0.0 --port 8000 --api-key <redacted> --no-enable-flashinfer-autotune --attention-backend FLASHINFER_MLA_SPARSE_SM120 --moe-backend flashinfer_cutlass --speculative-config '{"method":"dflash","model":"incoai/GLM-5.3-DFlash2","num_speculative_tokens":7}'

Prefill

prompt tok/s by context length

8K context3,6802,228 ms TTFT

16K context3,9664,091 ms TTFT

32K context4,0527,975 ms TTFT

64K context4,01716,055 ms TTFT

123K context3,83532,286 ms TTFT

128K context3,80733,847 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	82.8102 ms TTFT	147.1176 ms TTFT	209.4249 ms TTFT	277.2320 ms TTFT	486.3434 ms TTFT	Over capacity	Over capacity	Over capacity
16K	90.3113 ms TTFT	137.3170 ms TTFT	204.2285 ms TTFT	265.5365 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
32K	88.3140 ms TTFT	133.7214 ms TTFT	200.8341 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
64K	85.9182 ms TTFT	143.6264 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity
123K	93.4303 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	3,680.0	—	2,228 ms	—	—	1	—	—
Prefill	16K	1	Valid	3,966.0	—	4,091 ms	—	—	1	—	—
Prefill	32K	1	Valid	4,052.0	—	7,975 ms	—	—	1	—	—
Prefill	64K	1	Valid	4,017.0	—	16,055 ms	—	—	1	—	—
Prefill	123K	1	Valid	3,835.0	—	32,286 ms	—	—	1	—	—
Prefill	128K	1	Valid	3,807.0	—	33,847 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	82.8	85.1	102 ms	11.8 ms	1.0	0	2,485	30.0 s
Sustained decode	8K	2	Valid	147.1	71.8	176 ms	14.0 ms	2.0	0	4,411	30.0 s
Sustained decode	8K	4	Valid	209.4	52.7	249 ms	19.0 ms	4.0	0	6,273	30.0 s
Sustained decode	8K	8	Valid	277.2	34.4	320 ms	29.1 ms	8.0	0	8,296	29.9 s
Sustained decode	8K	16	Valid	486.3	30.1	434 ms	33.2 ms	16.0	0	14,568	30.0 s
Sustained decode	8K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	90.3	92.6	113 ms	10.8 ms	1.0	0	2,708	30.0 s
Sustained decode	16K	2	Valid	137.3	70.7	170 ms	14.2 ms	2.0	0	4,119	30.0 s
Sustained decode	16K	4	Valid	204.2	53.4	285 ms	18.7 ms	4.0	0	6,127	30.0 s
Sustained decode	16K	8	Valid	265.5	34.8	365 ms	28.7 ms	8.0	0	7,960	30.0 s
Sustained decode	16K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	88.3	90.2	140 ms	11.1 ms	1.0	0	2,648	30.0 s
Sustained decode	32K	2	Valid	133.7	68.4	214 ms	14.6 ms	2.0	0	4,012	30.0 s
Sustained decode	32K	4	Valid	200.8	52.4	341 ms	19.1 ms	4.0	0	6,016	30.0 s
Sustained decode	32K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	85.9	88.4	182 ms	11.3 ms	1.0	0	2,578	30.0 s
Sustained decode	64K	2	Valid	143.6	73.9	264 ms	13.5 ms	2.0	0	4,305	30.0 s
Sustained decode	64K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	93.4	95.2	303 ms	10.5 ms	1.0	0	2,800	30.0 s
Sustained decode	123K	2	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	4	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.