Speed test matrix

Muse-Glimmer-30B-FP8-block on RTX PRO 6000 Blackwell Server Edition 96GB

vllm · 0.30.1rc1.dev452+g73a583112 · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Oct 11, 2026, 09:21 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev452+g73a583112
Quantization: FP8
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 73a5831127a9

Run

Served model: muse-glimmer
Revision: main
Benchmark tool: llm-inference-bench v0.7.6
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/Muse-Glimmer-30B-FP8-block --served-model-name muse-glimmer --tensor-parallel-size 1 --reasoning-parser muse_glimmer --enable-auto-tool-choice --tool-call-parser muse_glimmer --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config auto

Prefill

prompt tok/s by context length

8K context8,0641,016 ms TTFT

16K context7,9422,037 ms TTFT

32K context9,0633,550 ms TTFT

64K context8,4437,599 ms TTFT

123K context7,35916,728 ms TTFT

128K context7,41117,286 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	42.4	65.9145 ms TTFT	147.0173 ms TTFT	264.5233 ms TTFT	472.6267 ms TTFT	898.3399 ms TTFT	1,389.5668 ms TTFT	1,813.51,166 ms TTFT
16K	42.3128 ms TTFT	65.2183 ms TTFT	152.0192 ms TTFT	277.1248 ms TTFT	500.0358 ms TTFT	855.2589 ms TTFT	1,337.11,008 ms TTFT	Over capacity
32K	42.1155 ms TTFT	62.3171 ms TTFT	130.6257 ms TTFT	224.5319 ms TTFT	481.2617 ms TTFT	824.0948 ms TTFT	1,254.61,763 ms TTFT	Over capacity
64K	41.4	65.3262 ms TTFT	126.4321 ms TTFT	253.2490 ms TTFT	448.71,204 ms TTFT	756.82,161 ms TTFT	Over capacity	Over capacity
123K	40.5336 ms TTFT	66.6588 ms TTFT	115.8637 ms TTFT	238.5957 ms TTFT	428.81,636 ms TTFT	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	8,064.0	—	1,016 ms	—	—	1	—	—
Prefill	16K	1	Valid	7,942.0	—	2,037 ms	—	—	1	—	—
Prefill	32K	1	Valid	9,063.0	—	3,550 ms	—	—	1	—	—
Prefill	64K	1	Valid	8,443.0	—	7,599 ms	—	—	1	—	—
Prefill	123K	1	Valid	7,359.0	—	16,728 ms	—	—	1	—	—
Prefill	128K	1	Valid	7,411.0	—	17,286 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	42.4	—	—	—	1.0	0	864	20.4 s
Sustained decode	8K	2	Valid	65.9	42.5	145 ms	23.6 ms	2.0	0	1,976	30.0 s
Sustained decode	8K	4	Valid	147.0	42.4	173 ms	23.6 ms	4.0	0	4,410	30.0 s
Sustained decode	8K	8	Valid	264.5	38.9	233 ms	25.7 ms	8.0	0	7,933	30.0 s
Sustained decode	8K	16	Valid	472.6	35.6	267 ms	28.1 ms	16.0	0	14,179	30.0 s
Sustained decode	8K	32	Valid	898.3	28.6	399 ms	34.9 ms	32.0	0	26,945	30.0 s
Sustained decode	8K	64	Valid	1,389.5	21.8	668 ms	45.9 ms	64.0	0	41,662	30.0 s
Sustained decode	8K	128	Valid	1,813.5	14.2	1,166 ms	70.2 ms	128.0	0	54,399	30.0 s
Sustained decode	16K	1	Valid	42.3	42.4	128 ms	23.6 ms	1.0	0	1,268	30.0 s
Sustained decode	16K	2	Valid	65.2	42.3	183 ms	23.7 ms	2.0	0	1,953	30.0 s
Sustained decode	16K	4	Valid	152.0	42.1	192 ms	23.7 ms	4.0	0	4,557	30.0 s
Sustained decode	16K	8	Valid	277.1	38.6	248 ms	25.9 ms	8.0	0	8,311	30.0 s
Sustained decode	16K	16	Valid	500.0	35.0	358 ms	28.6 ms	16.0	0	14,999	30.0 s
Sustained decode	16K	32	Valid	855.2	27.9	589 ms	35.8 ms	32.0	0	25,639	30.0 s
Sustained decode	16K	64	Valid	1,337.1	20.9	1,008 ms	47.8 ms	64.0	0	40,112	30.0 s
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	42.1	42.1	155 ms	23.7 ms	1.0	0	1,262	30.0 s
Sustained decode	32K	2	Valid	62.3	41.9	171 ms	23.8 ms	2.0	0	1,869	30.0 s
Sustained decode	32K	4	Valid	130.6	41.7	257 ms	24.0 ms	4.0	0	3,918	30.0 s
Sustained decode	32K	8	Valid	224.5	37.9	319 ms	26.4 ms	8.0	0	6,733	30.0 s
Sustained decode	32K	16	Valid	481.2	33.9	617 ms	29.5 ms	16.0	0	14,435	30.0 s
Sustained decode	32K	32	Valid	824.0	26.5	948 ms	37.8 ms	32.0	0	24,718	30.0 s
Sustained decode	32K	64	Valid	1,254.6	19.4	1,763 ms	51.6 ms	64.0	0	37,632	30.0 s
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	41.4	—	—	—	1.0	0	910	22.0 s
Sustained decode	64K	2	Valid	65.3	41.3	262 ms	24.2 ms	2.0	0	1,960	30.0 s
Sustained decode	64K	4	Valid	126.4	40.8	321 ms	24.5 ms	4.0	0	3,791	30.0 s
Sustained decode	64K	8	Valid	253.2	36.5	490 ms	27.4 ms	8.0	0	7,590	30.0 s
Sustained decode	64K	16	Valid	448.7	32.0	1,204 ms	31.3 ms	16.0	0	13,461	30.0 s
Sustained decode	64K	32	Valid	756.8	24.1	2,161 ms	41.6 ms	32.0	0	22,697	30.0 s
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	40.5	40.5	336 ms	24.7 ms	1.0	0	1,214	30.0 s
Sustained decode	123K	2	Valid	66.6	40.1	588 ms	24.9 ms	2.0	0	1,997	30.0 s
Sustained decode	123K	4	Valid	115.8	39.2	637 ms	25.5 ms	4.0	0	3,471	30.0 s
Sustained decode	123K	8	Valid	238.5	34.2	957 ms	29.2 ms	8.0	0	7,156	30.0 s
Sustained decode	123K	16	Valid	428.8	28.5	1,636 ms	35.1 ms	16.0	0	12,857	30.0 s
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.