Speed test matrix

Muse-Glimmer-30B on RTX PRO 6000 Blackwell Server Edition 96GB

vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Oct 11, 2026, 12:27 UTC

Hardware

Type: Discrete GPU
GPU: RTX PRO 6000 Blackwell Server Edition
VRAM: 96 GB
OS: linux

Engine

Engine: vllm
Version: 0.30.1rc1.dev452+g73a583112
Quantization: BF16
Backend: cuda
Repository: github.com/vllm-project/vllm
Build: cuda-13.2
Commit: 73a5831127a9

Run

Served model: muse-glimmer
Revision: main
Benchmark tool: llm-inference-bench v0.7.6
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/Muse-Glimmer-30B --served-model-name muse-glimmer --tensor-parallel-size 1 --reasoning-parser muse_glimmer --enable-auto-tool-choice --tool-call-parser muse_glimmer --max-model-len 131072 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config auto

Prefill

prompt tok/s by context length

8K context5,3911,520 ms TTFT

16K context6,3822,535 ms TTFT

32K context6,3515,066 ms TTFT

64K context5,75411,149 ms TTFT

123K context5,28523,293 ms TTFT

128K context5,24724,416 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	24.7184 ms TTFT	49.3209 ms TTFT	98.3233 ms TTFT	187.0288 ms TTFT	357.2348 ms TTFT	612.7469 ms TTFT	1,024.1734 ms TTFT	Over capacity
16K	24.6197 ms TTFT	48.2283 ms TTFT	97.5287 ms TTFT	185.9337 ms TTFT	353.2444 ms TTFT	602.2647 ms TTFT	Over capacity	Over capacity
32K	24.5224 ms TTFT	49.0277 ms TTFT	97.5332 ms TTFT	183.9462 ms TTFT	346.0652 ms TTFT	Over capacity	Over capacity	Over capacity
64K	24.3281 ms TTFT	48.5348 ms TTFT	91.5412 ms TTFT	178.7623 ms TTFT	332.71,123 ms TTFT	Over capacity	Over capacity	Over capacity
123K	24.0405 ms TTFT	45.2	87.3788 ms TTFT	172.21,098 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	5,391.0	—	1,520 ms	—	—	1	—	—
Prefill	16K	1	Valid	6,382.0	—	2,535 ms	—	—	1	—	—
Prefill	32K	1	Valid	6,351.0	—	5,066 ms	—	—	1	—	—
Prefill	64K	1	Valid	5,754.0	—	11,149 ms	—	—	1	—	—
Prefill	123K	1	Valid	5,285.0	—	23,293 ms	—	—	1	—	—
Prefill	128K	1	Valid	5,247.0	—	24,416 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	24.7	24.7	184 ms	40.4 ms	1.0	0	740	30.0 s
Sustained decode	8K	2	Valid	49.3	24.7	209 ms	40.5 ms	2.0	0	1,476	30.0 s
Sustained decode	8K	4	Valid	98.3	24.7	233 ms	40.5 ms	4.0	0	2,949	30.0 s
Sustained decode	8K	8	Valid	187.0	23.5	288 ms	42.6 ms	8.0	0	5,608	30.0 s
Sustained decode	8K	16	Valid	357.2	22.4	348 ms	44.7 ms	16.0	0	10,704	30.0 s
Sustained decode	8K	32	Valid	612.7	19.2	469 ms	52.0 ms	32.0	0	18,381	30.0 s
Sustained decode	8K	64	Valid	1,024.1	16.1	734 ms	62.1 ms	64.0	0	30,723	30.0 s
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	24.6	24.7	197 ms	40.5 ms	1.0	0	739	30.0 s
Sustained decode	16K	2	Valid	48.2	24.7	283 ms	40.6 ms	2.0	0	1,445	30.0 s
Sustained decode	16K	4	Valid	97.5	24.6	287 ms	40.6 ms	4.0	0	2,924	30.0 s
Sustained decode	16K	8	Valid	185.9	23.3	337 ms	42.9 ms	8.0	0	5,576	30.0 s
Sustained decode	16K	16	Valid	353.2	22.2	444 ms	45.1 ms	16.0	0	10,593	30.0 s
Sustained decode	16K	32	Valid	602.2	18.9	647 ms	52.9 ms	32.0	0	18,047	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	24.5	24.6	224 ms	40.6 ms	1.0	0	735	30.0 s
Sustained decode	32K	2	Valid	49.0	24.6	277 ms	40.7 ms	2.0	0	1,468	30.0 s
Sustained decode	32K	4	Valid	97.5	24.5	332 ms	40.9 ms	4.0	0	2,924	30.0 s
Sustained decode	32K	8	Valid	183.9	23.1	462 ms	43.3 ms	8.0	0	5,518	30.0 s
Sustained decode	32K	16	Valid	346.0	21.7	652 ms	46.1 ms	16.0	0	10,369	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	24.3	24.4	281 ms	41.0 ms	1.0	0	729	30.0 s
Sustained decode	64K	2	Valid	48.5	24.3	348 ms	41.1 ms	2.0	0	1,456	30.0 s
Sustained decode	64K	4	Valid	91.5	24.1	412 ms	41.4 ms	4.0	0	2,744	30.0 s
Sustained decode	64K	8	Valid	178.7	22.5	623 ms	44.4 ms	8.0	0	5,355	30.0 s
Sustained decode	64K	16	Valid	332.7	20.8	1,123 ms	48.0 ms	16.0	0	9,968	30.0 s
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	24.0	24.1	405 ms	41.6 ms	1.0	0	719	30.0 s
Sustained decode	123K	2	Valid	45.2	—	—	—	2.0	0	1,103	24.4 s
Sustained decode	123K	4	Valid	87.3	23.6	788 ms	42.4 ms	4.0	0	2,618	30.0 s
Sustained decode	123K	8	Valid	172.2	21.7	1,098 ms	46.1 ms	8.0	0	5,163	30.0 s
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache.