Speed test matrix

Muse-Glimmer-30B on H200 NVL 141GB

sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K

by @keennaylmx dev importMeasured Sep 19, 2026, 12:24 UTC

Hardware

Type: Discrete GPU
GPU: H200 NVL
VRAM: 141 GB
OS: linux

Engine

Engine: sglang
Version: 0.5.6.post3.dev9265+g05c584c44
Quantization: BF16
Backend: cuda
Repository: github.com/sgl-project/sglang
Build: cuda-13.2
Commit: 05c584c44fb0

Run

Served model: muse-glimmer
Revision: main
Benchmark tool: llm-inference-bench v0.4.29
Decode test: Fixed duration
Prefill mode: Measured during decode warm-up
Time per measurement: 30 s
Max output: 4,096 tokens

Launch command

"$HOME/env_h200-incoai-sglang/bin/sglang" serve --model-path meta-models/Muse-Glimmer-30B --served-model-name muse-glimmer --tp 1 --reasoning-parser muse --tool-call-parser muse --context-length 131072 --mem-fraction-static 0.858945 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-algorithm DFLASH --speculative-draft-model-path incoai/Muse-Glimmer-30B-DFlash2 --speculative-num-draft-tokens 16

Prefill

prompt tok/s by context length

8K context10.3K792 ms TTFT

16K context10.1K1,599 ms TTFT

32K context9,9083,246 ms TTFT

64K context9,3056,893 ms TTFT

123K context8,39714,657 ms TTFT

128K context8,40915,231 ms TTFT

Sustained decode

total tok/s · rows are context length, columns are simultaneous requests

Context	1	2	4	8	16	32	64	128
8K	442.096 ms TTFT	613.4138 ms TTFT	1,253.0126 ms TTFT	2,025.3149 ms TTFT	2,550.4438 ms TTFT	2,764.6742 ms TTFT	Not run	Over capacity
16K	428.8127 ms TTFT	677.7165 ms TTFT	1,305.4169 ms TTFT	2,255.5169 ms TTFT	2,274.2777 ms TTFT	3,278.51,551 ms TTFT	Over capacity	Over capacity
32K	405.1184 ms TTFT	604.2223 ms TTFT	1,554.8238 ms TTFT	2,008.5231 ms TTFT	2,526.21,793 ms TTFT	Over capacity	Over capacity	Over capacity
64K	262.2306 ms TTFT	655.5421 ms TTFT	1,233.9473 ms TTFT	1,855.2461 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity
123K	223.6530 ms TTFT	599.4544 ms TTFT	1,181.7893 ms TTFT	Over capacity	Over capacity	Over capacity	Over capacity	Over capacity

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.

TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.

All measurements

Latencies are medians. Total tok/s is output across all simultaneous requests.

Phase	Context	Requests at once	Status	Total tok/s	Per request tok/s	TTFT	Time between tokens	Average active requests	Completed requests	Output tokens	Measured for
Prefill	8K	1	Valid	10.3K	—	792 ms	—	—	1	—	—
Prefill	16K	1	Valid	10.1K	—	1,599 ms	—	—	1	—	—
Prefill	32K	1	Valid	9,908.0	—	3,246 ms	—	—	1	—	—
Prefill	64K	1	Valid	9,305.0	—	6,893 ms	—	—	1	—	—
Prefill	123K	1	Valid	8,397.0	—	14,657 ms	—	—	1	—	—
Prefill	128K	1	Valid	8,409.0	—	15,231 ms	—	—	1	—	—
Sustained decode	8K	1	Valid	442.0	430.5	96 ms	2.3 ms	1.0	3	13,259	30.0 s
Sustained decode	8K	2	Valid	613.4	278.7	138 ms	3.7 ms	2.0	4	18,393	30.0 s
Sustained decode	8K	4	Valid	1,253.0	340.8	126 ms	2.9 ms	4.0	8	37,566	30.0 s
Sustained decode	8K	8	Valid	2,025.3	269.0	149 ms	3.7 ms	8.7	13	60,758	30.0 s
Sustained decode	8K	16	Valid	2,550.4	157.3	438 ms	6.4 ms	17.4	13	76,495	30.0 s
Sustained decode	8K	32	Valid	2,764.6	82.3	742 ms	12.2 ms	38.2	1	82,762	29.9 s
Sustained decode	8K	64	Not run	—	—	—	—	—	—	—	—
Sustained decode	8K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	1	Valid	428.8	428.3	127 ms	2.3 ms	0.9	3	12,862	30.0 s
Sustained decode	16K	2	Valid	677.7	330.2	165 ms	3.2 ms	6.3	4	20,329	30.0 s
Sustained decode	16K	4	Valid	1,305.4	357.8	169 ms	2.8 ms	3.8	9	39,134	30.0 s
Sustained decode	16K	8	Valid	2,255.5	286.5	169 ms	3.5 ms	8.3	13	67,612	30.0 s
Sustained decode	16K	16	Valid	2,274.2	133.8	777 ms	7.5 ms	16.6	9	68,167	30.0 s
Sustained decode	16K	32	Valid	3,278.5	94.0	1,551 ms	10.6 ms	34.8	8	98,290	30.0 s
Sustained decode	16K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	16K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	1	Valid	405.1	394.6	184 ms	2.6 ms	1.0	3	12,150	30.0 s
Sustained decode	32K	2	Valid	604.2	233.7	223 ms	4.3 ms	7.5	4	18,119	30.0 s
Sustained decode	32K	4	Valid	1,554.8	368.9	238 ms	2.7 ms	4.1	10	46,645	30.0 s
Sustained decode	32K	8	Valid	2,008.5	251.8	231 ms	4.0 ms	8.1	13	60,206	30.0 s
Sustained decode	32K	16	Valid	2,526.2	140.3	1,793 ms	7.1 ms	17.3	15	75,698	30.0 s
Sustained decode	32K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	32K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	1	Valid	262.2	198.6	306 ms	5.0 ms	1.0	2	7,863	30.0 s
Sustained decode	64K	2	Valid	655.5	300.9	421 ms	3.3 ms	6.3	4	19,656	30.0 s
Sustained decode	64K	4	Valid	1,233.9	368.5	473 ms	2.7 ms	3.8	9	37,017	30.0 s
Sustained decode	64K	8	Valid	1,855.2	249.3	461 ms	4.0 ms	8.7	13	55,644	30.0 s
Sustained decode	64K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	64K	128	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	1	Valid	223.6	233.3	530 ms	4.4 ms	1.0	1	6,707	30.0 s
Sustained decode	123K	2	Valid	599.4	314.4	544 ms	3.2 ms	3.7	4	17,790	29.7 s
Sustained decode	123K	4	Valid	1,181.7	297.8	893 ms	3.4 ms	4.5	8	35,448	30.0 s
Sustained decode	123K	8	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	16	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	32	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	64	Over capacity	—	—	—	—	—	—	—	—
Sustained decode	123K	128	Over capacity	—	—	—	—	—	—	—	—

Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.