Speed test matrix
Muse-Glimmer-30B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- muse-glimmer
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-meta-sglang/bin/sglang" serve --model-path meta-models/Muse-Glimmer-30B --served-model-name muse-glimmer --tp 1 --reasoning-parser muse --tool-call-parser muse --context-length 131072 --mem-fraction-static 0.856257 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context10.4K790 ms TTFT
16K context10.2K1,590 ms TTFT
32K context9,9953,218 ms TTFT
64K context9,4126,815 ms TTFT
123K context8,47414,523 ms TTFT
128K context8,48915,087 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 59.3Requests queued122 ms TTFT | 114.2186 ms TTFT | 221.9200 ms TTFT | 455.3294 ms TTFT | 857.3422 ms TTFT | 1,583.9786 ms TTFT | 2,616.61,458 ms TTFT | Over capacity |
| 16K | 59.3147 ms TTFT | 84.3238 ms TTFT | 187.7296 ms TTFT | 448.1402 ms TTFT | 826.3772 ms TTFT | 1,510.11,610 ms TTFT | 2,405.53,176 ms TTFT | Over capacity |
| 32K | 59.1198 ms TTFT | 75.8316 ms TTFT | 162.6475 ms TTFT | 429.8898 ms TTFT | 771.91,629 ms TTFT | 1,378.73,244 ms TTFT | Over capacity | Over capacity |
| 64K | 58.7304 ms TTFT | 86.6517 ms TTFT | 192.6906 ms TTFT | 409.71,811 ms TTFT | 686.83,354 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 58.0531 ms TTFT | 89.3968 ms TTFT | 188.91,650 ms TTFT | 374.33,139 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.4K | — | 790 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.2K | — | 1,590 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,995.0 | — | 3,218 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,412.0 | — | 6,815 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 8,474.0 | — | 14,523 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,489.0 | — | 15,087 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 59.3 | 59.4 | 122 ms | 16.8 ms | 1.0 | 0 | 1,779 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 114.2 | 57.1 | 186 ms | 17.5 ms | 3.0 | 0 | 3,426 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 221.9 | 55.6 | 200 ms | 18.0 ms | 6.0 | 0 | 6,656 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 455.3 | 57.0 | 294 ms | 17.5 ms | 12.0 | 0 | 13,656 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 857.3 | 53.7 | 422 ms | 18.6 ms | 24.0 | 0 | 25,712 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,583.9 | 49.6 | 786 ms | 20.2 ms | 48.0 | 0 | 47,515 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,616.6 | 40.9 | 1,458 ms | 24.4 ms | 98.1 | 0 | 78,501 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 59.3 | 59.4 | 147 ms | 16.8 ms | 1.0 | 0 | 1,779 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 84.3 | 42.0 | 238 ms | 23.8 ms | 83.1 | 0 | 2,530 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 187.7 | 46.5 | 296 ms | 21.5 ms | 50.8 | 0 | 5,631 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 448.1 | 56.1 | 402 ms | 17.8 ms | 12.7 | 0 | 13,440 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 826.3 | 51.8 | 772 ms | 19.3 ms | 24.3 | 0 | 24,779 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,510.1 | 47.8 | 1,610 ms | 20.9 ms | 48.0 | 0 | 45,302 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,405.5 | 38.6 | 3,176 ms | 25.9 ms | 99.6 | 0 | 72,139 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 59.1 | 59.2 | 198 ms | 16.9 ms | 1.0 | 0 | 1,772 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 75.8 | 37.9 | 316 ms | 26.4 ms | 88.1 | 0 | 2,272 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 162.6 | 40.7 | 475 ms | 24.6 ms | 66.2 | 0 | 4,876 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 429.8 | 54.1 | 898 ms | 18.5 ms | 13.1 | 0 | 12,892 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 771.9 | 49.0 | 1,629 ms | 20.4 ms | 25.0 | 0 | 23,152 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,378.7 | 44.6 | 3,244 ms | 22.4 ms | 48.4 | 0 | 41,336 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 58.7 | 58.8 | 304 ms | 17.0 ms | 1.0 | 0 | 1,761 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 86.6 | 43.3 | 517 ms | 23.1 ms | 40.3 | 0 | 2,598 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 192.6 | 47.9 | 906 ms | 20.9 ms | 23.1 | 0 | 5,776 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 409.7 | 52.0 | 1,811 ms | 19.2 ms | 12.7 | 0 | 12,292 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 686.8 | 44.7 | 3,354 ms | 22.4 ms | 24.4 | 0 | 20,592 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 58.0 | 58.1 | 531 ms | 17.2 ms | 1.0 | 0 | 1,739 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 89.3 | 44.6 | 968 ms | 22.4 ms | 20.0 | 0 | 2,678 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 188.9 | 47.4 | 1,650 ms | 21.1 ms | 13.5 | 0 | 5,664 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 374.3 | 48.2 | 3,139 ms | 20.7 ms | 12.7 | 0 | 11,224 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.