Speed test matrix
Muse-Glimmer-30B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- muse-glimmer
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-sglang/bin/sglang" serve --model-path RedHatAI/Muse-Glimmer-30B --served-model-name muse-glimmer --tp 1 --reasoning-parser muse --tool-call-parser muse --context-length 131072 --mem-fraction-static 0.856257 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context10.4K790 ms TTFT
16K context10.3K1,571 ms TTFT
32K context9,9793,223 ms TTFT
64K context9,3816,837 ms TTFT
123K context8,46314,543 ms TTFT
128K context8,43015,193 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 59.5Requests queued124 ms TTFT | 113.4176 ms TTFT | 221.9201 ms TTFT | 455.1274 ms TTFT | 857.8513 ms TTFT | 1,586.4790 ms TTFT | 2,608.71,443 ms TTFT | Over capacity |
| 16K | 59.3161 ms TTFT | 83.9243 ms TTFT | 187.5285 ms TTFT | 448.7402 ms TTFT | 827.9763 ms TTFT | 1,510.11,580 ms TTFT | 2,401.82,988 ms TTFT | Over capacity |
| 32K | 59.1198 ms TTFT | 75.6327 ms TTFT | 162.3480 ms TTFT | 428.6873 ms TTFT | 772.41,717 ms TTFT | 1,380.23,372 ms TTFT | Over capacity | Over capacity |
| 64K | 58.7325 ms TTFT | 86.8545 ms TTFT | 193.9911 ms TTFT | 410.91,788 ms TTFT | 685.73,489 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 58.0526 ms TTFT | 89.3939 ms TTFT | 189.71,699 ms TTFT | 377.33,257 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.4K | — | 790 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.3K | — | 1,571 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,979.0 | — | 3,223 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,381.0 | — | 6,837 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 8,463.0 | — | 14,543 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,430.0 | — | 15,193 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Requests queued | 59.5 | 59.5 | 124 ms | 16.8 ms | 1.0 | 0 | 1,783 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 113.4 | 56.8 | 176 ms | 17.6 ms | 3.0 | 0 | 3,402 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 221.9 | 55.6 | 201 ms | 18.0 ms | 6.0 | 0 | 6,654 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 455.1 | 57.0 | 274 ms | 17.6 ms | 12.0 | 0 | 13,648 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 857.8 | 53.7 | 513 ms | 18.6 ms | 24.0 | 0 | 25,723 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,586.4 | 49.7 | 790 ms | 20.1 ms | 48.0 | 0 | 47,572 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 2,608.7 | 40.8 | 1,443 ms | 24.5 ms | 98.1 | 0 | 78,263 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 59.3 | 59.4 | 161 ms | 16.8 ms | 1.0 | 0 | 1,779 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 83.9 | 41.8 | 243 ms | 23.9 ms | 82.0 | 0 | 2,516 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 187.5 | 46.5 | 285 ms | 21.5 ms | 48.7 | 0 | 5,624 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 448.7 | 56.2 | 402 ms | 17.8 ms | 12.7 | 0 | 13,459 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 827.9 | 51.9 | 763 ms | 19.3 ms | 24.3 | 0 | 24,832 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,510.1 | 47.7 | 1,580 ms | 20.9 ms | 48.0 | 0 | 45,306 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,401.8 | 38.4 | 2,988 ms | 26.0 ms | 99.7 | 0 | 72,063 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 59.1 | 59.1 | 198 ms | 16.9 ms | 1.0 | 0 | 1,771 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 75.6 | 37.8 | 327 ms | 26.5 ms | 88.7 | 0 | 2,268 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 162.3 | 40.6 | 480 ms | 24.6 ms | 66.8 | 0 | 4,868 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 428.6 | 53.9 | 873 ms | 18.5 ms | 13.2 | 0 | 12,857 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 772.4 | 49.1 | 1,717 ms | 20.4 ms | 25.1 | 0 | 23,162 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,380.2 | 44.7 | 3,372 ms | 22.4 ms | 48.4 | 0 | 41,408 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 58.7 | 58.8 | 325 ms | 17.0 ms | 1.0 | 0 | 1,761 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 86.8 | 43.4 | 545 ms | 23.1 ms | 40.0 | 0 | 2,604 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 193.9 | 48.2 | 911 ms | 20.7 ms | 22.6 | 0 | 5,816 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 410.9 | 52.1 | 1,788 ms | 19.2 ms | 12.7 | 0 | 12,320 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 685.7 | 44.7 | 3,489 ms | 22.4 ms | 24.3 | 0 | 20,560 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 58.0 | 58.1 | 526 ms | 17.2 ms | 1.0 | 0 | 1,741 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 89.3 | 44.6 | 939 ms | 22.4 ms | 19.9 | 0 | 2,676 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 189.7 | 47.5 | 1,699 ms | 21.0 ms | 13.1 | 0 | 5,688 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 377.3 | 48.5 | 3,257 ms | 20.6 ms | 12.5 | 0 | 11,312 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache.