Speed test matrix
Muse-Glimmer-30B on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- muse-glimmer
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-meta-sglang/bin/sglang" serve --model-path meta-models/Muse-Glimmer-30B --served-model-name muse-glimmer --tp 1 --reasoning-parser muse --tool-call-parser muse --context-length 131072 --mem-fraction-static 0.859841 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --speculative-algorithm DFLASH --speculative-draft-model-path meta-models/Muse-Glimmer-30B-assistant
Prefill
prompt tok/s by context length8K context10.3K793 ms TTFT
16K context10.3K1,571 ms TTFT
32K context9,9063,247 ms TTFT
64K context9,2956,900 ms TTFT
123K context8,39114,668 ms TTFT
128K context8,43515,184 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 363.195 ms TTFT | 781.7134 ms TTFT | 1,445.2123 ms TTFT | 2,258.6148 ms TTFT | 2,483.2287 ms TTFT | 1,925.4749 ms TTFT | Not run | Over capacity |
| 16K | 371.1123 ms TTFT | 596.4161 ms TTFT | 1,349.1169 ms TTFT | 2,265.1167 ms TTFT | 2,464.4514 ms TTFT | 3,661.91,523 ms TTFT | Over capacity | Over capacity |
| 32K | 406.2176 ms TTFT | 316.8285 ms TTFT | 1,431.4213 ms TTFT | 2,239.8228 ms TTFT | 2,404.41,007 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 400.8283 ms TTFT | 619.6321 ms TTFT | 1,264.1472 ms TTFT | 2,180.2438 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
| 123K | 401.3476 ms TTFT | 657.1512 ms TTFT | 1,369.8Requests queued671 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity | Over capacity |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 10.3K | — | 793 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 10.3K | — | 1,571 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 9,906.0 | — | 3,247 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 9,295.0 | — | 6,900 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 8,391.0 | — | 14,668 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 8,435.0 | — | 15,184 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 363.1 | 338.4 | 95 ms | 3.0 ms | 0.9 | 2 | 10,892 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 781.7 | 350.9 | 134 ms | 2.8 ms | 2.1 | 5 | 23,438 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 1,445.2 | 330.6 | 123 ms | 3.0 ms | 4.0 | 9 | 43,355 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 2,258.6 | 262.9 | 148 ms | 3.8 ms | 8.1 | 16 | 67,760 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 2,483.2 | 126.0 | 287 ms | 7.9 ms | 18.1 | 16 | 74,487 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,925.4 | 58.1 | 749 ms | 17.2 ms | 43.2 | 0 | 57,761 | 30.0 s |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 371.1 | 339.2 | 123 ms | 2.9 ms | 1.0 | 2 | 11,129 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 596.4 | 232.5 | 161 ms | 4.3 ms | 8.7 | 4 | 17,881 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 1,349.1 | 323.3 | 169 ms | 3.1 ms | 4.1 | 8 | 40,461 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 2,265.1 | 278.4 | 167 ms | 3.6 ms | 7.7 | 16 | 67,951 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 2,464.4 | 130.1 | 514 ms | 7.8 ms | 18.0 | 16 | 73,854 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 3,661.9 | 76.7 | 1,523 ms | 13.1 ms | 33.9 | 25 | 109,777 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 406.2 | 394.2 | 176 ms | 2.5 ms | 1.0 | 3 | 12,180 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 316.8 | 149.3 | 285 ms | 6.7 ms | 20.9 | 2 | 9,499 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 1,431.4 | 343.7 | 213 ms | 2.9 ms | 4.0 | 8 | 42,915 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 2,239.8 | 285.7 | 228 ms | 3.5 ms | 7.7 | 16 | 67,172 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 2,404.4 | 142.2 | 1,007 ms | 7.0 ms | 18.1 | 16 | 72,094 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 400.8 | 377.1 | 283 ms | 2.7 ms | 0.9 | 3 | 12,025 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 619.6 | 258.9 | 321 ms | 3.9 ms | 5.8 | 4 | 18,579 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 1,264.1 | 321.7 | 472 ms | 3.1 ms | 3.9 | 8 | 37,897 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 2,180.2 | 262.9 | 438 ms | 3.8 ms | 7.8 | 16 | 65,402 | 30.0 s |
| Sustained decode | 64K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 401.3 | 400.6 | 476 ms | 2.5 ms | 0.9 | 3 | 12,037 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 657.1 | 336.8 | 512 ms | 3.0 ms | 3.4 | 4 | 19,708 | 30.0 s |
| Sustained decode | 123K | 4 | Requests queued | 1,369.8 | 335.4 | 671 ms | 3.0 ms | 3.8 | 8 | 41,080 | 30.0 s |
| Sustained decode | 123K | 8 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Requests queued: the server could not run all requests at once, so some waited in a queue. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.