Speed test matrix
Muse-Glimmer-30B-FP8-block on H200 NVL 141GB
sglang · 0.5.6.post3.dev9265+g05c584c44 · FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.6.post3.dev9265+g05c584c44
- Quantization
- FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
- Commit
05c584c44fb0
Run
- Served model
- muse-glimmer
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.4.29
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-redhatai-sglang/bin/sglang" serve --model-path RedHatAI/Muse-Glimmer-30B-FP8-block --served-model-name muse-glimmer --tp 1 --reasoning-parser muse --tool-call-parser muse --context-length 131072 --mem-fraction-static 0.855022 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted>
Prefill
prompt tok/s by context length8K context14.6K561 ms TTFT
16K context15.5K1,043 ms TTFT
32K context14.9K2,163 ms TTFT
64K context13.7K4,685 ms TTFT
123K context11.7K10,501 ms TTFT
128K context11.7K10,972 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 73.2126 ms TTFT | 143.1178 ms TTFT | 295.7191 ms TTFT | 591.1264 ms TTFT | 1,128.8443 ms TTFT | 2,108.3728 ms TTFT | 3,272.31,445 ms TTFT | 4,216.12,811 ms TTFT |
| 16K | 73.0148 ms TTFT | 64.4274 ms TTFT | 130.2318 ms TTFT | 458.8514 ms TTFT | 1,049.1746 ms TTFT | 1,969.91,424 ms TTFT | 2,950.72,847 ms TTFT | Over capacity |
| 32K | 72.7206 ms TTFT | 93.1326 ms TTFT | 255.7450 ms TTFT | 567.6865 ms TTFT | 1,013.91,691 ms TTFT | 1,744.13,236 ms TTFT | Over capacity | Over capacity |
| 64K | 72.1327 ms TTFT | 107.1529 ms TTFT | 284.8877 ms TTFT | 540.81,787 ms TTFT | 875.03,347 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 71.1497 ms TTFT | 111.2923 ms TTFT | 269.11,691 ms TTFT | 494.73,219 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 14.6K | — | 561 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.5K | — | 1,043 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 14.9K | — | 2,163 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 13.7K | — | 4,685 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 11.7K | — | 10,501 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 11.7K | — | 10,972 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 73.2 | 73.3 | 126 ms | 13.7 ms | 1.0 | 0 | 2,196 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 143.1 | 71.4 | 178 ms | 14.0 ms | 2.6 | 0 | 4,294 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 295.7 | 74.1 | 191 ms | 13.5 ms | 5.0 | 0 | 8,868 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 591.1 | 73.9 | 264 ms | 13.5 ms | 10.0 | 0 | 17,725 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 1,128.8 | 70.5 | 443 ms | 14.2 ms | 20.3 | 0 | 33,856 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 2,108.3 | 65.8 | 728 ms | 15.2 ms | 42.1 | 0 | 63,238 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 3,272.3 | 51.3 | 1,445 ms | 19.5 ms | 94.9 | 0 | 98,167 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 4,216.1 | 33.3 | 2,811 ms | 30.0 ms | 192.0 | 0 | 126,451 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 73.0 | 73.1 | 148 ms | 13.7 ms | 1.0 | 0 | 2,191 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 64.4 | 32.3 | 274 ms | 31.0 ms | 187.6 | 0 | 1,932 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 130.2 | 32.7 | 318 ms | 30.6 ms | 134.0 | 0 | 3,904 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 458.8 | 54.4 | 514 ms | 18.4 ms | 59.2 | 0 | 13,760 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 1,049.1 | 65.7 | 746 ms | 15.2 ms | 24.4 | 0 | 31,473 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,969.9 | 61.9 | 1,424 ms | 16.2 ms | 44.8 | 0 | 59,104 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 2,950.7 | 47.0 | 2,847 ms | 21.3 ms | 96.0 | 0 | 88,523 | 30.0 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 72.7 | 72.7 | 206 ms | 13.7 ms | 1.0 | 0 | 2,179 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 93.1 | 46.5 | 326 ms | 21.5 ms | 66.0 | 0 | 2,792 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 255.7 | 61.8 | 450 ms | 16.2 ms | 29.5 | 0 | 7,668 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 567.6 | 71.4 | 865 ms | 14.0 ms | 10.8 | 0 | 17,028 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 1,013.9 | 64.2 | 1,691 ms | 15.6 ms | 21.2 | 0 | 30,416 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 1,744.1 | 56.5 | 3,236 ms | 17.7 ms | 46.0 | 0 | 52,310 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 72.1 | 72.2 | 327 ms | 13.9 ms | 1.0 | 0 | 2,163 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 107.1 | 53.7 | 529 ms | 18.6 ms | 34.0 | 0 | 3,212 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 284.8 | 71.4 | 877 ms | 14.0 ms | 5.7 | 0 | 8,544 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 540.8 | 68.8 | 1,787 ms | 14.5 ms | 10.3 | 0 | 16,224 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 875.0 | 56.8 | 3,347 ms | 17.6 ms | 22.3 | 0 | 26,251 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 71.1 | 71.2 | 497 ms | 14.0 ms | 1.0 | 0 | 2,132 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 111.2 | 56.0 | 923 ms | 17.9 ms | 18.0 | 0 | 3,336 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 269.1 | 68.4 | 1,691 ms | 14.6 ms | 5.7 | 0 | 8,072 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 494.7 | 64.1 | 3,219 ms | 15.6 ms | 10.7 | 0 | 14,836 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.