Speed test matrix
Muse-Glimmer-30B-NVFP4 on H200 NVL 141GB
sglang · 0.5.20 · NVFP4+FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- sglang
- Version
- 0.5.20
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/sgl-project/sglang
- Build
- cuda-13.2
Run
- Served model
- muse-glimmer-nvfp4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-muse-sglang-v0520/bin/sglang" serve --model-path nvidia/Muse-Glimmer-30B-NVFP4 --served-model-name muse-glimmer-nvfp4 --tp 1 --reasoning-parser muse --tool-call-parser muse --context-length 131072 --mem-fraction-static 0.84 --enable-metrics --host 0.0.0.0 --port 8000 --api-key <redacted> --kv-cache-dtype fp8_e4m3 --attention-backend fa3 --speculative-algorithm DFLASH --speculative-draft-model-path meta-models/Muse-Glimmer-30B-assistant --speculative-draft-model-quantization fp8
Prefill
prompt tok/s by context length8K context6,3841,284 ms TTFT
16K context6,3902,532 ms TTFT
32K context6,2775,125 ms TTFT
64K context6,09410,528 ms TTFT
123K context5,74821,419 ms TTFT
128K context5,74022,315 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | Looped | Looped | Looped | Looped | Looped | Not run | Not run |
| 16K | Looped | Looped | Looped | Looped | Looped | Looped | Not run | Not run |
| 32K | Looped | Looped | Looped | Looped | Looped | Looped | Not run | Over capacity |
| 64K | Looped | Looped | Looped | Looped | Looped | Looped | Over capacity | Over capacity |
| 123K | Looped | Looped | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 6,384.0 | — | 1,284 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,390.0 | — | 2,532 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,277.0 | — | 5,125 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,094.0 | — | 10,528 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 5,748.0 | — | 21,419 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,740.0 | — | 22,315 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 8K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Not run | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache. Not run: the benchmark skipped this measurement during the run, or the run stopped early.