Speed test matrix
Muse-Glimmer-30B-NVFP4 on H200 NVL 141GB
vllm · 0.29.0 · NVFP4+FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- H200 NVL
- VRAM
- 141 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.29.0
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
Run
- Served model
- muse-glimmer-nvfp4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.6.2
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_h200-nvidia-muse-vllm-v0290/bin/vllm" serve nvidia/Muse-Glimmer-30B-NVFP4 --served-model-name muse-glimmer-nvfp4 --trust-remote-code --tensor-parallel-size 1 --reasoning-parser muse_glimmer --enable-auto-tool-choice --tool-call-parser muse_glimmer --max-model-len 131072 --gpu-memory-utilization 0.886040 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-prefix-caching --enable-chunked-prefill --kv-cache-dtype auto --max-num-seqs 64 --mamba-cache-mode align --attention-backend FLASH_ATTN --linear-backend cutlass
Prefill
prompt tok/s by context length8K context5,3371,535 ms TTFT
16K context6,3122,563 ms TTFT
32K context6,2805,122 ms TTFT
64K context5,96610,752 ms TTFT
123K context5,59322,010 ms TTFT
128K context5,68922,517 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 91.973 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 16K | 91.8109 ms TTFT | Looped | Looped | Looped | Looped | Looped | Looped | Looped |
| 32K | 91.4134 ms TTFT | 104.0200 ms TTFT | Looped | 557.6312 ms TTFT | Looped | Looped | Looped | Looped |
| 64K | 90.1221 ms TTFT | 156.5265 ms TTFT | Looped | Looped | Looped | Looped | 1,704.43,099 ms TTFT | Over capacity |
| 123K | 88.2331 ms TTFT | 169.4427 ms TTFT | 302.4650 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,337.0 | — | 1,535 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,312.0 | — | 2,563 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,280.0 | — | 5,122 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,966.0 | — | 10,752 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 5,593.0 | — | 22,010 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,689.0 | — | 22,517 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 91.9 | 92.0 | 73 ms | 10.9 ms | 1.0 | 0 | 2,757 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 91.8 | 92.1 | 109 ms | 10.9 ms | 1.0 | 0 | 778 | 8.5 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 91.4 | 91.6 | 134 ms | 10.9 ms | 1.0 | 0 | 499 | 5.5 s |
| Sustained decode | 32K | 2 | Valid | 104.0 | 91.1 | 200 ms | 11.0 ms | 2.0 | 0 | 3,121 | 30.0 s |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Valid | 557.6 | 87.5 | 312 ms | 11.4 ms | 8.0 | 0 | 4,789 | 8.6 s |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 64 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 1 | Valid | 90.1 | 90.6 | 221 ms | 11.0 ms | 1.0 | 0 | 771 | 8.6 s |
| Sustained decode | 64K | 2 | Valid | 156.5 | 89.7 | 265 ms | 11.2 ms | 2.0 | 0 | 1,108 | 7.1 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 64 | Valid | 1,704.4 | 37.8 | 3,099 ms | 26.4 ms | 64.0 | 0 | 51,122 | 30.0 s |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 88.2 | 88.8 | 331 ms | 11.3 ms | 1.0 | 0 | 513 | 5.8 s |
| Sustained decode | 123K | 2 | Valid | 169.4 | 87.8 | 427 ms | 11.4 ms | 2.0 | 0 | 1,314 | 7.8 s |
| Sustained decode | 123K | 4 | Valid | 302.4 | 83.8 | 650 ms | 11.9 ms | 4.0 | 0 | 1,970 | 6.5 s |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.