Speed test matrix
Muse-Glimmer-30B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP4+FP85 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4+FP8
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- muse-glimmer-nvfp4
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
"$HOME/env_rtxpro6k-nvidia-vllm-73a5831127a9/bin/vllm" serve nvidia/Muse-Glimmer-30B-NVFP4 --served-model-name muse-glimmer-nvfp4 --trust-remote-code --tensor-parallel-size 1 --reasoning-parser muse_glimmer --enable-auto-tool-choice --tool-call-parser muse_glimmer --max-model-len 131072 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --enable-prefix-caching --enable-chunked-prefill --kv-cache-dtype auto --max-num-seqs 64 --mamba-cache-mode align --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context5,3751,525 ms TTFT
16K context6,3822,535 ms TTFT
32K context6,3425,073 ms TTFT
64K context6,01510,665 ms TTFT
123K context5,40022,799 ms TTFT
128K context5,46223,451 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 57.7 | Looped | Looped | Looped | 566.8246 ms TTFT | 1,076.1368 ms TTFT | 1,582.3611 ms TTFT | Looped |
| 16K | 57.4 | Looped | 177.6173 ms TTFT | 312.2220 ms TTFT | 581.8340 ms TTFT | 1,037.0493 ms TTFT | 1,518.6877 ms TTFT | Looped |
| 32K | 56.9 | 72.8180 ms TTFT | 188.8 | Looped | 585.7557 ms TTFT | 972.9922 ms TTFT | 1,385.11,800 ms TTFT | Over capacity |
| 64K | 55.7 | 106.4 | Looped | 330.6 | 455.81,208 ms TTFT | 837.32,137 ms TTFT | Over capacity | Over capacity |
| 123K | 54.0 | 95.2 | 110.6759 ms TTFT | Looped | 383.21,779 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,375.0 | — | 1,525 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,382.0 | — | 2,535 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,342.0 | — | 5,073 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 6,015.0 | — | 10,665 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 5,400.0 | — | 22,799 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,462.0 | — | 23,451 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 57.7 | — | — | — | 1.0 | 0 | 626 | 10.9 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Valid | 566.8 | 46.0 | 246 ms | 21.7 ms | 16.0 | 0 | 17,004 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,076.1 | 35.6 | 368 ms | 28.1 ms | 32.0 | 0 | 32,286 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,582.3 | 24.9 | 611 ms | 40.2 ms | 64.0 | 0 | 47,469 | 30.0 s |
| Sustained decode | 8K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 1 | Valid | 57.4 | — | — | — | 1.0 | 0 | 801 | 14.0 s |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Valid | 177.6 | 57.2 | 173 ms | 17.5 ms | 4.0 | 0 | 5,328 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 312.2 | 51.2 | 220 ms | 19.5 ms | 8.0 | 0 | 9,362 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 581.8 | 45.1 | 340 ms | 22.2 ms | 16.0 | 0 | 17,453 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 1,037.0 | 34.5 | 493 ms | 29.0 ms | 32.0 | 0 | 31,095 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,518.6 | 23.7 | 877 ms | 42.1 ms | 64.0 | 0 | 45,559 | 30.0 s |
| Sustained decode | 16K | 128 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 1 | Valid | 56.9 | — | — | — | 1.0 | 0 | 715 | 12.6 s |
| Sustained decode | 32K | 2 | Valid | 72.8 | 56.8 | 180 ms | 17.6 ms | 2.0 | 0 | 2,184 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 188.8 | — | — | — | 4.0 | 0 | 2,801 | 14.8 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Valid | 585.7 | 43.4 | 557 ms | 23.1 ms | 16.0 | 0 | 17,561 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 972.9 | 32.3 | 922 ms | 31.0 ms | 32.0 | 0 | 29,181 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,385.1 | 21.6 | 1,800 ms | 46.2 ms | 64.0 | 0 | 41,554 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 55.7 | — | — | — | 1.0 | 0 | 959 | 17.2 s |
| Sustained decode | 64K | 2 | Valid | 106.4 | — | — | — | 2.0 | 0 | 1,852 | 17.4 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Valid | 330.6 | — | — | — | 8.0 | 0 | 6,204 | 18.8 s |
| Sustained decode | 64K | 16 | Valid | 455.8 | 40.1 | 1,208 ms | 24.9 ms | 16.0 | 0 | 13,671 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 837.3 | 28.8 | 2,137 ms | 34.8 ms | 32.0 | 0 | 25,101 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 54.0 | — | — | — | 1.0 | 0 | 699 | 12.9 s |
| Sustained decode | 123K | 2 | Valid | 95.2 | — | — | — | 2.0 | 0 | 973 | 10.2 s |
| Sustained decode | 123K | 4 | Valid | 110.6 | 52.0 | 759 ms | 19.2 ms | 4.0 | 0 | 3,315 | 30.0 s |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Valid | 383.2 | 34.9 | 1,779 ms | 28.7 ms | 16.0 | 0 | 11,496 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.