Speed test matrix
Muse-Glimmer-30B-INT4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · INT45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- INT4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- muse-glimmer
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/Muse-Glimmer-30B-W4A16 --served-model-name muse-glimmer --tensor-parallel-size 1 --reasoning-parser muse_glimmer --enable-auto-tool-choice --tool-call-parser muse_glimmer --max-model-len 131072 --gpu-memory-utilization 0.86 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config auto --speculative-config '{"method":"dflash","model":"meta-models/Muse-Glimmer-30B-assistant","num_speculative_tokens":15}'Prefill
prompt tok/s by context length8K context5,3741,525 ms TTFT
16K context5,3113,046 ms TTFT
32K context5,7715,575 ms TTFT
64K context5,27012,173 ms TTFT
123K context4,85625,351 ms TTFT
128K context4,85926,365 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | Looped | Looped | Looped | Looped | Looped | Looped | 765.41,293 ms TTFT | 845.82,568 ms TTFT |
| 16K | Looped | Looped | Looped | Looped | Looped | 749.31,068 ms TTFT | 721.11,766 ms TTFT | Over capacity |
| 32K | Looped | Looped | Looped | Looped | Looped | 586.71,147 ms TTFT | Over capacity | Over capacity |
| 64K | 176.6184 ms TTFT | 357.2242 ms TTFT | Looped | Looped | Looped | Over capacity | Over capacity | Over capacity |
| 123K | 145.7321 ms TTFT | Looped | Looped | Looped | 323.91,720 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,374.0 | — | 1,525 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 5,311.0 | — | 3,046 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 5,771.0 | — | 5,575 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,270.0 | — | 12,173 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 4,856.0 | — | 25,351 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 4,859.0 | — | 26,365 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 32 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 64 | Valid | 765.4 | 13.7 | 1,293 ms | 72.9 ms | 64.0 | 0 | 22,827 | 29.8 s |
| Sustained decode | 8K | 128 | Valid | 845.8 | 7.8 | 2,568 ms | 128.0 ms | 128.0 | 0 | 25,144 | 29.7 s |
| Sustained decode | 16K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 749.3 | 24.0 | 1,068 ms | 41.6 ms | 32.0 | 0 | 22,434 | 29.9 s |
| Sustained decode | 16K | 64 | Valid | 721.1 | 12.6 | 1,766 ms | 79.5 ms | 64.0 | 0 | 21,535 | 29.9 s |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 586.7 | 19.8 | 1,147 ms | 50.6 ms | 32.0 | 0 | 17,552 | 29.9 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 176.6 | 215.9 | 184 ms | 6.2 ms | 1.0 | 1 | 4,960 | 28.1 s |
| Sustained decode | 64K | 2 | Valid | 357.2 | 211.3 | 242 ms | 6.7 ms | 2.0 | 2 | 9,056 | 25.4 s |
| Sustained decode | 64K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 145.7 | 212.4 | 321 ms | 10.0 ms | 1.0 | 1 | 4,365 | 30.0 s |
| Sustained decode | 123K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Valid | 323.9 | 21.6 | 1,720 ms | 46.4 ms | 16.0 | 0 | 9,687 | 29.9 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.