Speed test matrix
Muse-Glimmer-30B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- muse-glimmer
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/Muse-Glimmer-30B --served-model-name muse-glimmer --tensor-parallel-size 1 --reasoning-parser muse_glimmer --enable-auto-tool-choice --tool-call-parser muse_glimmer --max-model-len 131072 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config auto
Prefill
prompt tok/s by context length8K context5,3911,520 ms TTFT
16K context6,3822,535 ms TTFT
32K context6,3515,066 ms TTFT
64K context5,75411,149 ms TTFT
123K context5,28523,293 ms TTFT
128K context5,24724,416 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 24.7184 ms TTFT | 49.3209 ms TTFT | 98.3233 ms TTFT | 187.0288 ms TTFT | 357.2348 ms TTFT | 612.7469 ms TTFT | 1,024.1734 ms TTFT | Over capacity |
| 16K | 24.6197 ms TTFT | 48.2283 ms TTFT | 97.5287 ms TTFT | 185.9337 ms TTFT | 353.2444 ms TTFT | 602.2647 ms TTFT | Over capacity | Over capacity |
| 32K | 24.5224 ms TTFT | 49.0277 ms TTFT | 97.5332 ms TTFT | 183.9462 ms TTFT | 346.0652 ms TTFT | Over capacity | Over capacity | Over capacity |
| 64K | 24.3281 ms TTFT | 48.5348 ms TTFT | 91.5412 ms TTFT | 178.7623 ms TTFT | 332.71,123 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 24.0405 ms TTFT | 45.2 | 87.3788 ms TTFT | 172.21,098 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,391.0 | — | 1,520 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,382.0 | — | 2,535 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,351.0 | — | 5,066 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,754.0 | — | 11,149 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 5,285.0 | — | 23,293 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,247.0 | — | 24,416 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 24.7 | 24.7 | 184 ms | 40.4 ms | 1.0 | 0 | 740 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 49.3 | 24.7 | 209 ms | 40.5 ms | 2.0 | 0 | 1,476 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 98.3 | 24.7 | 233 ms | 40.5 ms | 4.0 | 0 | 2,949 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 187.0 | 23.5 | 288 ms | 42.6 ms | 8.0 | 0 | 5,608 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 357.2 | 22.4 | 348 ms | 44.7 ms | 16.0 | 0 | 10,704 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 612.7 | 19.2 | 469 ms | 52.0 ms | 32.0 | 0 | 18,381 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,024.1 | 16.1 | 734 ms | 62.1 ms | 64.0 | 0 | 30,723 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 24.6 | 24.7 | 197 ms | 40.5 ms | 1.0 | 0 | 739 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 48.2 | 24.7 | 283 ms | 40.6 ms | 2.0 | 0 | 1,445 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 97.5 | 24.6 | 287 ms | 40.6 ms | 4.0 | 0 | 2,924 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 185.9 | 23.3 | 337 ms | 42.9 ms | 8.0 | 0 | 5,576 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 353.2 | 22.2 | 444 ms | 45.1 ms | 16.0 | 0 | 10,593 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 602.2 | 18.9 | 647 ms | 52.9 ms | 32.0 | 0 | 18,047 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 24.5 | 24.6 | 224 ms | 40.6 ms | 1.0 | 0 | 735 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 49.0 | 24.6 | 277 ms | 40.7 ms | 2.0 | 0 | 1,468 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 97.5 | 24.5 | 332 ms | 40.9 ms | 4.0 | 0 | 2,924 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 183.9 | 23.1 | 462 ms | 43.3 ms | 8.0 | 0 | 5,518 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 346.0 | 21.7 | 652 ms | 46.1 ms | 16.0 | 0 | 10,369 | 30.0 s |
| Sustained decode | 32K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 24.3 | 24.4 | 281 ms | 41.0 ms | 1.0 | 0 | 729 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 48.5 | 24.3 | 348 ms | 41.1 ms | 2.0 | 0 | 1,456 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 91.5 | 24.1 | 412 ms | 41.4 ms | 4.0 | 0 | 2,744 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 178.7 | 22.5 | 623 ms | 44.4 ms | 8.0 | 0 | 5,355 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 332.7 | 20.8 | 1,123 ms | 48.0 ms | 16.0 | 0 | 9,968 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 24.0 | 24.1 | 405 ms | 41.6 ms | 1.0 | 0 | 719 | 30.0 s |
| Sustained decode | 123K | 2 | Valid | 45.2 | — | — | — | 2.0 | 0 | 1,103 | 24.4 s |
| Sustained decode | 123K | 4 | Valid | 87.3 | 23.6 | 788 ms | 42.4 ms | 4.0 | 0 | 2,618 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 172.2 | 21.7 | 1,098 ms | 46.1 ms | 8.0 | 0 | 5,163 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.