Speed test matrix
Muse-Glimmer-30B on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · BF165 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- BF16
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- muse-glimmer
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-meta-models-vllm-73a5831127a9/bin/vllm" serve meta-models/Muse-Glimmer-30B --served-model-name muse-glimmer --tensor-parallel-size 1 --reasoning-parser muse_glimmer --enable-auto-tool-choice --tool-call-parser muse_glimmer --max-model-len 131072 --gpu-memory-utilization 0.84 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config auto --max-num-seqs 64 --attention-backend FLASH_ATTN
Prefill
prompt tok/s by context length8K context5,3861,521 ms TTFT
16K context6,3832,535 ms TTFT
32K context6,3515,066 ms TTFT
64K context5,75711,144 ms TTFT
123K context5,27523,337 ms TTFT
128K context5,26424,336 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 24.7184 ms TTFT | 49.3209 ms TTFT | 98.4252 ms TTFT | 186.9293 ms TTFT | 352.8368 ms TTFT | 611.9469 ms TTFT | 1,022.7760 ms TTFT | Over capacity |
| 16K | 24.6197 ms TTFT | 49.2242 ms TTFT | 97.9286 ms TTFT | 184.4336 ms TTFT | 349.8443 ms TTFT | 602.0633 ms TTFT | Over capacity | Over capacity |
| 32K | 24.5223 ms TTFT | 48.1235 ms TTFT | 92.0318 ms TTFT | 182.7469 ms TTFT | 345.0721 ms TTFT | 582.41,029 ms TTFT | Over capacity | Over capacity |
| 64K | 24.3282 ms TTFT | 48.5349 ms TTFT | 85.4440 ms TTFT | 177.8741 ms TTFT | 329.41,107 ms TTFT | Over capacity | Over capacity | Over capacity |
| 123K | 24.0 | 46.3611 ms TTFT | 94.1765 ms TTFT | 171.31,286 ms TTFT | Over capacity | Over capacity | Over capacity | Over capacity |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 5,386.0 | — | 1,521 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 6,383.0 | — | 2,535 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 6,351.0 | — | 5,066 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 5,757.0 | — | 11,144 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 5,275.0 | — | 23,337 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 5,264.0 | — | 24,336 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 24.7 | 24.8 | 184 ms | 40.4 ms | 1.0 | 0 | 740 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 49.3 | 24.7 | 209 ms | 40.4 ms | 2.0 | 0 | 1,478 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 98.4 | 24.7 | 252 ms | 40.5 ms | 4.0 | 0 | 2,952 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 186.9 | 23.5 | 293 ms | 42.6 ms | 8.0 | 0 | 5,600 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 352.8 | 22.4 | 368 ms | 44.7 ms | 16.0 | 0 | 10,579 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 611.9 | 19.2 | 469 ms | 52.0 ms | 32.0 | 0 | 18,360 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,022.7 | 16.0 | 760 ms | 62.3 ms | 64.0 | 0 | 30,657 | 30.0 s |
| Sustained decode | 8K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 1 | Valid | 24.6 | 24.7 | 197 ms | 40.5 ms | 1.0 | 0 | 738 | 30.0 s |
| Sustained decode | 16K | 2 | Valid | 49.2 | 24.7 | 242 ms | 40.5 ms | 2.0 | 0 | 1,473 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 97.9 | 24.6 | 286 ms | 40.7 ms | 4.0 | 0 | 2,933 | 30.0 s |
| Sustained decode | 16K | 8 | Valid | 184.4 | 23.3 | 336 ms | 42.9 ms | 8.0 | 0 | 5,532 | 30.0 s |
| Sustained decode | 16K | 16 | Valid | 349.8 | 22.1 | 443 ms | 45.2 ms | 16.0 | 0 | 10,483 | 30.0 s |
| Sustained decode | 16K | 32 | Valid | 602.0 | 18.9 | 633 ms | 53.0 ms | 32.0 | 0 | 18,051 | 30.0 s |
| Sustained decode | 16K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 16K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 1 | Valid | 24.5 | 24.6 | 223 ms | 40.6 ms | 1.0 | 0 | 736 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 48.1 | 24.6 | 235 ms | 40.7 ms | 2.0 | 0 | 1,442 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 92.0 | 24.4 | 318 ms | 40.9 ms | 4.0 | 0 | 2,761 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 182.7 | 23.1 | 469 ms | 43.3 ms | 8.0 | 0 | 5,478 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 345.0 | 21.7 | 721 ms | 46.1 ms | 16.0 | 0 | 10,340 | 30.0 s |
| Sustained decode | 32K | 32 | Valid | 582.4 | 18.3 | 1,029 ms | 54.7 ms | 32.0 | 0 | 17,470 | 30.0 s |
| Sustained decode | 32K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 24.3 | 24.4 | 282 ms | 41.0 ms | 1.0 | 0 | 729 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 48.5 | 24.3 | 349 ms | 41.1 ms | 2.0 | 0 | 1,454 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 85.4 | 24.2 | 440 ms | 41.4 ms | 4.0 | 0 | 2,561 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 177.8 | 22.6 | 741 ms | 44.3 ms | 8.0 | 0 | 5,327 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 329.4 | 20.8 | 1,107 ms | 48.1 ms | 16.0 | 0 | 9,875 | 30.0 s |
| Sustained decode | 64K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 24.0 | — | — | — | 1.0 | 0 | 604 | 25.2 s |
| Sustained decode | 123K | 2 | Valid | 46.3 | 23.9 | 611 ms | 41.8 ms | 2.0 | 0 | 1,387 | 30.0 s |
| Sustained decode | 123K | 4 | Valid | 94.1 | 23.6 | 765 ms | 42.4 ms | 4.0 | 0 | 2,821 | 30.0 s |
| Sustained decode | 123K | 8 | Valid | 171.3 | 21.7 | 1,286 ms | 46.1 ms | 8.0 | 0 | 5,137 | 30.0 s |
| Sustained decode | 123K | 16 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Over capacity: not run because this many requests at this context would exceed the server's KV cache.