Speed test matrix
Muse-Glimmer-30B-NVFP4 on RTX PRO 6000 Blackwell Server Edition 96GB
vllm · 0.30.1rc1.dev452+g73a583112 · NVFP45 contexts (8K–123K) × 8 concurrency levels (1–128) · prefill at 8K, 16K, 32K, 64K, 123K, 128K
Hardware
- Type
- Discrete GPU
- GPU
- RTX PRO 6000 Blackwell Server Edition
- VRAM
- 96 GB
- OS
- linux
Engine
- Engine
- vllm
- Version
- 0.30.1rc1.dev452+g73a583112
- Quantization
- NVFP4
- Backend
- cuda
- Repository
- github.com/vllm-project/vllm
- Build
- cuda-13.2
- Commit
73a5831127a9
Run
- Served model
- muse-glimmer
- Revision
- main
- Benchmark tool
- llm-inference-bench v0.7.6
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 4,096 tokens
Launch command
env MAX_JOBS=2 FLASHINFER_NVCC_THREADS=1 TORCHINDUCTOR_COMPILE_THREADS=1 "$HOME/env_rtxpro6k-redhatai-vllm-73a5831127a9d2b8/bin/vllm" serve RedHatAI/Muse-Glimmer-30B-NVFP4 --served-model-name muse-glimmer --tensor-parallel-size 1 --reasoning-parser muse_glimmer --enable-auto-tool-choice --tool-call-parser muse_glimmer --max-model-len 131072 --gpu-memory-utilization 0.83 --host 0.0.0.0 --port 8000 --api-key <redacted> --generation-config auto
Prefill
prompt tok/s by context length8K context16.1K509 ms TTFT
16K context15.9K1,016 ms TTFT
32K context15.9K2,028 ms TTFT
64K context12.7K5,068 ms TTFT
123K context10.6K11,657 ms TTFT
128K context10.4K12,280 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|---|---|---|---|---|---|---|---|---|
| 8K | 58.590 ms TTFT | Looped | Looped | 324.1151 ms TTFT | 564.0254 ms TTFT | 1,056.9349 ms TTFT | 1,654.5665 ms TTFT | 2,069.61,104 ms TTFT |
| 16K | 58.3 | 74.7110 ms TTFT | 177.7 | Looped | Looped | 975.2538 ms TTFT | 1,545.0912 ms TTFT | 1,962.81,682 ms TTFT |
| 32K | 57.8 | 81.7140 ms TTFT | 127.0231 ms TTFT | Looped | Looped | 967.6892 ms TTFT | 1,428.71,638 ms TTFT | Over capacity |
| 64K | 56.5 | 113.1235 ms TTFT | 137.6 | 316.1482 ms TTFT | 534.5698 ms TTFT | 897.21,468 ms TTFT | Over capacity | Over capacity |
| 123K | 54.9 | 89.8 | 179.4 | Looped | 468.31,575 ms TTFT | Over capacity | Over capacity | Over capacity |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 16.1K | — | 509 ms | — | — | 1 | — | — |
| Prefill | 16K | 1 | Valid | 15.9K | — | 1,016 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 15.9K | — | 2,028 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 12.7K | — | 5,068 ms | — | — | 1 | — | — |
| Prefill | 123K | 1 | Valid | 10.6K | — | 11,657 ms | — | — | 1 | — | — |
| Prefill | 128K | 1 | Valid | 10.4K | — | 12,280 ms | — | — | 1 | — | — |
| Sustained decode | 8K | 1 | Valid | 58.5 | 58.6 | 90 ms | 17.1 ms | 1.0 | 0 | 1,755 | 30.0 s |
| Sustained decode | 8K | 2 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 4 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 8K | 8 | Valid | 324.1 | 52.3 | 151 ms | 19.1 ms | 8.0 | 0 | 9,720 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 564.0 | 46.7 | 254 ms | 21.4 ms | 16.0 | 0 | 16,919 | 30.0 s |
| Sustained decode | 8K | 32 | Valid | 1,056.9 | 36.2 | 349 ms | 27.6 ms | 32.0 | 0 | 31,707 | 30.0 s |
| Sustained decode | 8K | 64 | Valid | 1,654.5 | 26.1 | 665 ms | 38.3 ms | 64.0 | 0 | 49,611 | 30.0 s |
| Sustained decode | 8K | 128 | Valid | 2,069.6 | 16.2 | 1,104 ms | 61.6 ms | 128.0 | 0 | 62,088 | 30.0 s |
| Sustained decode | 16K | 1 | Valid | 58.3 | — | — | — | 1.0 | 0 | 627 | 10.8 s |
| Sustained decode | 16K | 2 | Valid | 74.7 | 58.5 | 110 ms | 17.1 ms | 2.0 | 0 | 2,240 | 30.0 s |
| Sustained decode | 16K | 4 | Valid | 177.7 | — | — | — | 4.0 | 0 | 2,873 | 16.2 s |
| Sustained decode | 16K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 16K | 32 | Valid | 975.2 | 35.0 | 538 ms | 28.5 ms | 32.0 | 0 | 29,246 | 30.0 s |
| Sustained decode | 16K | 64 | Valid | 1,545.0 | 24.9 | 912 ms | 40.2 ms | 64.0 | 0 | 46,337 | 30.0 s |
| Sustained decode | 16K | 128 | Valid | 1,962.8 | 15.3 | 1,682 ms | 65.5 ms | 128.0 | 0 | 58,886 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 57.8 | — | — | — | 1.0 | 0 | 840 | 14.5 s |
| Sustained decode | 32K | 2 | Valid | 81.7 | 57.9 | 140 ms | 17.3 ms | 2.0 | 0 | 2,450 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 127.0 | 58.3 | 231 ms | 17.1 ms | 4.0 | 0 | 3,810 | 30.0 s |
| Sustained decode | 32K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 16 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 32K | 32 | Valid | 967.6 | 32.8 | 892 ms | 30.4 ms | 32.0 | 0 | 29,009 | 30.0 s |
| Sustained decode | 32K | 64 | Valid | 1,428.7 | 22.5 | 1,638 ms | 44.5 ms | 64.0 | 0 | 42,862 | 30.0 s |
| Sustained decode | 32K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 1 | Valid | 56.5 | — | — | — | 1.0 | 0 | 622 | 11.0 s |
| Sustained decode | 64K | 2 | Valid | 113.1 | 56.6 | 235 ms | 17.7 ms | 2.0 | 0 | 3,392 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 137.6 | — | — | — | 4.0 | 0 | 2,868 | 20.8 s |
| Sustained decode | 64K | 8 | Valid | 316.1 | 48.1 | 482 ms | 20.8 ms | 8.0 | 0 | 9,482 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 534.5 | 40.4 | 698 ms | 24.8 ms | 16.0 | 0 | 16,035 | 30.0 s |
| Sustained decode | 64K | 32 | Valid | 897.2 | 28.8 | 1,468 ms | 34.7 ms | 32.0 | 0 | 26,899 | 30.0 s |
| Sustained decode | 64K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 64K | 128 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 1 | Valid | 54.9 | — | — | — | 1.0 | 0 | 567 | 10.3 s |
| Sustained decode | 123K | 2 | Valid | 89.8 | — | — | — | 2.0 | 0 | 1,141 | 12.7 s |
| Sustained decode | 123K | 4 | Valid | 179.4 | — | — | — | 4.0 | 0 | 2,869 | 16.0 s |
| Sustained decode | 123K | 8 | Looped | — | — | — | — | — | 0 | 0 | — |
| Sustained decode | 123K | 16 | Valid | 468.3 | 35.1 | 1,575 ms | 28.5 ms | 16.0 | 0 | 14,049 | 30.0 s |
| Sustained decode | 123K | 32 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 64 | Over capacity | — | — | — | — | — | — | — | — |
| Sustained decode | 123K | 128 | Over capacity | — | — | — | — | — | — | — | — |
Looped: the model kept repeating the same text, so the benchmark discarded this measurement. Over capacity: not run because this many requests at this context would exceed the server's KV cache.