Speed test matrix
Muse-Glimmer-30B on Intel Arc Pro B65 32GB ×4
vllm · 0.32.0 · BF164 contexts (0–64K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.32.0
- Quantization
- BF16
- Build
- vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c
Run
- Served model
- muse
- Revision
- a4e59da52a7bc87ae7251dd5545c0dd437c44b68
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-muse-u-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/muse-u-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:v0.32.0@sha256:20509fabe50dda62c09c0786d1a6effa299d3dbdddcc49331b93e65144652a8c serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 131072 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context3,8982,102 ms TTFT
32K context3,7808,512 ms TTFT
64K context3,58817,879 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 26.770 ms TTFT | 52.8133 ms TTFT | 104.5175 ms TTFT | 206.3230 ms TTFT | 399.3317 ms TTFT |
| 8K | 26.671 ms TTFT | 52.6126 ms TTFT | 103.4173 ms TTFT | 203.0204 ms TTFT | 391.5190 ms TTFT |
| 32K | 26.1175 ms TTFT | 51.4226 ms TTFT | 99.7295 ms TTFT | 192.4366 ms TTFT | 363.4418 ms TTFT |
| 64K | 25.6268 ms TTFT | 50.3319 ms TTFT | 96.0498 ms TTFT | 183.0555 ms TTFT | 333.0608 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 3,898.0 | — | 2,102 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 3,780.0 | — | 8,512 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 3,588.0 | — | 17,879 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 26.7 | 26.8 | 70 ms | 37.3 ms | 1.0 | 3 | 801 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 52.8 | 26.7 | 133 ms | 37.5 ms | 1.9 | 6 | 1,582 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 104.5 | 26.5 | 175 ms | 37.8 ms | 4.0 | 12 | 3,132 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 206.3 | 26.3 | 230 ms | 38.1 ms | 8.0 | 24 | 6,184 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 399.3 | 25.7 | 317 ms | 39.0 ms | 16.0 | 48 | 11,968 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 26.6 | 26.7 | 71 ms | 37.5 ms | 1.0 | 3 | 797 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 52.6 | 26.5 | 126 ms | 37.7 ms | 2.0 | 6 | 1,576 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 103.4 | 26.2 | 173 ms | 38.2 ms | 3.9 | 12 | 3,099 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 203.0 | 25.7 | 204 ms | 38.9 ms | 8.0 | 24 | 6,086 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 391.5 | 24.9 | 190 ms | 40.2 ms | 15.6 | 48 | 11,744 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 26.1 | 26.4 | 175 ms | 37.8 ms | 1.0 | 3 | 782 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 51.4 | 26.2 | 226 ms | 38.2 ms | 2.0 | 6 | 1,542 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 99.7 | 25.6 | 295 ms | 39.1 ms | 3.8 | 12 | 2,988 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 192.4 | 24.8 | 366 ms | 40.3 ms | 7.9 | 24 | 5,768 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 363.4 | 23.2 | 418 ms | 43.1 ms | 15.8 | 40 | 10,892 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 25.6 | 26.2 | 268 ms | 38.2 ms | 1.0 | 3 | 767 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 50.3 | 25.8 | 319 ms | 38.7 ms | 2.0 | 6 | 1,508 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 96.0 | 25.0 | 498 ms | 40.0 ms | 3.9 | 12 | 2,880 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 183.0 | 23.7 | 555 ms | 42.2 ms | 7.9 | 22 | 5,463 | 29.8 s |
| Sustained decode | 64K | 16 | Valid | 333.0 | 21.4 | 608 ms | 46.8 ms | 15.7 | 32 | 9,982 | 30.0 s |