Speed test matrix
Muse-Glimmer-30B on Intel Arc Pro B65 32GB ×4
vllm · 0.31.0+xpu.llmscaler · FP84 contexts (0–64K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.31.0+xpu.llmscaler
- Quantization
- FP8
- Build
- intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704
Run
- Served model
- muse
- Revision
- a4e59da52a7bc87ae7251dd5545c0dd437c44b68
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-muse-if8-tp4-ctx --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/muse-if8-tp4-ctx/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 131072 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --attention-backend FLASH_ATTN --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context2,7093,025 ms TTFT
32K context2,64512,162 ms TTFT
64K context2,55025,157 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 36.3103 ms TTFT | 68.7149 ms TTFT | 136.0184 ms TTFT | 258.4250 ms TTFT | 476.4414 ms TTFT |
| 8K | 35.9127 ms TTFT | 68.1160 ms TTFT | 134.5186 ms TTFT | 250.6194 ms TTFT | 466.5304 ms TTFT |
| 32K | 35.3179 ms TTFT | 66.7210 ms TTFT | 128.4264 ms TTFT | 240.3339 ms TTFT | 425.1446 ms TTFT |
| 64K | 34.4254 ms TTFT | 64.5317 ms TTFT | 122.3467 ms TTFT | 225.5518 ms TTFT | 372.4592 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,709.0 | — | 3,025 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 2,645.0 | — | 12,162 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 2,550.0 | — | 25,157 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 36.3 | 36.7 | 103 ms | 27.2 ms | 1.0 | 5 | 1,087 | 29.9 s |
| Sustained decode | 0 | 2 | Valid | 68.7 | 34.9 | 149 ms | 28.6 ms | 1.9 | 8 | 2,062 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 136.0 | 34.7 | 184 ms | 28.8 ms | 3.9 | 16 | 4,080 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 258.4 | 33.3 | 250 ms | 30.1 ms | 8.0 | 32 | 7,744 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 476.4 | 31.4 | 414 ms | 31.9 ms | 15.5 | 64 | 14,288 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 35.9 | 36.4 | 127 ms | 27.5 ms | 1.0 | 4 | 1,077 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 68.1 | 34.6 | 160 ms | 28.9 ms | 1.9 | 8 | 2,042 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 134.5 | 34.3 | 186 ms | 29.1 ms | 4.0 | 16 | 4,032 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 250.6 | 32.1 | 194 ms | 31.2 ms | 8.0 | 32 | 7,515 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 466.5 | 30.0 | 304 ms | 33.3 ms | 15.7 | 48 | 13,994 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 35.3 | 36.0 | 179 ms | 27.8 ms | 1.0 | 4 | 1,058 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 66.7 | 34.1 | 210 ms | 29.3 ms | 2.0 | 8 | 2,000 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 128.4 | 33.1 | 264 ms | 30.2 ms | 4.0 | 16 | 3,851 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 240.3 | 31.0 | 339 ms | 32.2 ms | 7.9 | 32 | 7,195 | 29.9 s |
| Sustained decode | 32K | 16 | Valid | 425.1 | 27.6 | 446 ms | 36.2 ms | 15.6 | 48 | 12,751 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 34.4 | 35.5 | 254 ms | 28.2 ms | 0.9 | 4 | 1,032 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 64.5 | 33.5 | 317 ms | 29.8 ms | 1.9 | 8 | 1,934 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 122.3 | 32.2 | 467 ms | 31.0 ms | 3.9 | 16 | 3,668 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 225.5 | 29.5 | 518 ms | 33.9 ms | 7.7 | 24 | 6,762 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 372.4 | 24.8 | 592 ms | 40.4 ms | 15.3 | 48 | 11,165 | 30.0 s |