Speed test matrix
Muse-Glimmer-30B on Intel Arc Pro B65 32GB ×4
vllm · 0.26.1.dev0+xpu.llmscaler-1.5 · FP84 contexts (0–64K) × 5 concurrency levels (1–16) · prefill at 8K, 32K, 64K
Hardware
- Type
- Discrete GPU
- GPU
- Intel Arc Pro B65
- GPU count
- 4
- VRAM
- 128 GB (4 × 32 GB)
- CPU
- AMD EPYC 7343 16-Core Processor
- RAM
- 499 GB
- OS
- Ubuntu 26.04.1 LTS
Engine
- Engine
- vllm
- Version
- 0.26.1.dev0+xpu.llmscaler-1.5
- Quantization
- FP8
- Build
- intel/llm-scaler-vllm:0.26.0-b2@sha256:52218ad85513ab6686d4c090c83c2bd8c5b02423c63aa4dabd41837fe641fe3b
Run
- Served model
- muse
- Revision
- a4e59da52a7bc87ae7251dd5545c0dd437c44b68
- Benchmark tool
- llm-inference-bench v0.8.0
- Decode test
- Fixed duration
- Prefill mode
- Measured during decode warm-up
- Time per measurement
- 30 s
- Max output
- 256 tokens
Launch command
docker run --rm -d --name lsv-muse-if8-tp4-ctx-stable --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-ctx-20261019/session-a/muse-if8-tp4-ctx-stable/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.26.0-b2@sha256:52218ad85513ab6686d4c090c83c2bd8c5b02423c63aa4dabd41837fe641fe3b serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 131072 --max-num-seqs 16 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --attention-backend FLASH_ATTN --cpu-offload-gb 0 --port 8000Prefill
prompt tok/s by context length8K context2,6793,059 ms TTFT
32K context2,62512,259 ms TTFT
64K context2,53925,268 ms TTFT
Sustained decode
total tok/s · rows are context length, columns are simultaneous requests| Context | 1 | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| 0 | 48.898 ms TTFT | 51.4141 ms TTFT | 97.1173 ms TTFT | 184.3244 ms TTFT | 343.7414 ms TTFT |
| 8K | 47.6127 ms TTFT | 50.9158 ms TTFT | 95.8171 ms TTFT | 179.8246 ms TTFT | 336.4306 ms TTFT |
| 32K | 46.4189 ms TTFT | 50.0238 ms TTFT | 92.7268 ms TTFT | 173.3382 ms TTFT | 306.5543 ms TTFT |
| 64K | 45.1278 ms TTFT | 49.0319 ms TTFT | 90.1433 ms TTFT | 162.7589 ms TTFT | 271.5753 ms TTFT |
TTFT is measured after the benchmark warms the server's prompt cache, so it mostly excludes prefill.
All measurements
Latencies are medians. Total tok/s is output across all simultaneous requests.| Phase | Context | Requests at once | Status | Total tok/s | Per request tok/s | TTFT | Time between tokens | Average active requests | Completed requests | Output tokens | Measured for |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Prefill | 8K | 1 | Valid | 2,679.0 | — | 3,059 ms | — | — | 1 | — | — |
| Prefill | 32K | 1 | Valid | 2,625.0 | — | 12,259 ms | — | — | 1 | — | — |
| Prefill | 64K | 1 | Valid | 2,539.0 | — | 25,268 ms | — | — | 1 | — | — |
| Sustained decode | 0 | 1 | Valid | 48.8 | 49.4 | 98 ms | 20.2 ms | 1.0 | 6 | 1,462 | 30.0 s |
| Sustained decode | 0 | 2 | Valid | 51.4 | 26.0 | 141 ms | 38.5 ms | 2.0 | 6 | 1,543 | 30.0 s |
| Sustained decode | 0 | 4 | Valid | 97.1 | 24.6 | 173 ms | 40.7 ms | 3.9 | 12 | 2,912 | 30.0 s |
| Sustained decode | 0 | 8 | Valid | 184.3 | 23.5 | 244 ms | 42.6 ms | 8.0 | 24 | 5,528 | 30.0 s |
| Sustained decode | 0 | 16 | Valid | 343.7 | 22.0 | 414 ms | 45.4 ms | 16.0 | 32 | 10,304 | 30.0 s |
| Sustained decode | 8K | 1 | Valid | 47.6 | 48.6 | 127 ms | 20.6 ms | 1.0 | 6 | 1,427 | 30.0 s |
| Sustained decode | 8K | 2 | Valid | 50.9 | 25.7 | 158 ms | 38.9 ms | 2.0 | 6 | 1,526 | 30.0 s |
| Sustained decode | 8K | 4 | Valid | 95.8 | 24.3 | 171 ms | 41.2 ms | 4.0 | 12 | 2,874 | 30.0 s |
| Sustained decode | 8K | 8 | Valid | 179.8 | 23.0 | 246 ms | 43.5 ms | 7.9 | 24 | 5,393 | 30.0 s |
| Sustained decode | 8K | 16 | Valid | 336.4 | 21.3 | 306 ms | 46.9 ms | 15.6 | 32 | 10,081 | 30.0 s |
| Sustained decode | 32K | 1 | Valid | 46.4 | 47.9 | 189 ms | 20.9 ms | 0.9 | 6 | 1,391 | 30.0 s |
| Sustained decode | 32K | 2 | Valid | 50.0 | 25.4 | 238 ms | 39.3 ms | 1.9 | 6 | 1,500 | 30.0 s |
| Sustained decode | 32K | 4 | Valid | 92.7 | 23.7 | 268 ms | 42.2 ms | 4.0 | 12 | 2,780 | 30.0 s |
| Sustained decode | 32K | 8 | Valid | 173.3 | 22.1 | 382 ms | 45.2 ms | 8.0 | 16 | 5,196 | 30.0 s |
| Sustained decode | 32K | 16 | Valid | 306.5 | 19.7 | 543 ms | 50.9 ms | 15.7 | 32 | 9,180 | 30.0 s |
| Sustained decode | 64K | 1 | Valid | 45.1 | 47.1 | 278 ms | 21.2 ms | 1.0 | 5 | 1,352 | 30.0 s |
| Sustained decode | 64K | 2 | Valid | 49.0 | 25.2 | 319 ms | 39.7 ms | 1.9 | 6 | 1,470 | 30.0 s |
| Sustained decode | 64K | 4 | Valid | 90.1 | 23.1 | 433 ms | 43.3 ms | 3.9 | 11 | 2,702 | 30.0 s |
| Sustained decode | 64K | 8 | Valid | 162.7 | 21.0 | 589 ms | 47.7 ms | 7.8 | 16 | 4,876 | 30.0 s |
| Sustained decode | 64K | 16 | Valid | 271.5 | 17.9 | 753 ms | 55.7 ms | 15.5 | 32 | 8,142 | 30.0 s |