imthekaiser

Model

Qwen/Qwen3-4B

Display name

Qwen3-4B

Base model

Qwen3-4B-Base

Revision

1cfa9a7208912126459214e8b04321603b3df60c

Family

Qwen

Parameters

4B

Active params

—

MoE

no

Output

84.5 tok/s

Prefill

6,935 tok/s

Total

180 tok/s

TTFT

42.7 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

296

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q34-u-tp2-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q34-u-tp2-m0/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3-4B --host 0.0.0.0 --served-model-name q34 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q34-u-tp2-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

KASA B65 rig. Config q34-u-tp2-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 84.5, 84.5, 84.5 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 1:47 PM

Last edited

—

Model

Qwen/Qwen3-4B

Display name

Qwen3-4B

Base model

Qwen3-4B-Base

Revision

1cfa9a7208912126459214e8b04321603b3df60c

Family

Qwen

Parameters

4B

Active params

—

MoE

no

Output

62.4 tok/s

Prefill

5,187 tok/s

Total

133 tok/s

TTFT

57.1 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

296

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q34-u-tp1-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q34-u-tp1-m0/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3-4B --host 0.0.0.0 --served-model-name q34 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q34-u-tp1-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

KASA B65 rig. Config q34-u-tp1-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 62.4, 62.4, 62.4 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 1:47 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

227 tok/s

Prefill

4,826 tok/s

Total

473 tok/s

TTFT

63.0 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-if8-tp4-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-if8-tp4-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-orn9-if8-tp4-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config orn9-if8-tp4-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 227.4, 227.1, 227.3 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

136 tok/s

Prefill

3,615 tok/s

Total

283 tok/s

TTFT

83.0 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

300

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-if8-tp2-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-if8-tp2-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-orn9-if8-tp2-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config orn9-if8-tp2-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 135.5, 137.5, 135.5 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

103 tok/s

Prefill

2,307 tok/s

Total

214 tok/s

TTFT

131 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

303

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-if8-tp1-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-if8-tp1-m4/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-orn9-if8-tp1-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config orn9-if8-tp1-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 102.6, 99.8, 108.1 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

175 tok/s

Prefill

5,310 tok/s

Total

370 tok/s

TTFT

57.3 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-u-tp4-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-u-tp4-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-orn9-u-tp4-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config orn9-u-tp4-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 175.3, 172.9, 175.5 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

103 tok/s

Prefill

4,125 tok/s

Total

218 tok/s

TTFT

72.7 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

300

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-u-tp2-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-u-tp2-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-orn9-u-tp2-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config orn9-u-tp2-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 100.1, 102.9, 102.9 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

77.7 tok/s

Prefill

3,291 tok/s

Total

167 tok/s

TTFT

93.0 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

306

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-u-tp1-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-u-tp1-m4/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-orn9-u-tp1-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config orn9-u-tp1-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 78.9, 76.6, 77.7 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-35B-A3B

Display name

Ornith-1.5-35B-A3B

Base model

—

Revision

10fbf86fed7ecee4a061f8b499a618f46001cac1

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

182 tok/s

Prefill

3,324 tok/s

Total

373 tok/s

TTFT

90.0 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

299

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornb-iq4-tp4-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/ornb-iq4-tp4-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-35B-A3B --host 0.0.0.0 --served-model-name ornb --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-ornb-iq4-tp4-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config ornb-iq4-tp4-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 182.3, 182.2, 178 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-35B-A3B

Display name

Ornith-1.5-35B-A3B

Base model

—

Revision

10fbf86fed7ecee4a061f8b499a618f46001cac1

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

145 tok/s

Prefill

3,366 tok/s

Total

304 tok/s

TTFT

90.9 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

306

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornb-iq4-tp2-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/ornb-iq4-tp2-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-35B-A3B --host 0.0.0.0 --served-model-name ornb --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-ornb-iq4-tp2-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config ornb-iq4-tp2-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 144.9, 139.9, 149.7 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-35B-A3B

Display name

Ornith-1.5-35B-A3B

Base model

—

Revision

10fbf86fed7ecee4a061f8b499a618f46001cac1

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

123 tok/s

Prefill

2,298 tok/s

Total

253 tok/s

TTFT

131 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

302

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.92

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornb-iq4-tp1-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/ornb-iq4-tp1-m4/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-35B-A3B --host 0.0.0.0 --served-model-name ornb --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-ornb-iq4-tp1-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config ornb-iq4-tp1-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 125.6, 118.5, 122.7 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-35B-A3B

Display name

Ornith-1.5-35B-A3B

Base model

—

Revision

10fbf86fed7ecee4a061f8b499a618f46001cac1

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

193 tok/s

Prefill

3,558 tok/s

Total

401 tok/s

TTFT

86.6 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

308

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornb-if8-tp4-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/ornb-if8-tp4-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-35B-A3B --host 0.0.0.0 --served-model-name ornb --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-ornb-if8-tp4-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config ornb-if8-tp4-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 190.9, 193.1, 193.1 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-35B-A3B

Display name

Ornith-1.5-35B-A3B

Base model

—

Revision

10fbf86fed7ecee4a061f8b499a618f46001cac1

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

151 tok/s

Prefill

3,374 tok/s

Total

316 tok/s

TTFT

90.1 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornb-if8-tp2-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/ornb-if8-tp2-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-35B-A3B --host 0.0.0.0 --served-model-name ornb --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-ornb-if8-tp2-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config ornb-if8-tp2-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 151.4, 149.6, 153.5 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-35B-A3B

Display name

Ornith-1.5-35B-A3B

Base model

—

Revision

10fbf86fed7ecee4a061f8b499a618f46001cac1

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

181 tok/s

Prefill

3,328 tok/s

Total

371 tok/s

TTFT

90.1 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

300

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

3

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornb-u-tp4-m3 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/ornb-u-tp4-m3/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-35B-A3B --host 0.0.0.0 --served-model-name ornb --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":3\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:3\} --rm --d --name lsv-ornb-u-tp4-m3 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config ornb-u-tp4-m3 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 181, 178.8, 180.9 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

meta-models/Muse-Glimmer-30B

Display name

Muse-Glimmer-30B

Base model

—

Revision

a4e59da52a7bc87ae7251dd5545c0dd437c44b68

Family

—

Parameters

30B

Active params

—

MoE

no

Output

401 tok/s

Prefill

2,083 tok/s

Total

733 tok/s

TTFT

155 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

323

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

FLASH_ATTN

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

DFlash

Spec model

—

Spec draft model

—

Spec tokens

15

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-muse-if8-tp4-df15 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/muse-if8-tp4-df15/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets -e VLLM_USE_V2_MODEL_RUNNER=1 --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"dflash\"\,\"model\":\"/llm/models/Muse-Glimmer-30B-assistant\"\,\"num_speculative_tokens\":15\} --attention-backend FLASH_ATTN --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\dflash\\,\model\:\/llm/models/Muse-Glimmer-30B-assistant\\,\num_speculative_tokens\:15\} --rm --d --name lsv-muse-if8-tp4-df15 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --e VLLM_USE_V2_MODEL_RUNNER=1 --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config muse-if8-tp4-df15 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 401.4, 403.6, 401.4 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:50 PM

Last edited

—

Model

meta-models/Muse-Glimmer-30B

Display name

Muse-Glimmer-30B

Base model

—

Revision

a4e59da52a7bc87ae7251dd5545c0dd437c44b68

Family

—

Parameters

30B

Active params

—

MoE

no

Output

270 tok/s

Prefill

1,255 tok/s

Total

483 tok/s

TTFT

259 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

325

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

FLASH_ATTN

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

DFlash

Spec model

—

Spec draft model

—

Spec tokens

15

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-muse-if8-tp2-df15 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/muse-if8-tp2-df15/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets -e VLLM_USE_V2_MODEL_RUNNER=1 --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"dflash\"\,\"model\":\"/llm/models/Muse-Glimmer-30B-assistant\"\,\"num_speculative_tokens\":15\} --attention-backend FLASH_ATTN --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\dflash\\,\model\:\/llm/models/Muse-Glimmer-30B-assistant\\,\num_speculative_tokens\:15\} --rm --d --name lsv-muse-if8-tp2-df15 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --e VLLM_USE_V2_MODEL_RUNNER=1 --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config muse-if8-tp2-df15 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 270, 254.5, 270.2 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:49 PM

Last edited

—

Model

meta-models/Muse-Glimmer-30B

Display name

Muse-Glimmer-30B

Base model

—

Revision

a4e59da52a7bc87ae7251dd5545c0dd437c44b68

Family

—

Parameters

30B

Active params

—

MoE

no

Output

26.8 tok/s

Prefill

1,720 tok/s

Total

59.3 tok/s

TTFT

186 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

320

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-muse-u-tp4-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/muse-u-tp4-m0/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-muse-u-tp4-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config muse-u-tp4-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 26.8, 26.8, 26.8 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:49 PM

Last edited

—

Model

meta-models/Muse-Glimmer-30B

Display name

Muse-Glimmer-30B

Base model

—

Revision

a4e59da52a7bc87ae7251dd5545c0dd437c44b68

Family

—

Parameters

30B

Active params

—

MoE

no

Output

14.6 tok/s

Prefill

1,888 tok/s

Total

33.0 tok/s

TTFT

173 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

326

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-muse-u-tp2-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/muse-u-tp2-m0/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-muse-u-tp2-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config muse-u-tp2-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 14.6, 14.6, 14.6 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:49 PM

Last edited

—

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

114 tok/s

Prefill

2,010 tok/s

Total

235 tok/s

TTFT

153 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

308

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g31-iq4-tp4-g4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g31-iq4-tp4-g4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"gemma4_mtp\"\,\"model\":\"/llm/models/gemma-4-31B-it-assistant\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\gemma4_mtp\\,\model\:\/llm/models/gemma-4-31B-it-assistant\\,\num_speculative_tokens\:4\} --rm --d --name lsv-g31-iq4-tp4-g4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config g31-iq4-tp4-g4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 111.9, 115.5, 113.5 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:49 PM

Last edited

—

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

88.9 tok/s

Prefill

1,378 tok/s

Total

181 tok/s

TTFT

220 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

303

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

2

Pipeline parallel