imthekaiser

Model

Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8

Display name

Qwen3-Coder-30B-A3B-Instruct-FP8

Base model

—

Revision

dcaee4d4dfc5ee71ad501f01f530e5652438fde0

Family

Qwen

Parameters

30B

Active params

3B

MoE

yes

Output

48.9 tok/s

Prefill

1,424 tok/s

Total

102 tok/s

TTFT

210 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

299

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-qc30-u-tp4-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/qc30-u-tp4-m0/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3-Coder-30B-A3B-Instruct-FP8 --host 0.0.0.0 --served-model-name qc30 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-qc30-u-tp4-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

KASA B65 rig. Config qc30-u-tp4-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 48.9, 48.9, 48.9 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:49 PM

Last edited

—

Model

Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8

Display name

Qwen3-Coder-30B-A3B-Instruct-FP8

Base model

—

Revision

dcaee4d4dfc5ee71ad501f01f530e5652438fde0

Family

Qwen

Parameters

30B

Active params

3B

MoE

yes

Output

81.9 tok/s

Prefill

3,833 tok/s

Total

174 tok/s

TTFT

77.8 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

298

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-qc30-u-tp2-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/qc30-u-tp2-m0/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3-Coder-30B-A3B-Instruct-FP8 --host 0.0.0.0 --served-model-name qc30 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-qc30-u-tp2-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

KASA B65 rig. Config qc30-u-tp2-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 81.9, 81.9, 81.8 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 12:49 PM

Last edited

—

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

72.8 tok/s

Prefill

857 tok/s

Total

145 tok/s

TTFT

356 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

305

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.92

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g31-iq4-tp1-g4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g31-iq4-tp1-g4/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"gemma4_mtp\"\,\"model\":\"/llm/models/gemma-4-31B-it-assistant\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\gemma4_mtp\\,\model\:\/llm/models/gemma-4-31B-it-assistant\\,\num_speculative_tokens\:4\} --rm --d --name lsv-g31-iq4-tp1-g4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config g31-iq4-tp1-g4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 75.3, 72.8, 72.7 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:49 AM

Last edited

—

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

88.9 tok/s

Prefill

1,884 tok/s

Total

185 tok/s

TTFT

162 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

305

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g31-if8-tp4-g4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g31-if8-tp4-g4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"gemma4_mtp\"\,\"model\":\"/llm/models/gemma-4-31B-it-assistant\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\gemma4_mtp\\,\model\:\/llm/models/gemma-4-31B-it-assistant\\,\num_speculative_tokens\:4\} --rm --d --name lsv-g31-if8-tp4-g4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config g31-if8-tp4-g4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 88.9, 93.1, 85.8 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:49 AM

Last edited

—

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

61.0 tok/s

Prefill

1,275 tok/s

Total

127 tok/s

TTFT

240 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

306

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g31-if8-tp2-g4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g31-if8-tp2-g4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"gemma4_mtp\"\,\"model\":\"/llm/models/gemma-4-31B-it-assistant\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\gemma4_mtp\\,\model\:\/llm/models/gemma-4-31B-it-assistant\\,\num_speculative_tokens\:4\} --rm --d --name lsv-g31-if8-tp2-g4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config g31-if8-tp2-g4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 61, 60.9, 62.1 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:49 AM

Last edited

—

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

22.4 tok/s

Prefill

2,079 tok/s

Total

48.7 tok/s

TTFT

147 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

305

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g31-u-tp4-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g31-u-tp4-m0/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-g31-u-tp4-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config g31-u-tp4-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 22.4, 22.4, 22.4 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:49 AM

Last edited

—

Model

google/gemma-4-26B-A4B-it

Display name

gemma-4-26B-A4B-it

Base model

gemma-4-26B-A4B

Revision

4d7ae4984b7db7de8f8457170b3f1a419ee76d52

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

166 tok/s

Prefill

3,793 tok/s

Total

349 tok/s

TTFT

81.2 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

308

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g26-if8-tp4-g4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g26-if8-tp4-g4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-26B-A4B-it --host 0.0.0.0 --served-model-name g26 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"gemma4_mtp\"\,\"model\":\"/llm/models/gemma-4-26B-A4B-it-assistant\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\gemma4_mtp\\,\model\:\/llm/models/gemma-4-26B-A4B-it-assistant\\,\num_speculative_tokens\:4\} --rm --d --name lsv-g26-if8-tp4-g4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config g26-if8-tp4-g4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 165.9, 166.1, 169.1 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:49 AM

Last edited

—

Model

google/gemma-4-26B-A4B-it

Display name

gemma-4-26B-A4B-it

Base model

gemma-4-26B-A4B

Revision

4d7ae4984b7db7de8f8457170b3f1a419ee76d52

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

135 tok/s

Prefill

3,084 tok/s

Total

282 tok/s

TTFT

99.2 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

306

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g26-if8-tp2-g4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g26-if8-tp2-g4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-26B-A4B-it --host 0.0.0.0 --served-model-name g26 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"gemma4_mtp\"\,\"model\":\"/llm/models/gemma-4-26B-A4B-it-assistant\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\gemma4_mtp\\,\model\:\/llm/models/gemma-4-26B-A4B-it-assistant\\,\num_speculative_tokens\:4\} --rm --d --name lsv-g26-if8-tp2-g4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config g26-if8-tp2-g4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 134.6, 134.5, 130.2 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

google/gemma-4-12B-it

Display name

gemma-4-12B-it

Base model

gemma-4-12B

Revision

707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7

Family

Gemma

Parameters

12B

Active params

—

MoE

no

Output

30.3 tok/s

Prefill

2,938 tok/s

Total

65.8 tok/s

TTFT

103 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g12-u-tp2-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g12-u-tp2-m0/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-12B-it --host 0.0.0.0 --served-model-name g12 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-g12-u-tp2-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config g12-u-tp2-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 30.3, 30.3, 30.3 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

google/gemma-4-12B-it

Display name

gemma-4-12B-it

Base model

gemma-4-12B

Revision

707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7

Family

Gemma

Parameters

12B

Active params

—

MoE

no

Output

21.7 tok/s

Prefill

1,999 tok/s

Total

46.9 tok/s

TTFT

152 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

303

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g12-u-tp1-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g12-u-tp1-m0/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-12B-it --host 0.0.0.0 --served-model-name g12 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-g12-u-tp1-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config g12-u-tp1-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 21.7, 21.7, 21.7 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

devan-carlin/Qwen3.8-Flash-Next-W4A16

Display name

Qwen3.8-Flash-Next-W4A16

Base model

Qwen3.8-Flash-Next

Revision

40b8f18df4d4a32cb6e687a51c78207e5e438522

Family

Qwen

Parameters

180B

Active params

—

MoE

yes

Output

477 tok/s

Prefill

29.6 tok/s

Total

1,117 tok/s

TTFT

11,697 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-fn4-u-tp4-eager-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/fn4-u-tp4-eager-c128/session:/bench -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:/workspace/vllm/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/__init__.py:/workspace/vllm/vllm/models/qwen4_exp/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/__init__.py:/workspace/vllm/vllm/models/qwen4_exp/xpu/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/model.py:/workspace/vllm/vllm/models/qwen4_exp/xpu/model.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/mtp.py:/workspace/vllm/vllm/models/qwen4_exp/xpu/mtp.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/spec_decode/llm_base_proposer.py:/workspace/vllm/vllm/v1/spec_decode/llm_base_proposer.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/worker/gpu/spec_decode/speculator.py:/workspace/vllm/vllm/v1/worker/gpu/spec_decode/speculator.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/__init__.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/__init__.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/xpu/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/model.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/xpu/model.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/mtp.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/xpu/mtp.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/spec_decode/llm_base_proposer.py:/opt/venv/lib/python3.12/site-packages/vllm/v1/spec_decode/llm_base_proposer.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/worker/gpu/spec_decode/speculator.py:/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/speculator.py:ro -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=1 -e DISABLE_ESIMD_INT4_GEMM=1 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets -e PLE_TABLE_PATH=/llm/models/Qwen3.8-Flash-Next-W4A16/ple_table_qwen4exp.pt --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 s

Extra flags

--rm --d --name lsv-fn4-u-tp4-eager-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=1 --e DISABLE_ESIMD_INT4_GEMM=1 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --e PLE_TABLE_PATH=/llm/models/Qwen3.8-Flash-Next-W4A16/ple_table_qwen4exp.pt --entrypoint vllm

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 476.8 tok/s is the aggregate decode rate across all 128 streams, about 3.7 tok/s per stream, and the 11697 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 476.8, 479.8, 476.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config fn4-u-tp4-eager-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

devan-carlin/Qwen3.8-Flash-Next-W4A16

Display name

Qwen3.8-Flash-Next-W4A16

Base model

Qwen3.8-Flash-Next

Revision

40b8f18df4d4a32cb6e687a51c78207e5e438522

Family

Qwen

Parameters

180B

Active params

—

MoE

yes

Output

296 tok/s

Prefill

32.0 tok/s

Total

692 tok/s

TTFT

10,833 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

64

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

64

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-fn4-u-tp4-eager-c64 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/fn4-u-tp4-eager-c64/session:/bench -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:/workspace/vllm/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/__init__.py:/workspace/vllm/vllm/models/qwen4_exp/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/__init__.py:/workspace/vllm/vllm/models/qwen4_exp/xpu/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/model.py:/workspace/vllm/vllm/models/qwen4_exp/xpu/model.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/mtp.py:/workspace/vllm/vllm/models/qwen4_exp/xpu/mtp.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/spec_decode/llm_base_proposer.py:/workspace/vllm/vllm/v1/spec_decode/llm_base_proposer.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/worker/gpu/spec_decode/speculator.py:/workspace/vllm/vllm/v1/worker/gpu/spec_decode/speculator.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/__init__.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/__init__.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/xpu/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/model.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/xpu/model.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/mtp.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/xpu/mtp.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/spec_decode/llm_base_proposer.py:/opt/venv/lib/python3.12/site-packages/vllm/v1/spec_decode/llm_base_proposer.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/worker/gpu/spec_decode/speculator.py:/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/speculator.py:ro -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=1 -e DISABLE_ESIMD_INT4_GEMM=1 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets -e PLE_TABLE_PATH=/llm/models/Qwen3.8-Flash-Next-W4A16/ple_table_qwen4exp.pt --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 ser

Extra flags

--rm --d --name lsv-fn4-u-tp4-eager-c64 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=1 --e DISABLE_ESIMD_INT4_GEMM=1 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --e PLE_TABLE_PATH=/llm/models/Qwen3.8-Flash-Next-W4A16/ple_table_qwen4exp.pt --entrypoint vllm

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 64 requests sent at once (batch 64, lmx --concurrency 64); 296.1 tok/s is the aggregate decode rate across all 64 streams, about 4.6 tok/s per stream, and the 10833 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 64: 280.7, 296.1, 301.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config fn4-u-tp4-eager-c64, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

devan-carlin/Qwen3.8-Flash-Next-W4A16

Display name

Qwen3.8-Flash-Next-W4A16

Base model

Qwen3.8-Flash-Next

Revision

40b8f18df4d4a32cb6e687a51c78207e5e438522

Family

Qwen

Parameters

180B

Active params

—

MoE

yes

Output

20.9 tok/s

Prefill

854 tok/s

Total

48.0 tok/s

TTFT

409 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

349

Output tokens

256

Prefill tokens

0

Context length

8192

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

3

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-fn4-u-tp4-eager-m3 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/fn4-u-tp4-eager-m3/session:/bench -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:/workspace/vllm/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/__init__.py:/workspace/vllm/vllm/models/qwen4_exp/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/__init__.py:/workspace/vllm/vllm/models/qwen4_exp/xpu/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/model.py:/workspace/vllm/vllm/models/qwen4_exp/xpu/model.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/mtp.py:/workspace/vllm/vllm/models/qwen4_exp/xpu/mtp.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/spec_decode/llm_base_proposer.py:/workspace/vllm/vllm/v1/spec_decode/llm_base_proposer.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/worker/gpu/spec_decode/speculator.py:/workspace/vllm/vllm/v1/worker/gpu/spec_decode/speculator.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:/opt/venv/lib/python3.12/site-packages/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/__init__.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/__init__.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/xpu/__init__.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/model.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/xpu/model.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/models/qwen4_exp/xpu/mtp.py:/opt/venv/lib/python3.12/site-packages/vllm/models/qwen4_exp/xpu/mtp.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/spec_decode/llm_base_proposer.py:/opt/venv/lib/python3.12/site-packages/vllm/v1/spec_decode/llm_base_proposer.py:ro -v /models/bench/lmx-models-s12a/stage/src/ai/bare-metal/gpu/2026-10-15-b65-localmaxxing-models/overlays/pr61607/head/vllm/v1/worker/gpu/spec_decode/speculator.py:/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/gpu/spec_decode/speculator.py:ro -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=1 -e DISABLE_ESIMD_INT4_GEMM=1 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets -e PLE_TABLE_PATH=/llm/models/Qwen3.8-Flash-Next-W4A16/ple_table_qwen4exp.pt --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve

Extra flags

--rm --d --name lsv-fn4-u-tp4-eager-m3 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=1 --e DISABLE_ESIMD_INT4_GEMM=1 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --e PLE_TABLE_PATH=/llm/models/Qwen3.8-Flash-Next-W4A16/ple_table_qwen4exp.pt --entrypoint vllm

Notes

KASA B65 rig. Config fn4-u-tp4-eager-m3 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 20.9, 21.4, 20.2 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

Qwen/Qwen3.6-35B-A3B

Display name

Qwen3.6-35B-A3B

Base model

—

Revision

995ad96eacd98c81ed38be0c5b274b04031597b0

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

234 tok/s

Prefill

3,430 tok/s

Total

476 tok/s

TTFT

88.9 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

305

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

5

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36x-if8-tp4-m5 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q36x-if8-tp4-m5/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-35B-A3B --host 0.0.0.0 --served-model-name q36x --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":5\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:5\} --rm --d --name lsv-q36x-if8-tp4-m5 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q36x-if8-tp4-m5 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 233.8, 234.1, 234.2 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

Qwen/Qwen3.6-35B-A3B

Display name

Qwen3.6-35B-A3B

Base model

—

Revision

995ad96eacd98c81ed38be0c5b274b04031597b0

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

176 tok/s

Prefill

3,443 tok/s

Total

364 tok/s

TTFT

88.3 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36x-if8-tp2-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q36x-if8-tp2-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-35B-A3B --host 0.0.0.0 --served-model-name q36x --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q36x-if8-tp2-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q36x-if8-tp2-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 178.5, 176, 175.7 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

Qwen/Qwen3.6-27B-FP8

Display name

Qwen3.6-27B-FP8

Base model

Qwen3.6-27B

Revision

e89b16ebf1988b3d6befa7de50abc2d76f26eb09

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

89.9 tok/s

Prefill

1,674 tok/s

Total

186 tok/s

TTFT

182 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

305

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36e-u-tp4-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q36e-u-tp4-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.6-27B-FP8 --host 0.0.0.0 --served-model-name q36e --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q36e-u-tp4-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q36e-u-tp4-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 89.9, 88.5, 89.9 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

Qwen/Qwen3.6-27B-FP8

Display name

Qwen3.6-27B-FP8

Base model

Qwen3.6-27B

Revision

e89b16ebf1988b3d6befa7de50abc2d76f26eb09

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

51.2 tok/s

Prefill

1,046 tok/s

Total

106 tok/s

TTFT

292 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

305

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36e-u-tp2-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q36e-u-tp2-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.6-27B-FP8 --host 0.0.0.0 --served-model-name q36e --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q36e-u-tp2-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q36e-u-tp2-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 51.2, 50.4, 51.2 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

Qwen/Qwen3.6-27B

Display name

Qwen3.6-27B

Base model

—

Revision

6a9e13bd6fc8f0983b9b99948120bc37f49c13e9

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

158 tok/s

Prefill

2,383 tok/s

Total

318 tok/s

TTFT

125 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

299

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36d-iq4-tp4-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q36d-iq4-tp4-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-27B --host 0.0.0.0 --served-model-name q36d --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q36d-iq4-tp4-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q36d-iq4-tp4-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 155.2, 157.7, 157.6 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

Qwen/Qwen3.6-27B

Display name

Qwen3.6-27B

Base model

—

Revision

6a9e13bd6fc8f0983b9b99948120bc37f49c13e9

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

103 tok/s

Prefill

1,659 tok/s

Total

211 tok/s

TTFT

183 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

303

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36d-iq4-tp2-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q36d-iq4-tp2-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-27B --host 0.0.0.0 --served-model-name q36d --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q36d-iq4-tp2-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q36d-iq4-tp2-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 101.8, 103.4, 103.4 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:48 AM

Last edited

—

Model

Qwen/Qwen3.6-27B

Display name

Qwen3.6-27B

Base model

—

Revision

6a9e13bd6fc8f0983b9b99948120bc37f49c13e9

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

72.8 tok/s

Prefill

962 tok/s

Total

146 tok/s

TTFT

314 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

302

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

1

Pipeline parallel