imthekaiser

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

353 tok/s

Prefill

6.8 tok/s

Total

770 tok/s

TTFT

44,668 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g31-iq4-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/g31-iq4-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-g31-iq4-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 352.7 tok/s is the aggregate decode rate across all 192 streams, about 1.8 tok/s per stream, and the 44668 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 352.1, 352.7, 353.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config g31-iq4-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 3:24 PM

Last edited

—

Model

google/gemma-4-12B-it

Display name

gemma-4-12B-it

Base model

gemma-4-12B

Revision

707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7

Family

Gemma

Parameters

12B

Active params

—

MoE

no

Output

1,239 tok/s

Prefill

51.2 tok/s

Total

2,698 tok/s

TTFT

5,973 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g12-if8-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/g12-if8-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-12B-it --host 0.0.0.0 --served-model-name g12 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-g12-if8-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 1239.4 tok/s is the aggregate decode rate across all 192 streams, about 6.5 tok/s per stream, and the 5973 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 1236.6, 1240.8, 1239.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config g12-if8-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 3:24 PM

Last edited

—

Model

google/gemma-4-12B-it

Display name

gemma-4-12B-it

Base model

gemma-4-12B

Revision

707f0a3b8a3c7ad586ed01e27eafbad8a27dd0f7

Family

Gemma

Parameters

12B

Active params

—

MoE

no

Output

576 tok/s

Prefill

12.4 tok/s

Total

1,257 tok/s

TTFT

24,490 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g12-u-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/g12-u-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-12B-it --host 0.0.0.0 --served-model-name g12 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-g12-u-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 575.8 tok/s is the aggregate decode rate across all 192 streams, about 3.0 tok/s per stream, and the 24490 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 577.2, 575.2, 575.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config g12-u-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 3:24 PM

Last edited

—

Model

dbirks/Qwen3.8-27B-W4A16-AutoRound

Display name

Qwen3.8-27B-W4A16-AutoRound

Base model

Qwen3.8-27B

Revision

1f05c441c4e64ae0549de44fa9ea5a6d43610314

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

1,079 tok/s

Prefill

30.3 tok/s

Total

2,520 tok/s

TTFT

11,424 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q38d-u-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/q38d-u-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=1 -e DISABLE_ESIMD_INT4_GEMM=1 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.8-27B-W4A16-AutoRound-dbirks --host 0.0.0.0 --served-model-name q38d --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-q38d-u-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=1 --e DISABLE_ESIMD_INT4_GEMM=1 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 1078.6 tok/s is the aggregate decode rate across all 192 streams, about 5.6 tok/s per stream, and the 11424 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 1079.5, 1077.5, 1078.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q38d-u-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 3:24 PM

Last edited

—

Model

Qwen/Qwen3-4B

Display name

Qwen3-4B

Base model

Qwen3-4B-Base

Revision

1cfa9a7208912126459214e8b04321603b3df60c

Family

Qwen

Parameters

4B

Active params

—

MoE

no

Output

5,464 tok/s

Prefill

201 tok/s

Total

11,664 tok/s

TTFT

1,467 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

296

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q34-u-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/q34-u-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3-4B --host 0.0.0.0 --served-model-name q34 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-q34-u-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 5463.9 tok/s is the aggregate decode rate across all 192 streams, about 28.5 tok/s per stream, and the 1467 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 5447, 5463.9, 5572.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q34-u-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 3:24 PM

Last edited

—

Model

palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4

Display name

Qwen3.6-35B-A3B-GPTQ-Int4

Base model

Qwen3.6-35B-A3B

Revision

00a66983516f8f8057741221277eed4141c9e431

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

4,081 tok/s

Prefill

156 tok/s

Total

8,798 tok/s

TTFT

1,955 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

GPTQ-4bit

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36g-u-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/q36g-u-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.6-35B-A3B-GPTQ-Int4 --host 0.0.0.0 --served-model-name qwen36-35b --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-q36g-u-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 4081.2 tok/s is the aggregate decode rate across all 192 streams, about 21.3 tok/s per stream, and the 1955 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 4076.4, 4081.2, 4089 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q36g-u-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

3,688 tok/s

Prefill

99.4 tok/s

Total

7,990 tok/s

TTFT

3,048 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-if8-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/orn9-if8-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-orn9-if8-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 3687.7 tok/s is the aggregate decode rate across all 192 streams, about 19.2 tok/s per stream, and the 3048 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 3687.5, 3701, 3687.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-if8-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

3,294 tok/s

Prefill

104 tok/s

Total

7,153 tok/s

TTFT

2,948 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-u-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/orn9-u-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-orn9-u-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 3293.5 tok/s is the aggregate decode rate across all 192 streams, about 17.2 tok/s per stream, and the 2948 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 3308.7, 3293.1, 3293.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-u-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-35B-A3B

Display name

Ornith-1.5-35B-A3B

Base model

—

Revision

10fbf86fed7ecee4a061f8b499a618f46001cac1

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

2,260 tok/s

Prefill

157 tok/s

Total

4,900 tok/s

TTFT

1,945 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornb-iq4-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/ornb-iq4-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-35B-A3B --host 0.0.0.0 --served-model-name ornb --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-ornb-iq4-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 2259.9 tok/s is the aggregate decode rate across all 192 streams, about 11.8 tok/s per stream, and the 1945 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 2269, 2259.9, 2256.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config ornb-iq4-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

amd/gemma-4-26B-A4B-it-w4a16-llmcompressor

Display name

gemma-4-26B-A4B-it-w4a16-llmcompressor

Base model

gemma-4-26B-A4B-it

Revision

5f5d9fa560e8eeabead101c85f1d639e1b5f8716

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

1,538 tok/s

Prefill

90.9 tok/s

Total

3,352 tok/s

TTFT

3,339 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.97

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-gema-u-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/gema-u-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-26B-A4B-it-w4a16 --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.97 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-gema-u-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 1538.4 tok/s is the aggregate decode rate across all 192 streams, about 8.0 tok/s per stream, and the 3339 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 1537.3, 1538.4, 1539.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config gema-u-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic

Display name

gemma-4-26B-A4B-it-FP8-dynamic

Base model

gemma-4-26B-A4B-it

Revision

ed35d7abe5d940da41b4ff06eb482feb0be8cb44

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

342 tok/s

Prefill

7.5 tok/s

Total

746 tok/s

TTFT

40,736 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.97

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-gemr-u-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/gemr-u-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-26B-A4B-it-FP8-dynamic --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.97 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-gemr-u-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 342.3 tok/s is the aggregate decode rate across all 192 streams, about 1.8 tok/s per stream, and the 40736 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 342.3, 344.6, 340.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config gemr-u-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

Qwen/Qwen3.6-35B-A3B

Display name

Qwen3.6-35B-A3B

Base model

—

Revision

995ad96eacd98c81ed38be0c5b274b04031597b0

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

2,384 tok/s

Prefill

158 tok/s

Total

5,170 tok/s

TTFT

1,931 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.9

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36s2-i-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/q36s2-i-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-35B-A3B --host 0.0.0.0 --served-model-name qwen36-35b --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-q36s2-i-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 2383.9 tok/s is the aggregate decode rate across all 192 streams, about 12.4 tok/s per stream, and the 1931 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 2378.9, 2384, 2383.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q36s2-i-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

Qwen/Qwen3.6-27B

Display name

Qwen3.6-27B

Base model

—

Revision

6a9e13bd6fc8f0983b9b99948120bc37f49c13e9

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

532 tok/s

Prefill

27.0 tok/s

Total

1,159 tok/s

TTFT

11,295 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36d-iq4-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/q36d-iq4-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.6-27B --host 0.0.0.0 --served-model-name q36d --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-q36d-iq4-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 531.5 tok/s is the aggregate decode rate across all 192 streams, about 2.8 tok/s per stream, and the 11295 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 531.5, 531.5, 531.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q36d-iq4-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

Qwen/Qwen3.5-9B

Display name

Qwen3.5-9B

Base model

Qwen3.5-9B-Base

Revision

c202236235762e1c871ad0ccb60c8ee5ba337b9a

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

3,695 tok/s

Prefill

107 tok/s

Total

8,013 tok/s

TTFT

2,813 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q359-if8-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/q359-if8-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.5-9B --host 0.0.0.0 --served-model-name q359 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-q359-if8-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 3694.8 tok/s is the aggregate decode rate across all 192 streams, about 19.2 tok/s per stream, and the 2813 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 3699.7, 3693.4, 3694.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q359-if8-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

Qwen/Qwen3.5-9B

Display name

Qwen3.5-9B

Base model

Qwen3.5-9B-Base

Revision

c202236235762e1c871ad0ccb60c8ee5ba337b9a

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

3,294 tok/s

Prefill

118 tok/s

Total

7,158 tok/s

TTFT

2,577 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q359-u-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/q359-u-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.5-9B --host 0.0.0.0 --served-model-name q359 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-q359-u-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 3294.3 tok/s is the aggregate decode rate across all 192 streams, about 17.2 tok/s per stream, and the 2577 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 3295.5, 3294.3, 3291.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q359-u-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

MIRALABS/Ornith-1.5-35B-A3B-W4A16-SYM

Display name

Ornith-1.5-35B-A3B-W4A16-SYM

Base model

Ornith-1.5-35B-A3B

Revision

08e6d31037779e5ecc02094a9b8c76eecff9db45

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

2,557 tok/s

Prefill

151 tok/s

Total

5,542 tok/s

TTFT

2,022 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornw-u-tp1r3-m0-c192-r0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12c/arms/ornw-u-tp1r3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-35B-A3B-W4A16 --host 0.0.0.0 --served-model-name ornith-15 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8001

Extra flags

--rm --d --name lsv-ornw-u-tp1r3-m0-c192-r0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 2557.2 tok/s is the aggregate decode rate across all 192 streams, about 13.3 tok/s per stream, and the 2022 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 independent one-card vLLM servers (one per card; identical but for card and port) behind a round-robin nginx router; the command shown is the first server's; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 2560.2, 2557.2, 2506.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config ornw-u-tp1r3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

zai-org/GLM-4.7-Flash

Display name

GLM-4.7-Flash

Base model

—

Revision

7dd20894a642a0aa287e9827cb1a1f7f91386b67

Family

—

Parameters

31B

Active params

—

MoE

yes

Output

214 tok/s

Prefill

72.2 tok/s

Total

446 tok/s

TTFT

3,864 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

278

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-glm47-u-tp4-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12b/arms/glm47-u-tp4-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/GLM-4.7-Flash --host 0.0.0.0 --served-model-name glm47 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-glm47-u-tp4-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 214 tok/s is the aggregate decode rate across all 256 streams, about 0.8 tok/s per stream, and the 3864 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (278 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 214, 214, 214 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config glm47-u-tp4-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:37 PM

Last edited

—

Model

zai-org/GLM-4.7-Flash

Display name

GLM-4.7-Flash

Base model

—

Revision

7dd20894a642a0aa287e9827cb1a1f7f91386b67

Family

—

Parameters

31B

Active params

—

MoE

yes

Output

36.4 tok/s

Prefill

2,934 tok/s

Total

75.4 tok/s

TTFT

95.1 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

279

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-glm47-u-tp4-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12b/arms/glm47-u-tp4-m0/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/GLM-4.7-Flash --host 0.0.0.0 --served-model-name glm47 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-glm47-u-tp4-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

KASA B65 rig. Config glm47-u-tp4-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 36.4, 36.4, 36.4 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 17, 2026, 1:36 PM

Last edited

—

Model

inclusionAI/Ling-3.0-tiny

Display name

Ling-3.0-tiny

Base model

—

Revision

9a98e35fe1c9ee255f78dd64771c7ae15a799481

Family

—

Parameters

8B

Active params

—

MoE

yes

Output

1,341 tok/s

Prefill

162 tok/s

Total

3,003 tok/s

TTFT

1,990 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

318

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ling-u-tp4dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12b/arms/ling-u-tp4dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ling-3.0-tiny --host 0.0.0.0 --served-model-name ling --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-ling-u-tp4dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 1341.4 tok/s is the aggregate decode rate across all 256 streams, about 5.2 tok/s per stream, and the 1990 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (318 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 1343.5, 1341.4, 1340.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config ling-u-tp4dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 1:36 PM

Last edited

—

Model

inclusionAI/Ling-3.0-tiny

Display name

Ling-3.0-tiny

Base model

—

Revision

9a98e35fe1c9ee255f78dd64771c7ae15a799481

Family

—

Parameters

8B

Active params

—

MoE

yes

Output

118 tok/s

Prefill

3,424 tok/s

Total

254 tok/s

TTFT

92.3 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

316

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel