imthekaiser

Model

devan-carlin/Qwen3.8-27B-int4-AutoRound

Display name

Qwen3.8-27B-int4-AutoRound

Base model

Qwen3.8-27B

Revision

bce40cacab0a4535b92fb3d57615c2bea9adf3d1

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

532 tok/s

Prefill

16.3 tok/s

Total

1,248 tok/s

TTFT

21,319 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.29.0+xpu

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

gptq

SGLang quant

—

GPU mem util

0.95

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-lab4b-tp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/lab4b-tp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_TOPO_P2P_ACCESS=1 -e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm kasa-lab-vllm-xpu:r310-int4-b2bab159f@sha256:57b8bc5c5a93279e6ff189897add38f683c56ecd0a7c7eaf5c266723a886a31f serve --model /llm/models/Qwen3.8-27B-int4-AutoRound-gptq --host 0.0.0.0 --served-model-name qwen38-27b-int4 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization gptq --compilation-config '<lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode>' --no-enable-prefix-caching --trust-remote-code --language-model-only --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-lab4b-tp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_TOPO_P2P_ACCESS=1 --e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config <lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode> --no-enable-prefix-caching --language-model-only

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 531.9 tok/s is the aggregate decode rate across all 256 streams, about 2.1 tok/s per stream, and the 21319 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.29.0+xpu (kasa-lab-vllm-xpu:r310-int4-b2bab159f); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 531.8, 531.9, 531.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config lab4b-tp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

devan-carlin/Qwen3.8-27B-int4-AutoRound

Display name

Qwen3.8-27B-int4-AutoRound

Base model

Qwen3.8-27B

Revision

bce40cacab0a4535b92fb3d57615c2bea9adf3d1

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

247 tok/s

Prefill

7.2 tok/s

Total

578 tok/s

TTFT

48,321 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.29.0+xpu

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

gptq

SGLang quant

—

GPU mem util

0.95

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-lab4b-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/lab4b-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_TOPO_P2P_ACCESS=1 -e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm kasa-lab-vllm-xpu:r310-int4-b2bab159f@sha256:57b8bc5c5a93279e6ff189897add38f683c56ecd0a7c7eaf5c266723a886a31f serve --model /llm/models/Qwen3.8-27B-int4-AutoRound-gptq --host 0.0.0.0 --served-model-name qwen38-27b-int4 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization gptq --compilation-config '<lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode>' --no-enable-prefix-caching --trust-remote-code --language-model-only --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-lab4b-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_TOPO_P2P_ACCESS=1 --e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config <lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode> --no-enable-prefix-caching --language-model-only

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 246.8 tok/s is the aggregate decode rate across all 128 streams, about 1.9 tok/s per stream, and the 48321 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.29.0+xpu (kasa-lab-vllm-xpu:r310-int4-b2bab159f); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 246.9, 246.8, 246.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config lab4b-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

RedHatAI/Qwen3.8-27B-INT4

Display name

Qwen3.8-27B-INT4

Base model

Qwen3.8-27B

Revision

2fb0debc365fb6c1683d7d3ad7722470919627a8

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

963 tok/s

Prefill

28.6 tok/s

Total

2,250 tok/s

TTFT

12,139 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q38i-u-tp1dp3-m0-c192 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/q38i-u-tp1dp3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=1 -e DISABLE_ESIMD_INT4_GEMM=1 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.8-27B-INT4 --host 0.0.0.0 --served-model-name qwen38-uvc --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 3 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q38i-u-tp1dp3-m0-c192 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=1 --e DISABLE_ESIMD_INT4_GEMM=1 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 3 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 962.7 tok/s is the aggregate decode rate across all 192 streams, about 5.0 tok/s per stream, and the 12139 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 engine replicas (data parallel), 1 card each; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 965.2, 962.6, 962.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q38i-u-tp1dp3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

RedHatAI/Qwen3.8-27B-INT4

Display name

Qwen3.8-27B-INT4

Base model

Qwen3.8-27B

Revision

2fb0debc365fb6c1683d7d3ad7722470919627a8

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

731 tok/s

Prefill

17.1 tok/s

Total

1,714 tok/s

TTFT

20,268 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q38i-u-tp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/q38i-u-tp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=1 -e DISABLE_ESIMD_INT4_GEMM=1 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.8-27B-INT4 --host 0.0.0.0 --served-model-name qwen38-uvc --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q38i-u-tp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=1 --e DISABLE_ESIMD_INT4_GEMM=1 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 730.5 tok/s is the aggregate decode rate across all 256 streams, about 2.9 tok/s per stream, and the 20268 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 730.5, 730.6, 730.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q38i-u-tp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

RedHatAI/Qwen3.8-27B-INT4

Display name

Qwen3.8-27B-INT4

Base model

Qwen3.8-27B

Revision

2fb0debc365fb6c1683d7d3ad7722470919627a8

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

365 tok/s

Prefill

8.9 tok/s

Total

855 tok/s

TTFT

39,189 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q38i-u-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/q38i-u-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=1 -e DISABLE_ESIMD_INT4_GEMM=1 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.8-27B-INT4 --host 0.0.0.0 --served-model-name qwen38-uvc --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q38i-u-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=1 --e DISABLE_ESIMD_INT4_GEMM=1 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 364.8 tok/s is the aggregate decode rate across all 128 streams, about 2.9 tok/s per stream, and the 39189 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 364.4, 365.8, 364.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q38i-u-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

Qwen/Qwen3.8-27B-FP8

Display name

Qwen3.8-27B-FP8

Base model

Qwen3.8-27B

Revision

017b9c7af6b5689d5dd426a76e0bc077eb5ca20a

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

392 tok/s

Prefill

8.6 tok/s

Total

921 tok/s

TTFT

40,283 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q38f-u-tp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/q38f-u-tp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.8-27B-FP8 --host 0.0.0.0 --served-model-name qwen38-uvc --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q38f-u-tp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 392.3 tok/s is the aggregate decode rate across all 256 streams, about 1.5 tok/s per stream, and the 40283 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 392.3, 392.4, 392.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q38f-u-tp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

Qwen/Qwen3.8-27B-FP8

Display name

Qwen3.8-27B-FP8

Base model

Qwen3.8-27B

Revision

017b9c7af6b5689d5dd426a76e0bc077eb5ca20a

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

454 tok/s

Prefill

17.1 tok/s

Total

1,064 tok/s

TTFT

20,233 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q38f-u-tp2-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/q38f-u-tp2-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.8-27B-FP8 --host 0.0.0.0 --served-model-name qwen38-uvc --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q38f-u-tp2-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 454.4 tok/s is the aggregate decode rate across all 128 streams, about 3.5 tok/s per stream, and the 20233 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 454.4, 454.3, 454.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q38f-u-tp2-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

Qwen/Qwen3.8-27B

Display name

Qwen3.8-27B

Base model

—

Revision

1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

461 tok/s

Prefill

28.9 tok/s

Total

1,082 tok/s

TTFT

11,939 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.9

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q38s-i-tp1dp3-m0-c192 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/q38s-i-tp1dp3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.8-27B --host 0.0.0.0 --served-model-name qwen38-27b --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 3 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q38s-i-tp1dp3-m0-c192 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 3 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 461.1 tok/s is the aggregate decode rate across all 192 streams, about 2.4 tok/s per stream, and the 11939 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 engine replicas (data parallel), 1 card each; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 460.8, 486, 461.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q38s-i-tp1dp3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

Qwen/Qwen3.8-27B

Display name

Qwen3.8-27B

Base model

—

Revision

1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

732 tok/s

Prefill

17.3 tok/s

Total

1,717 tok/s

TTFT

20,064 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.9

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q38s-i-tp2dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/q38s-i-tp2dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.8-27B --host 0.0.0.0 --served-model-name qwen38-27b --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q38s-i-tp2dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 732 tok/s is the aggregate decode rate across all 256 streams, about 2.9 tok/s per stream, and the 20064 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 732.2, 732, 731.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q38s-i-tp2dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

Qwen/Qwen3.8-27B

Display name

Qwen3.8-27B

Base model

—

Revision

1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

198 tok/s

Prefill

20.7 tok/s

Total

466 tok/s

TTFT

16,604 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.9

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q38s-i-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/q38s-i-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.8-27B --host 0.0.0.0 --served-model-name qwen38-27b --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q38s-i-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 198.3 tok/s is the aggregate decode rate across all 128 streams, about 1.5 tok/s per stream, and the 16604 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 198.2, 198.3, 198.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q38s-i-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

Qwen/Qwen3.6-35B-A3B-FP8

Display name

Qwen3.6-35B-A3B-FP8

Base model

Qwen3.6-35B-A3B

Revision

95a723d08a9490559dae23d0cff1d9466213d989

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

3,093 tok/s

Prefill

80.0 tok/s

Total

6,694 tok/s

TTFT

3,810 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36f-u-tp2dp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/q36f-u-tp2dp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.6-35B-A3B-FP8 --host 0.0.0.0 --served-model-name qwen36-35b --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 2 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q36f-u-tp2dp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 2 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 3093.2 tok/s is the aggregate decode rate across all 256 streams, about 12.1 tok/s per stream, and the 3810 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, 2 engine replicas (data parallel), 2 cards each; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 3093.2, 3109.5, 3046.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q36f-u-tp2dp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 1:55 AM

Last edited

—

Model

google/gemma-4-26B-A4B-it

Display name

gemma-4-26B-A4B-it

Base model

gemma-4-26B-A4B

Revision

4d7ae4984b7db7de8f8457170b3f1a419ee76d52

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

1,729 tok/s

Prefill

64.1 tok/s

Total

3,733 tok/s

TTFT

4,770 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-gem82-u-tp1dp4-m0-g92-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/gem82-u-tp1dp4-m0-g92-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-26B-A4B-it --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 4 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --quantization fp8 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-gem82-u-tp1dp4-m0-g92-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 4 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 1729.4 tok/s is the aggregate decode rate across all 256 streams, about 6.8 tok/s per stream, and the 4770 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, 4 engine replicas (data parallel), 1 card each; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 1726.3, 1740.6, 1729.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config gem82-u-tp1dp4-m0-g92-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 15, 2026, 11:52 PM

Last edited

—

Model

MIRALABS/Ornith-1.5-35B-A3B-W4A16-SYM

Display name

Ornith-1.5-35B-A3B-W4A16-SYM

Base model

Ornith-1.5-35B-A3B

Revision

08e6d31037779e5ecc02094a9b8c76eecff9db45

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

2,498 tok/s

Prefill

81.3 tok/s

Total

5,440 tok/s

TTFT

3,722 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornw-u-tp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/ornw-u-tp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-35B-A3B-W4A16 --host 0.0.0.0 --served-model-name ornith-15 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-ornw-u-tp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 2497.7 tok/s is the aggregate decode rate across all 256 streams, about 9.8 tok/s per stream, and the 3722 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 2496.1, 2500.1, 2497.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config ornw-u-tp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 15, 2026, 11:52 PM

Last edited

—

Model

MIRALABS/Ornith-1.5-35B-A3B-W4A16-SYM

Display name

Ornith-1.5-35B-A3B-W4A16-SYM

Base model

Ornith-1.5-35B-A3B

Revision

08e6d31037779e5ecc02094a9b8c76eecff9db45

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

944 tok/s

Prefill

21.0 tok/s

Total

2,055 tok/s

TTFT

14,501 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornw-u-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/ornw-u-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-35B-A3B-W4A16 --host 0.0.0.0 --served-model-name ornith-15 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-ornw-u-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 944.2 tok/s is the aggregate decode rate across all 128 streams, about 7.4 tok/s per stream, and the 14501 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 935.4, 945.9, 944.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config ornw-u-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 15, 2026, 11:52 PM

Last edited

—

Model

Vishva007/Qwen3.8-27B-W4A16-AutoRound-GPTQ

Display name

Qwen3.8-27B-W4A16-AutoRound-GPTQ

Base model

Qwen3.8-27B

Revision

1b9d56b428e9f841d7ad566f87660b5f47553c97

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

939 tok/s

Prefill

24.6 tok/s

Total

2,201 tok/s

TTFT

14,091 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.29.0+xpu

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

gptq

SGLang quant

—

GPU mem util

0.95

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-labv-tp4-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s10b/arms/labv-tp4-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0,1,2,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_TOPO_P2P_ACCESS=1 -e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm kasa-lab-vllm-xpu:r310-int4-b2bab159f@sha256:57b8bc5c5a93279e6ff189897add38f683c56ecd0a7c7eaf5c266723a886a31f serve --model /llm/models/Qwen3.8-27B-W4A16-AutoRound-GPTQ --host 0.0.0.0 --served-model-name qwen38-27b-int4 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization gptq --compilation-config '<lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode>' --no-enable-prefix-caching --trust-remote-code --language-model-only --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-labv-tp4-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0,1,2,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_TOPO_P2P_ACCESS=1 --e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config <lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode> --no-enable-prefix-caching --language-model-only

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 938.9 tok/s is the aggregate decode rate across all 256 streams, about 3.7 tok/s per stream, and the 14091 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.29.0+xpu (kasa-lab-vllm-xpu:r310-int4-b2bab159f); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 939.1, 938.9, 938.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config labv-tp4-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 15, 2026, 11:28 PM

Last edited

—

Model

Vishva007/Qwen3.8-27B-W4A16-AutoRound-GPTQ

Display name

Qwen3.8-27B-W4A16-AutoRound-GPTQ

Base model

Qwen3.8-27B

Revision

1b9d56b428e9f841d7ad566f87660b5f47553c97

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

918 tok/s

Prefill

48.5 tok/s

Total

2,146 tok/s

TTFT

7,106 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.29.0+xpu

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

gptq

SGLang quant

—

GPU mem util

0.95

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-labv-tp4-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s10b/arms/labv-tp4-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0,1,2,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_TOPO_P2P_ACCESS=1 -e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm kasa-lab-vllm-xpu:r310-int4-b2bab159f@sha256:57b8bc5c5a93279e6ff189897add38f683c56ecd0a7c7eaf5c266723a886a31f serve --model /llm/models/Qwen3.8-27B-W4A16-AutoRound-GPTQ --host 0.0.0.0 --served-model-name qwen38-27b-int4 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization gptq --compilation-config '<lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode>' --no-enable-prefix-caching --trust-remote-code --language-model-only --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-labv-tp4-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0,1,2,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_TOPO_P2P_ACCESS=1 --e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config <lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode> --no-enable-prefix-caching --language-model-only

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 918 tok/s is the aggregate decode rate across all 128 streams, about 7.2 tok/s per stream, and the 7106 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.29.0+xpu (kasa-lab-vllm-xpu:r310-int4-b2bab159f); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 918.3, 918, 917.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config labv-tp4-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 15, 2026, 11:28 PM

Last edited

—

Model

useful-quants/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16

Display name

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16

Base model

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Revision

cc977f002c7a396da87dc34c7bb4e5cdba710e24

Family

—

Parameters

30B

Active params

3B

MoE

yes

Output

137 tok/s

Prefill

3,761 tok/s

Total

290 tok/s

TTFT

81.9 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

308

Output tokens

256

Prefill tokens

0

Context length

32768

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

32

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-nemw-u-tp2-m0-ep --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/nemw-u-tp2-m0-ep/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Nemotron-3.5-Lightning-30B-A3B-W4A16 --host 0.0.0.0 --served-model-name nemotron-35 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 32768 --max-num-seqs 32 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --enable-expert-parallel --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-nemw-u-tp2-m0-ep --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 32768 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --enable-expert-parallel

Notes

KASA B65 rig. Config nemw-u-tp2-m0-ep served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 136.7, 136.7, 136.9 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 15, 2026, 11:28 PM

Last edited

—

Model

google/gemma-4-26B-A4B-it

Display name

gemma-4-26B-A4B-it

Base model

gemma-4-26B-A4B

Revision

4d7ae4984b7db7de8f8457170b3f1a419ee76d52

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

1,583 tok/s

Prefill

116 tok/s

Total

3,383 tok/s

TTFT

2,609 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

32

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-gem82-u-tp1dp4-m0-g92-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/gem82-u-tp1dp4-m0-g92-c128/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-26B-A4B-it --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 4 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 32 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --quantization fp8 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-gem82-u-tp1dp4-m0-g92-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 4 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 1582.8 tok/s is the aggregate decode rate across all 128 streams, about 12.4 tok/s per stream, and the 2609 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, 4 engine replicas (data parallel), 1 card each; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 32 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 1583.3, 1582.8, 1582.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config gem82-u-tp1dp4-m0-g92-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 15, 2026, 11:28 PM

Last edited

—

Model

google/gemma-4-26B-A4B-it

Display name

gemma-4-26B-A4B-it

Base model

gemma-4-26B-A4B

Revision

4d7ae4984b7db7de8f8457170b3f1a419ee76d52

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

88.0 tok/s

Prefill

4,235 tok/s

Total

189 tok/s

TTFT

72.3 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

306

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.95

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-gem82-u-tp4-m0 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/gem82-u-tp4-m0/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-26B-A4B-it --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --quantization fp8 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-gem82-u-tp4-m0 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config gem82-u-tp4-m0 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 88, 88, 88 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 15, 2026, 11:27 PM

Last edited

—

Model

google/gemma-4-26B-A4B-it

Display name

gemma-4-26B-A4B-it

Base model

gemma-4-26B-A4B

Revision

4d7ae4984b7db7de8f8457170b3f1a419ee76d52

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

74.6 tok/s

Prefill

3,386 tok/s

Total

159 tok/s

TTFT

88.9 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

301

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel