imthekaiser

Model

Qwen/Qwen3-4B

Display name

Qwen3-4B

Base model

Qwen3-4B-Base

Revision

1cfa9a7208912126459214e8b04321603b3df60c

Family

Qwen

Parameters

4B

Active params

—

MoE

no

Output

5,332 tok/s

Prefill

152 tok/s

Total

11,449 tok/s

TTFT

1,934 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

296

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q34-u-tp4dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q34-u-tp4dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3-4B --host 0.0.0.0 --served-model-name q34 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q34-u-tp4dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 5331.7 tok/s is the aggregate decode rate across all 256 streams, about 20.8 tok/s per stream, and the 1934 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 5330.9, 5331.7, 5336.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q34-u-tp4dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 12:36 PM

Last edited

—

Model

Qwen/Qwen3-4B

Display name

Qwen3-4B

Base model

Qwen3-4B-Base

Revision

1cfa9a7208912126459214e8b04321603b3df60c

Family

Qwen

Parameters

4B

Active params

—

MoE

no

Output

4,048 tok/s

Prefill

226 tok/s

Total

8,689 tok/s

TTFT

1,313 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

296

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q34-u-tp1dp3-m0-c192 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q34-u-tp1dp3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3-4B --host 0.0.0.0 --served-model-name q34 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 3 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q34-u-tp1dp3-m0-c192 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 3 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 4048.3 tok/s is the aggregate decode rate across all 192 streams, about 21.1 tok/s per stream, and the 1313 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 engine replicas (data parallel), 1 card each; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 4080.9, 4048.3, 2870.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q34-u-tp1dp3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 12:35 PM

Last edited

—

Model

Qwen/Qwen3-4B

Display name

Qwen3-4B

Base model

Qwen3-4B-Base

Revision

1cfa9a7208912126459214e8b04321603b3df60c

Family

Qwen

Parameters

4B

Active params

—

MoE

no

Output

3,853 tok/s

Prefill

103 tok/s

Total

8,275 tok/s

TTFT

2,881 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

296

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q34-u-tp2dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q34-u-tp2dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3-4B --host 0.0.0.0 --served-model-name q34 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q34-u-tp2dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 3853.1 tok/s is the aggregate decode rate across all 256 streams, about 15.1 tok/s per stream, and the 2881 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 3857.9, 3853.1, 3849.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q34-u-tp2dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 12:35 PM

Last edited

—

Model

Qwen/Qwen3-4B

Display name

Qwen3-4B

Base model

Qwen3-4B-Base

Revision

1cfa9a7208912126459214e8b04321603b3df60c

Family

Qwen

Parameters

4B

Active params

—

MoE

no

Output

2,437 tok/s

Prefill

117 tok/s

Total

5,215 tok/s

TTFT

2,542 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

296

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q34-u-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q34-u-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3-4B --host 0.0.0.0 --served-model-name q34 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q34-u-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 2437.1 tok/s is the aggregate decode rate across all 128 streams, about 19.0 tok/s per stream, and the 2542 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (296 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 2441.9, 2437.1, 2433.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q34-u-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 12:35 PM

Last edited

—

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

504 tok/s

Prefill

23.4 tok/s

Total

1,102 tok/s

TTFT

12,968 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g31-if8-tp4dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g31-if8-tp4dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-g31-if8-tp4dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 504.1 tok/s is the aggregate decode rate across all 256 streams, about 2.0 tok/s per stream, and the 12968 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 503.9, 504.1, 504.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config g31-if8-tp4dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 17, 2026, 4:09 AM

Last edited

—

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

239 tok/s

Prefill

2.9 tok/s

Total

522 tok/s

TTFT

103,016 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g31-if8-tp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g31-if8-tp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-g31-if8-tp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 239 tok/s is the aggregate decode rate across all 256 streams, about 0.9 tok/s per stream, and the 103016 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 239.3, 239, 238.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config g31-if8-tp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 6:41 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

3,809 tok/s

Prefill

93.1 tok/s

Total

8,287 tok/s

TTFT

3,247 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-if8-tp4dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-if8-tp4dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-orn9-if8-tp4dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 3808.9 tok/s is the aggregate decode rate across all 256 streams, about 14.9 tok/s per stream, and the 3247 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 3814.2, 3808.9, 3800.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-if8-tp4dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 6:41 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

2,341 tok/s

Prefill

90.3 tok/s

Total

5,092 tok/s

TTFT

3,377 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-if8-tp1dp3-m0-c192 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-if8-tp1dp3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 3 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-orn9-if8-tp1dp3-m0-c192 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 3 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 2341.3 tok/s is the aggregate decode rate across all 192 streams, about 12.2 tok/s per stream, and the 3377 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 engine replicas (data parallel), 1 card each; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 2342.3, 2341.3, 2338.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-if8-tp1dp3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 6:41 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

2,611 tok/s

Prefill

61.1 tok/s

Total

5,689 tok/s

TTFT

4,960 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-if8-tp2dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-if8-tp2dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-orn9-if8-tp2dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 2611 tok/s is the aggregate decode rate across all 256 streams, about 10.2 tok/s per stream, and the 4960 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 2611.5, 2611, 2610.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-if8-tp2dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 6:41 PM

Last edited

—

Model

meta-models/Muse-Glimmer-30B

Display name

Muse-Glimmer-30B

Base model

—

Revision

a4e59da52a7bc87ae7251dd5545c0dd437c44b68

Family

—

Parameters

30B

Active params

—

MoE

no

Output

1,117 tok/s

Prefill

18.5 tok/s

Total

2,515 tok/s

TTFT

17,649 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

322

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

FLASH_ATTN

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.95

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-muse-if8-tp4dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/muse-if8-tp4dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --attention-backend FLASH_ATTN --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-muse-if8-tp4dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 1116.8 tok/s is the aggregate decode rate across all 256 streams, about 4.4 tok/s per stream, and the 17649 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (322 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 1117.1, 1116.7, 1116.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config muse-if8-tp4dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 6:41 PM

Last edited

—

Model

meta-models/Muse-Glimmer-30B

Display name

Muse-Glimmer-30B

Base model

—

Revision

a4e59da52a7bc87ae7251dd5545c0dd437c44b68

Family

—

Parameters

30B

Active params

—

MoE

no

Output

842 tok/s

Prefill

13.3 tok/s

Total

1,895 tok/s

TTFT

24,423 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

322

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

FLASH_ATTN

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.95

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-muse-if8-tp2dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/muse-if8-tp2dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --attention-backend FLASH_ATTN --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-muse-if8-tp2dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 841.5 tok/s is the aggregate decode rate across all 256 streams, about 3.3 tok/s per stream, and the 24423 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (322 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 841.5, 841.5, 841.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config muse-if8-tp2dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 6:41 PM

Last edited

—

Model

meta-models/Muse-Glimmer-30B

Display name

Muse-Glimmer-30B

Base model

—

Revision

a4e59da52a7bc87ae7251dd5545c0dd437c44b68

Family

—

Parameters

30B

Active params

—

MoE

no

Output

1,455 tok/s

Prefill

27.1 tok/s

Total

3,275 tok/s

TTFT

12,043 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

322

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-muse-u-tp4dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/muse-u-tp4dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Muse-Glimmer-30B --host 0.0.0.0 --served-model-name muse --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-muse-u-tp4dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 1455.1 tok/s is the aggregate decode rate across all 256 streams, about 5.7 tok/s per stream, and the 12043 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (322 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 1455, 1455.1, 1455.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config muse-u-tp4dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 6:41 PM

Last edited

—

Model

google/gemma-4-31B-it

Display name

gemma-4-31B-it

Base model

gemma-4-31B

Revision

842da3794eaa0b77d5f08bae87a17459d91ff475

Family

Gemma

Parameters

31B

Active params

—

MoE

no

Output

387 tok/s

Prefill

5.0 tok/s

Total

846 tok/s

TTFT

61,360 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-g31-u-tp4-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/g31-u-tp4-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-31B-it --host 0.0.0.0 --served-model-name g31 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-g31-u-tp4-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 387 tok/s is the aggregate decode rate across all 256 streams, about 1.5 tok/s per stream, and the 61360 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 386.7, 388.1, 387 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config g31-u-tp4-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 6:40 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

1,584 tok/s

Prefill

70.6 tok/s

Total

3,443 tok/s

TTFT

4,284 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-if8-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-if8-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-orn9-if8-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 1584.3 tok/s is the aggregate decode rate across all 128 streams, about 12.4 tok/s per stream, and the 4284 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 1584.3, 1582.8, 1589.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-if8-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:22 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

3,633 tok/s

Prefill

89.3 tok/s

Total

7,907 tok/s

TTFT

3,372 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-u-tp4dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-u-tp4dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-orn9-u-tp4dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 3632.6 tok/s is the aggregate decode rate across all 256 streams, about 14.2 tok/s per stream, and the 3372 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 3634.2, 3632.6, 3624.2 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-u-tp4dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:22 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

2,152 tok/s

Prefill

90.0 tok/s

Total

4,687 tok/s

TTFT

3,385 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-u-tp1dp3-m0-c192 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-u-tp1dp3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 3 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-orn9-u-tp1dp3-m0-c192 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 3 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 2152.3 tok/s is the aggregate decode rate across all 192 streams, about 11.2 tok/s per stream, and the 3385 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 engine replicas (data parallel), 1 card each; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 1525.3, 2152.7, 2152.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-u-tp1dp3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:22 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

2,506 tok/s

Prefill

60.4 tok/s

Total

5,461 tok/s

TTFT

5,013 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-u-tp2dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-u-tp2dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-orn9-u-tp2dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 2506.3 tok/s is the aggregate decode rate across all 256 streams, about 9.8 tok/s per stream, and the 5013 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 2507, 2506.2, 2506.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-u-tp2dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:22 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-9B

Display name

Ornith-1.5-9B

Base model

—

Revision

489cb97981b8654bcfcf30ce1f94ed1b62e07b53

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

1,497 tok/s

Prefill

74.3 tok/s

Total

3,257 tok/s

TTFT

4,067 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-orn9-u-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/orn9-u-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-9B --host 0.0.0.0 --served-model-name orn9 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-orn9-u-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 1496.9 tok/s is the aggregate decode rate across all 128 streams, about 11.7 tok/s per stream, and the 4067 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 1506.9, 1496.9, 1496.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config orn9-u-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:22 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-35B-A3B

Display name

Ornith-1.5-35B-A3B

Base model

—

Revision

10fbf86fed7ecee4a061f8b499a618f46001cac1

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

3,729 tok/s

Prefill

111 tok/s

Total

8,104 tok/s

TTFT

2,732 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

sym_int4

SGLang quant

—

GPU mem util

0.92

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornb-iq4-tp4dp1-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/ornb-iq4-tp4dp1-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Ornith-1.5-35B-A3B --host 0.0.0.0 --served-model-name ornb --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization sym_int4 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-ornb-iq4-tp4dp1-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 3729.4 tok/s is the aggregate decode rate across all 256 streams, about 14.6 tok/s per stream, and the 2732 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, tensor parallel across 4 cards; vLLM 0.31.0+xpu.llmscaler (intel/llm-scaler-vllm:0.31.0-b1-experimental); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 3739.2, 3729.4, 3724.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config ornb-iq4-tp4dp1-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:22 PM

Last edited

—

Model

ornith-ai/Ornith-1.5-35B-A3B

Display name

Ornith-1.5-35B-A3B

Base model

—

Revision

10fbf86fed7ecee4a061f8b499a618f46001cac1

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

2,727 tok/s

Prefill

85.7 tok/s

Total

5,936 tok/s

TTFT

3,535 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

INT4

Backend

xpu

Tensor parallel

2

Pipeline parallel