imthekaiser

Model

Qwen/Qwen3.5-9B

Display name

Qwen3.5-9B

Base model

Qwen3.5-9B-Base

Revision

c202236235762e1c871ad0ccb60c8ee5ba337b9a

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

128 tok/s

Prefill

3,737 tok/s

Total

270 tok/s

TTFT

81.1 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

303

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q359-if8-tp2-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q359-if8-tp2-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.5-9B --host 0.0.0.0 --served-model-name q359 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q359-if8-tp2-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q359-if8-tp2-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 131.6, 126.4, 128.1 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:47 AM

Last edited

—

Model

Qwen/Qwen3.5-9B

Display name

Qwen3.5-9B

Base model

Qwen3.5-9B-Base

Revision

c202236235762e1c871ad0ccb60c8ee5ba337b9a

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

102 tok/s

Prefill

2,308 tok/s

Total

212 tok/s

TTFT

131 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

303

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.31.0+xpu.llmscaler

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

fp8

SGLang quant

—

GPU mem util

0.92

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q359-if8-tp1-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q359-if8-tp1-m4/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm intel/llm-scaler-vllm:0.31.0-b1-experimental@sha256:a397e04d66494637862b1927ee92ab4dd8b2f03f0dc9d359ec9536959a88e704 serve --model /llm/models/Qwen3.5-9B --host 0.0.0.0 --served-model-name q359 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --quantization fp8 --compilation-config \{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"\,\"mode\":0\} --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q359-if8-tp1-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --compilation-config \{\cudagraph_mode\:\FULL_DECODE_ONLY\\,\mode\:0\} --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q359-if8-tp1-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 101.9, 95, 101.9 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:47 AM

Last edited

—

Model

Qwen/Qwen3.5-9B

Display name

Qwen3.5-9B

Base model

Qwen3.5-9B-Base

Revision

c202236235762e1c871ad0ccb60c8ee5ba337b9a

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

181 tok/s

Prefill

5,292 tok/s

Total

382 tok/s

TTFT

57.6 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

305

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

4

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q359-u-tp4-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q359-u-tp4-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.5-9B --host 0.0.0.0 --served-model-name q359 --tensor-parallel-size 4 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q359-u-tp4-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q359-u-tp4-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 186.3, 180.6, 175.5 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:47 AM

Last edited

—

Model

Qwen/Qwen3.5-9B

Display name

Qwen3.5-9B

Base model

Qwen3.5-9B-Base

Revision

c202236235762e1c871ad0ccb60c8ee5ba337b9a

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

106 tok/s

Prefill

4,114 tok/s

Total

226 tok/s

TTFT

73.9 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q359-u-tp2-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q359-u-tp2-m4/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.5-9B --host 0.0.0.0 --served-model-name q359 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q359-u-tp2-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q359-u-tp2-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 109.2, 105.9, 100.1 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:47 AM

Last edited

—

Model

Qwen/Qwen3.5-9B

Display name

Qwen3.5-9B

Base model

Qwen3.5-9B-Base

Revision

c202236235762e1c871ad0ccb60c8ee5ba337b9a

Family

Qwen

Parameters

9B

Active params

—

MoE

no

Output

77.7 tok/s

Prefill

3,177 tok/s

Total

166 tok/s

TTFT

96.0 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

305

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

BF16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.92

Max running seqs

4

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

MTP

Spec model

—

Spec draft model

—

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

yes

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q359-u-tp1-m4 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s12a/arms/q359-u-tp1-m4/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.5-9B --host 0.0.0.0 --served-model-name q359 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 4 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --speculative-config \{\"method\":\"qwen3_5_mtp\"\,\"num_speculative_tokens\":4\} --cpu-offload-gb 0 --port 8000

Extra flags

--speculative-config \{\method\:\qwen3_5_mtp\\,\num_speculative_tokens\:4\} --rm --d --name lsv-q359-u-tp1-m4 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

KASA B65 rig. Config q359-u-tp1-m4 served by kasa-testing scripts/bench serve.sh after its correctness gate. Canonical reasoning-v1, lmx single-shot requests after one discarded warm-up. Measured requests: 77.7, 76.6, 80.1 tok/s; this is the median (declared rule, tie to the earlier). Prompt/output samples, engineTimingsRaw and MTP counts were added from the lmx run file and exact per-request vLLM /metrics deltas, because lmx does not send them.

Reactions

—

Submitted

Oct 16, 2026, 11:47 AM

Last edited

—

Model

Vishva007/Qwen3.8-27B-W4A16-AutoRound-GPTQ

Display name

Qwen3.8-27B-W4A16-AutoRound-GPTQ

Base model

Qwen3.8-27B

Revision

1b9d56b428e9f841d7ad566f87660b5f47553c97

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

753 tok/s

Prefill

39.3 tok/s

Total

1,757 tok/s

TTFT

8,771 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.29.0+xpu

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

gptq

SGLang quant

—

GPU mem util

0.92

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-labv-tp1dp3-m0-c192 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11b/arms/labv-tp1dp3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0,1,2 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_TOPO_P2P_ACCESS=1 -e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm kasa-lab-vllm-xpu:r310-int4-b2bab159f@sha256:57b8bc5c5a93279e6ff189897add38f683c56ecd0a7c7eaf5c266723a886a31f serve --model /llm/models/Qwen3.8-27B-W4A16-AutoRound-GPTQ --host 0.0.0.0 --served-model-name qwen38-27b-int4 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 3 --gpu-memory-utilization 0.92 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization gptq --compilation-config '<lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode>' --no-enable-prefix-caching --trust-remote-code --language-model-only --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-labv-tp1dp3-m0-c192 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0,1,2 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_TOPO_P2P_ACCESS=1 --e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm --data-parallel-size 3 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config <lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode> --no-enable-prefix-caching --language-model-only

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 752.7 tok/s is the aggregate decode rate across all 192 streams, about 3.9 tok/s per stream, and the 8771 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 engine replicas (data parallel), 1 card each; vLLM 0.29.0+xpu (kasa-lab-vllm-xpu:r310-int4-b2bab159f); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 755.7, 752.7, 748.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config labv-tp1dp3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 11:47 AM

Last edited

—

Model

Vishva007/Qwen3.8-27B-W4A16-AutoRound-GPTQ

Display name

Qwen3.8-27B-W4A16-AutoRound-GPTQ

Base model

Qwen3.8-27B

Revision

1b9d56b428e9f841d7ad566f87660b5f47553c97

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

528 tok/s

Prefill

16.2 tok/s

Total

1,237 tok/s

TTFT

21,342 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.29.0+xpu

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

gptq

SGLang quant

—

GPU mem util

0.95

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-labv-tp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11b/arms/labv-tp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_TOPO_P2P_ACCESS=1 -e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm kasa-lab-vllm-xpu:r310-int4-b2bab159f@sha256:57b8bc5c5a93279e6ff189897add38f683c56ecd0a7c7eaf5c266723a886a31f serve --model /llm/models/Qwen3.8-27B-W4A16-AutoRound-GPTQ --host 0.0.0.0 --served-model-name qwen38-27b-int4 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization gptq --compilation-config '<lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode>' --no-enable-prefix-caching --trust-remote-code --language-model-only --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-labv-tp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_TOPO_P2P_ACCESS=1 --e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config <lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode> --no-enable-prefix-caching --language-model-only

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 527.6 tok/s is the aggregate decode rate across all 256 streams, about 2.1 tok/s per stream, and the 21342 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.29.0+xpu (kasa-lab-vllm-xpu:r310-int4-b2bab159f); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 527.7, 527.6, 527.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config labv-tp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 11:47 AM

Last edited

—

Model

amd/gemma-4-26B-A4B-it-w4a16-llmcompressor

Display name

gemma-4-26B-A4B-it-w4a16-llmcompressor

Base model

gemma-4-26B-A4B-it

Revision

5f5d9fa560e8eeabead101c85f1d639e1b5f8716

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

515 tok/s

Prefill

59.5 tok/s

Total

1,124 tok/s

TTFT

5,118 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.97

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-gema-u-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11b/arms/gema-u-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-26B-A4B-it-w4a16 --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.97 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-gema-u-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 515.2 tok/s is the aggregate decode rate across all 128 streams, about 4.0 tok/s per stream, and the 5118 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 512.8, 515.2, 515.7 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config gema-u-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:40 AM

Last edited

—

Model

Vishva007/Qwen3.8-27B-W4A16-AutoRound-GPTQ

Display name

Qwen3.8-27B-W4A16-AutoRound-GPTQ

Base model

Qwen3.8-27B

Revision

1b9d56b428e9f841d7ad566f87660b5f47553c97

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

238 tok/s

Prefill

7.0 tok/s

Total

559 tok/s

TTFT

49,693 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

346

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.29.0+xpu

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

gptq

SGLang quant

—

GPU mem util

0.95

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-labv-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11b/arms/labv-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_TOPO_P2P_ACCESS=1 -e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm kasa-lab-vllm-xpu:r310-int4-b2bab159f@sha256:57b8bc5c5a93279e6ff189897add38f683c56ecd0a7c7eaf5c266723a886a31f serve --model /llm/models/Qwen3.8-27B-W4A16-AutoRound-GPTQ --host 0.0.0.0 --served-model-name qwen38-27b-int4 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype float16 --quantization gptq --compilation-config '<lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode>' --no-enable-prefix-caching --trust-remote-code --language-model-only --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-labv-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_TOPO_P2P_ACCESS=1 --e CCL_ZE_IPC_EXCHANGE=pidfd --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --compilation-config <lab-full-decode-c320: exact JSON in kasa-testing scripts/bench/schema/server-config.v1.json execution_mode> --no-enable-prefix-caching --language-model-only

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 238.3 tok/s is the aggregate decode rate across all 128 streams, about 1.9 tok/s per stream, and the 49693 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.29.0+xpu (kasa-lab-vllm-xpu:r310-int4-b2bab159f); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (346 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 238.3, 238.3, 238.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config labv-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:40 AM

Last edited

—

Model

RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic

Display name

gemma-4-26B-A4B-it-FP8-dynamic

Base model

gemma-4-26B-A4B-it

Revision

ed35d7abe5d940da41b4ff06eb482feb0be8cb44

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

465 tok/s

Prefill

16.7 tok/s

Total

1,015 tok/s

TTFT

18,159 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-gemr-u-tp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11b/arms/gemr-u-tp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-26B-A4B-it-FP8-dynamic --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-gemr-u-tp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 465.1 tok/s is the aggregate decode rate across all 256 streams, about 1.8 tok/s per stream, and the 18159 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 465.1, 466, 464.4 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config gemr-u-tp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:40 AM

Last edited

—

Model

RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic

Display name

gemma-4-26B-A4B-it-FP8-dynamic

Base model

gemma-4-26B-A4B-it

Revision

ed35d7abe5d940da41b4ff06eb482feb0be8cb44

Family

Gemma

Parameters

26B

Active params

4B

MoE

yes

Output

115 tok/s

Prefill

2.6 tok/s

Total

251 tok/s

TTFT

115,519 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.97

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-gemr-u-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11b/arms/gemr-u-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/gemma-4-26B-A4B-it-FP8-dynamic --host 0.0.0.0 --served-model-name gemma-4 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.97 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-gemr-u-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 114.9 tok/s is the aggregate decode rate across all 128 streams, about 0.9 tok/s per stream, and the 115519 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 115.5, 114.9, 113.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config gemr-u-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:39 AM

Last edited

—

Model

RedHatAI/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8

Display name

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8

Base model

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Revision

bff279eb543e15f7c0e3f922661e66a1b78ac2e0

Family

—

Parameters

30B

Active params

3B

MoE

yes

Output

528 tok/s

Prefill

18.4 tok/s

Total

1,157 tok/s

TTFT

16,690 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

307

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-nem8-u-tp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11b/arms/nem8-u-tp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Nemotron-3.5-Lightning-30B-A3B-FP8 --host 0.0.0.0 --served-model-name nemotron-35 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-nem8-u-tp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 527.6 tok/s is the aggregate decode rate across all 256 streams, about 2.1 tok/s per stream, and the 16690 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (307 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 527.1, 528.5, 527.6 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config nem8-u-tp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 5:39 AM

Last edited

—

Model

palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4

Display name

Qwen3.6-35B-A3B-GPTQ-Int4

Base model

Qwen3.6-35B-A3B

Revision

00a66983516f8f8057741221277eed4141c9e431

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

1,357 tok/s

Prefill

106 tok/s

Total

2,944 tok/s

TTFT

2,872 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB

GPU slots

—

GPU count

1

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

GPTQ-4bit

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-q36g-u-tp1-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11b/arms/q36g-u-tp1-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Qwen3.6-35B-A3B-GPTQ-Int4 --host 0.0.0.0 --served-model-name qwen36-35b --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-q36g-u-tp1-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 1357 tok/s is the aggregate decode rate across all 128 streams, about 10.6 tok/s per stream, and the 2872 ms time to first token is under that load. Setup: KASA B65 rig, 1 x Intel Arc Pro B65, one card; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 1365.4, 1357, 1354.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config q36g-u-tp1-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 4:21 AM

Last edited

—

Model

RedHatAI/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8

Display name

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8

Base model

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Revision

bff279eb543e15f7c0e3f922661e66a1b78ac2e0

Family

—

Parameters

30B

Active params

3B

MoE

yes

Output

214 tok/s

Prefill

30.4 tok/s

Total

470 tok/s

TTFT

10,148 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

307

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.95

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-nem8-u-tp2-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11b/arms/nem8-u-tp2-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Nemotron-3.5-Lightning-30B-A3B-FP8 --host 0.0.0.0 --served-model-name nemotron-35 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.95 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-nem8-u-tp2-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 214 tok/s is the aggregate decode rate across all 128 streams, about 1.7 tok/s per stream, and the 10148 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (307 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 215.1, 212.4, 214 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config nem8-u-tp2-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 4:21 AM

Last edited

—

Model

useful-quants/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16

Display name

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16

Base model

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Revision

cc977f002c7a396da87dc34c7bb4e5cdba710e24

Family

—

Parameters

30B

Active params

3B

MoE

yes

Output

2,298 tok/s

Prefill

166 tok/s

Total

4,920 tok/s

TTFT

1,854 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

307

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×4

GPU slots

—

GPU count

4

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

32

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-nemw-u-tp1dp4-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/nemw-u-tp1dp4-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2\,3 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Nemotron-3.5-Lightning-30B-A3B-W4A16 --host 0.0.0.0 --served-model-name nemotron-35 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 4 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 32 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --enable-expert-parallel --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-nemw-u-tp1dp4-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2\,3 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 4 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --enable-expert-parallel

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 2297.5 tok/s is the aggregate decode rate across all 128 streams, about 17.9 tok/s per stream, and the 1854 ms time to first token is under that load. Setup: KASA B65 rig, 4 x Intel Arc Pro B65, 4 engine replicas (data parallel), 1 card each; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 32 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (307 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 2292.2, 2297.5, 2318.3 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config nemw-u-tp1dp4-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 4:21 AM

Last edited

—

Model

useful-quants/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16

Display name

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16

Base model

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Revision

cc977f002c7a396da87dc34c7bb4e5cdba710e24

Family

—

Parameters

30B

Active params

3B

MoE

yes

Output

2,332 tok/s

Prefill

105 tok/s

Total

5,075 tok/s

TTFT

2,923 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

307

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

192

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-nemw-u-tp1dp3-m0-c192 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/nemw-u-tp1dp3-m0-c192/session:/bench -e ZE_AFFINITY_MASK=0\,1\,2 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Nemotron-3.5-Lightning-30B-A3B-W4A16 --host 0.0.0.0 --served-model-name nemotron-35 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 3 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --enable-expert-parallel --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-nemw-u-tp1dp3-m0-c192 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1\,2 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 3 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --enable-expert-parallel

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 192 requests sent at once (batch 192, lmx --concurrency 192); 2332.3 tok/s is the aggregate decode rate across all 192 streams, about 12.1 tok/s per stream, and the 2923 ms time to first token is under that load. Setup: KASA B65 rig, 3 x Intel Arc Pro B65, 3 engine replicas (data parallel), 1 card each; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (307 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 192: 2352.4, 2332.3, 2311.8 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config nemw-u-tp1dp3-m0-c192, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 4:21 AM

Last edited

—

Model

useful-quants/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16

Display name

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16

Base model

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Revision

cc977f002c7a396da87dc34c7bb4e5cdba710e24

Family

—

Parameters

30B

Active params

3B

MoE

yes

Output

1,527 tok/s

Prefill

100 tok/s

Total

3,314 tok/s

TTFT

3,082 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

308

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

INT4_W4A16

Backend

xpu

Tensor parallel

1

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

64

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-nemw-u-tp1dp2-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/nemw-u-tp1dp2-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=1 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Nemotron-3.5-Lightning-30B-A3B-W4A16 --host 0.0.0.0 --served-model-name nemotron-35 --tensor-parallel-size 1 --pipeline-parallel-size 1 --data-parallel-size 2 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 64 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype auto --dtype bfloat16 --no-enable-prefix-caching --trust-remote-code --enable-expert-parallel --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-nemw-u-tp1dp2-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=1 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 2 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype auto --no-enable-prefix-caching --enable-expert-parallel

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 1526.8 tok/s is the aggregate decode rate across all 128 streams, about 11.9 tok/s per stream, and the 3082 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, 2 engine replicas (data parallel), 1 card each; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 64 per replica, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (308 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 1526.8, 1529.4, 1342.1 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config nemw-u-tp1dp2-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 3:18 AM

Last edited

—

Model

shisa-ai/Ornith-1.5-35B-A3B-MTP-FP8

Display name

Ornith-1.5-35B-A3B-MTP-FP8

Base model

Ornith-1.5-35B-A3B

Revision

5575a37e89719dde89754032208eada72e887dd3

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

785 tok/s

Prefill

22.8 tok/s

Total

1,709 tok/s

TTFT

13,272 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

256

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

256

Scheduler delay

—

Num parallel

—

Concurrency

256

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornf-u-tp2-m0-c256 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/ornf-u-tp2-m0-c256/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-35B-A3B-MTP-FP8 --host 0.0.0.0 --served-model-name ornith-15 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 256 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-ornf-u-tp2-m0-c256 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 256 requests sent at once (batch 256, lmx --concurrency 256); 785.2 tok/s is the aggregate decode rate across all 256 streams, about 3.1 tok/s per stream, and the 13272 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 256, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 256: 785.2, 781.1, 790.9 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config ornf-u-tp2-m0-c256, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 3:18 AM

Last edited

—

Model

shisa-ai/Ornith-1.5-35B-A3B-MTP-FP8

Display name

Ornith-1.5-35B-A3B-MTP-FP8

Base model

Ornith-1.5-35B-A3B

Revision

5575a37e89719dde89754032208eada72e887dd3

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

390 tok/s

Prefill

37.8 tok/s

Total

849 tok/s

TTFT

8,068 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

304

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

128

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×2

GPU slots

—

GPU count

2

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

FP8

Backend

xpu

Tensor parallel

2

Pipeline parallel

1

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

8192

Continuous batching

—

CPU offload

0GB

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.9

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

128

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

docker run --rm -d --name lsv-ornf-u-tp2-m0-c128 --net=host --shm-size=32g --device /dev/dri:/dev/dri -v /dev/dri:/dev/dri -v /models/weights:/llm/models:ro -v /models/bench/lmx-models-s11a/arms/ornf-u-tp2-m0-c128/session:/bench -e ZE_AFFINITY_MASK=0\,1 -e VLLM_XPU_ENABLE_XPU_GRAPH=1 -e DISABLE_ESIMD_GDN_OUTPROJ=0 -e DISABLE_ESIMD_INT4=0 -e DISABLE_ESIMD_INT4_GEMM=0 -e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 -e VLLM_WORKER_MULTIPROC_METHOD=spawn -e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989@sha256:a410323607625d551d370253e4c5e1a038bbdf8ba2535c2a30bdf5ac531fc4f4 serve --model /llm/models/Ornith-1.5-35B-A3B-MTP-FP8 --host 0.0.0.0 --served-model-name ornith-15 --tensor-parallel-size 2 --pipeline-parallel-size 1 --data-parallel-size 1 --gpu-memory-utilization 0.9 --max-model-len 4096 --max-num-seqs 128 --max-num-batched-tokens 8192 --block-size 64 --kv-cache-dtype auto --mamba-ssm-cache-dtype float16 --dtype float16 --no-enable-prefix-caching --trust-remote-code --limit-mm-per-prompt \{\"image\":0\,\"video\":0\} --cpu-offload-gb 0 --port 8000

Extra flags

--rm --d --name lsv-ornf-u-tp2-m0-c128 --net host --shm-size 32g --e ZE_AFFINITY_MASK=0\,1 --e VLLM_XPU_ENABLE_XPU_GRAPH=1 --e DISABLE_ESIMD_GDN_OUTPROJ=0 --e DISABLE_ESIMD_INT4=0 --e DISABLE_ESIMD_INT4_GEMM=0 --e VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=1 --e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --e VLLM_WORKER_MULTIPROC_METHOD=spawn --e CCL_ZE_IPC_EXCHANGE=sockets --entrypoint vllm --data-parallel-size 1 --max-model-len 4096 --block-size 64 --mamba-ssm-cache-dtype float16 --no-enable-prefix-caching --limit-mm-per-prompt \{\image\:0\,\video\:0\}

Notes

CONCURRENCY TEST, NOT A SINGLE-USER RESULT. 128 requests sent at once (batch 128, lmx --concurrency 128); 389.8 tok/s is the aggregate decode rate across all 128 streams, about 3.0 tok/s per stream, and the 8068 ms time to first token is under that load. Setup: KASA B65 rig, 2 x Intel Arc Pro B65, tensor parallel across 2 cards; vLLM 0.32.0rc1.dev100+g77154ca61 (vllm/vllm-openai-xpu:nightly-77154ca6116ae34cbfe314beef1016bde9b63989); max_num_seqs 128, max_model_len 4096; speculative decoding off. Each request: LocalMaxxing's canonical reasoning-v1 prompt (304 tokens) with a unique nonce, 256 output tokens, temperature 0. One discarded warm-up batch, then 3 measured batches of 128: 389.9, 389.8, 389.5 tok/s; this is the median. Not eligible for the Verified badge, which requires batch 1. Config ornf-u-tp2-m0-c128, kasa-testing campaign 2026-10-15-b65-localmaxxing-models.

Reactions

—

Submitted

Oct 16, 2026, 3:18 AM

Last edited

—

Model

openai/gpt-oss-20b

Display name

gpt-oss-20b

Base model

—

Revision

6cee5e81ee83917806bbde320786a8fb61efebee

Family

Gpt

Parameters

20B

Active params

—

MoE

yes

Output

2,700 tok/s

Prefill

23.6 tok/s

Total

5,062 tok/s

TTFT

14,164 ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

334

Output tokens

256

Prefill tokens

0

Context length

4096

Batch size

192

Hardware class

DISCRETE_GPU

Hardware

Intel Arc Pro B65 32GB ×3

GPU slots

—

GPU count

3

VRAM

32GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD EPYC 7343 16-Core Processor

RAM

499GB

OS

Ubuntu 26.04.1 LTS

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev100+g77154ca61

Quantization

MXFP4

Backend

xpu

Tensor parallel

1

Pipeline parallel