Sandberg

Model

orcarouter/Flint-abliterated-GGUF

Display name

Flint-abliterated-GGUF

Base model

flint

Revision

4d618d1cb1d7782d4b08762fb664487bf5e4d3c4

Family

Llama

Parameters

40B

Active params

—

MoE

yes

Output

108 tok/s

Prefill

3,195 tok/s

Total

1,713 tok/s

TTFT

5,127 ms

Peak VRAM

23.3GB

Power draw

402.4W

Hardware cost

—

Prompt tokens

16380

Output tokens

512

Prefill tokens

16380

Context length

65536

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04.5

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

Mix-24G

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Flint-abliterated-Mix-24G.gguf -ngl 999 -c 65536 -ctk q4_0 -ctv q4_0 -np 1 -fa on -fit off

Extra flags

--ctk q4_0 --ctv q4_0 --fit off

Notes

GGUF file: Flint-abliterated-Mix-24G.gguf (Mix-24G). Flint-abliterated (abliterated RecursiveSI/flint, 40B MoE) GGUF by OrcaRouter, mixed-precision imatrix build. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q4_0 KV, 1 slot, 65536 context (native 262144 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q4_0. 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 16380 tokens of context (16k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 16380. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 108.0-108.1 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 5, 2026, 4:06 AM

Last edited

—

Model

orcarouter/Flint-abliterated-GGUF

Display name

Flint-abliterated-GGUF

Base model

flint

Revision

4d618d1cb1d7782d4b08762fb664487bf5e4d3c4

Family

Llama

Parameters

40B

Active params

—

MoE

yes

Output

117 tok/s

Prefill

3,237 tok/s

Total

1,262 tok/s

TTFT

2,530 ms

Peak VRAM

23.3GB

Power draw

382.5W

Hardware cost

—

Prompt tokens

8189

Output tokens

512

Prefill tokens

8189

Context length

65536

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04.5

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

Mix-24G

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Flint-abliterated-Mix-24G.gguf -ngl 999 -c 65536 -ctk q4_0 -ctv q4_0 -np 1 -fa on -fit off

Extra flags

--ctk q4_0 --ctv q4_0 --fit off

Notes

GGUF file: Flint-abliterated-Mix-24G.gguf (Mix-24G). Flint-abliterated (abliterated RecursiveSI/flint, 40B MoE) GGUF by OrcaRouter, mixed-precision imatrix build. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q4_0 KV, 1 slot, 65536 context (native 262144 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q4_0. 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 8189 tokens of context (8k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 8189. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 117.2-117.3 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 5, 2026, 4:06 AM

Last edited

—

Model

orcarouter/Flint-abliterated-GGUF

Display name

Flint-abliterated-GGUF

Base model

flint

Revision

4d618d1cb1d7782d4b08762fb664487bf5e4d3c4

Family

Llama

Parameters

40B

Active params

—

MoE

yes

Output

122 tok/s

Prefill

3,106 tok/s

Total

837 tok/s

TTFT

1,319 ms

Peak VRAM

23.3GB

Power draw

361.6W

Hardware cost

—

Prompt tokens

4095

Output tokens

512

Prefill tokens

4095

Context length

65536

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04.5

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

Mix-24G

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Flint-abliterated-Mix-24G.gguf -ngl 999 -c 65536 -ctk q4_0 -ctv q4_0 -np 1 -fa on -fit off

Extra flags

--ctk q4_0 --ctv q4_0 --fit off

Notes

GGUF file: Flint-abliterated-Mix-24G.gguf (Mix-24G). Flint-abliterated (abliterated RecursiveSI/flint, 40B MoE) GGUF by OrcaRouter, mixed-precision imatrix build. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q4_0 KV, 1 slot, 65536 context (native 262144 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q4_0. 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 4095 tokens of context (4k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 4095. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 122.3-122.3 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 5, 2026, 4:06 AM

Last edited

—

Model

orcarouter/Flint-abliterated-GGUF

Display name

Flint-abliterated-GGUF

Base model

flint

Revision

4d618d1cb1d7782d4b08762fb664487bf5e4d3c4

Family

Llama

Parameters

40B

Active params

—

MoE

yes

Output

125 tok/s

Prefill

2,857 tok/s

Total

531 tok/s

TTFT

717 ms

Peak VRAM

23.3GB

Power draw

348.8W

Hardware cost

—

Prompt tokens

2047

Output tokens

512

Prefill tokens

2047

Context length

65536

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04.5

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

Mix-24G

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Flint-abliterated-Mix-24G.gguf -ngl 999 -c 65536 -ctk q4_0 -ctv q4_0 -np 1 -fa on -fit off

Extra flags

--ctk q4_0 --ctv q4_0 --fit off

Notes

GGUF file: Flint-abliterated-Mix-24G.gguf (Mix-24G). Flint-abliterated (abliterated RecursiveSI/flint, 40B MoE) GGUF by OrcaRouter, mixed-precision imatrix build. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q4_0 KV, 1 slot, 65536 context (native 262144 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q4_0. 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 2047 tokens of context (2k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 2047. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 124.9-124.9 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 5, 2026, 4:06 AM

Last edited

—

Model

orcarouter/Flint-abliterated-GGUF

Display name

Flint-abliterated-GGUF

Base model

flint

Revision

4d618d1cb1d7782d4b08762fb664487bf5e4d3c4

Family

Llama

Parameters

40B

Active params

—

MoE

yes

Output

130 tok/s

Prefill

1,401 tok/s

Total

174 tok/s

TTFT

141 ms

Peak VRAM

23.3GB

Power draw

321.8W

Hardware cost

—

Prompt tokens

197

Output tokens

512

Prefill tokens

0

Context length

65536

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04.5

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

Mix-24G

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Flint-abliterated-Mix-24G.gguf -ngl 999 -c 65536 -ctk q4_0 -ctv q4_0 -np 1 -fa on -fit off

Extra flags

--ctk q4_0 --ctv q4_0 --fit off

Notes

GGUF file: Flint-abliterated-Mix-24G.gguf (Mix-24G). Flint-abliterated (abliterated RecursiveSI/flint, 40B MoE) GGUF by OrcaRouter, mixed-precision imatrix build. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q4_0 KV, 1 slot, 65536 context (native 262144 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q4_0. 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Method: LocalMaxxing canonical prompt code-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking off. Median of 5 runs (range 129.8-129.9 tok/s). tokSOut = (completion - 1) / time after first token.

Reactions

—

Submitted

Nov 5, 2026, 4:02 AM

Last edited

—

Model

orcarouter/Flint-abliterated-GGUF

Display name

Flint-abliterated-GGUF

Base model

flint

Revision

4d618d1cb1d7782d4b08762fb664487bf5e4d3c4

Family

Llama

Parameters

40B

Active params

—

MoE

yes

Output

130 tok/s

Prefill

1,671 tok/s

Total

187 tok/s

TTFT

154 ms

Peak VRAM

23.3GB

Power draw

321.8W

Hardware cost

—

Prompt tokens

258

Output tokens

512

Prefill tokens

0

Context length

65536

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04.5

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

Mix-24G

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Flint-abliterated-Mix-24G.gguf -ngl 999 -c 65536 -ctk q4_0 -ctv q4_0 -np 1 -fa on -fit off

Extra flags

--ctk q4_0 --ctv q4_0 --fit off

Notes

GGUF file: Flint-abliterated-Mix-24G.gguf (Mix-24G). Flint-abliterated (abliterated RecursiveSI/flint, 40B MoE) GGUF by OrcaRouter, mixed-precision imatrix build. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q4_0 KV, 1 slot, 65536 context (native 262144 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q4_0. 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Method: LocalMaxxing canonical prompt reasoning-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking off. Median of 5 runs (range 128.6-129.5 tok/s). tokSOut = (completion - 1) / time after first token.

Reactions

—

Submitted

Nov 5, 2026, 4:02 AM

Last edited

—

Model

Apexintelligence-AI/Apex-35B-V1.0

Display name

Apex-35B-V1.0

Base model

Qwen3.6-35B-A3B

Revision

5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

88.4 tok/s

Prefill

5,069 tok/s

Total

4,566 tok/s

TTFT

51,505 ms

Peak VRAM

120.4GB

Power draw

346.6W

Hardware cost

—

Prompt tokens

261094

Output tokens

512

Prefill tokens

261094

Context length

262144

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

2

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.94

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Apexintelligence-AI/Apex-35B-V1.0@5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Extra flags

--max-model-len 262144 --language-model-only

Notes

Untuned out-of-the-box baseline (same server config as our LMX matrix for this model). Spec decode: none. KV cache: auto (bf16). 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image. Depth run: 261094 tokens of context (255k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 261094. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 88.4-88.8 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 8:45 AM

Last edited

—

Model

Apexintelligence-AI/Apex-35B-V1.0

Display name

Apex-35B-V1.0

Base model

Qwen3.6-35B-A3B

Revision

5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

121 tok/s

Prefill

11,009 tok/s

Total

4,620 tok/s

TTFT

2,976 ms

Peak VRAM

120.4GB

Power draw

302.5W

Hardware cost

—

Prompt tokens

32765

Output tokens

512

Prefill tokens

32765

Context length

262144

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

2

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.94

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Apexintelligence-AI/Apex-35B-V1.0@5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Extra flags

--max-model-len 262144 --language-model-only

Notes

Untuned out-of-the-box baseline (same server config as our LMX matrix for this model). Spec decode: none. KV cache: auto (bf16). 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image. Depth run: 32765 tokens of context (32k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 32765. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 120.9-121.3 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 8:45 AM

Last edited

—

Model

Apexintelligence-AI/Apex-35B-V1.0

Display name

Apex-35B-V1.0

Base model

Qwen3.6-35B-A3B

Revision

5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

126 tok/s

Prefill

11,693 tok/s

Total

3,087 tok/s

TTFT

1,401 ms

Peak VRAM

120.4GB

Power draw

282W

Hardware cost

—

Prompt tokens

16382

Output tokens

512

Prefill tokens

16382

Context length

262144

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

2

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.94

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Apexintelligence-AI/Apex-35B-V1.0@5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Extra flags

--max-model-len 262144 --language-model-only

Notes

Untuned out-of-the-box baseline (same server config as our LMX matrix for this model). Spec decode: none. KV cache: auto (bf16). 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image. Depth run: 16382 tokens of context (16k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 16382. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 125.7-126.0 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 8:45 AM

Last edited

—

Model

Apexintelligence-AI/Apex-35B-V1.0

Display name

Apex-35B-V1.0

Base model

Qwen3.6-35B-A3B

Revision

5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

128 tok/s

Prefill

11,516 tok/s

Total

1,846 tok/s

TTFT

711 ms

Peak VRAM

120.4GB

Power draw

273.1W

Hardware cost

—

Prompt tokens

8191

Output tokens

512

Prefill tokens

8191

Context length

262144

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

2

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.94

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Apexintelligence-AI/Apex-35B-V1.0@5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Extra flags

--max-model-len 262144 --language-model-only

Notes

Untuned out-of-the-box baseline (same server config as our LMX matrix for this model). Spec decode: none. KV cache: auto (bf16). 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image. Depth run: 8191 tokens of context (8k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 8191. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 127.8-128.1 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 8:45 AM

Last edited

—

Model

Apexintelligence-AI/Apex-35B-V1.0

Display name

Apex-35B-V1.0

Base model

Qwen3.6-35B-A3B

Revision

5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

130 tok/s

Prefill

10,417 tok/s

Total

1,065 tok/s

TTFT

393 ms

Peak VRAM

120.4GB

Power draw

263.7W

Hardware cost

—

Prompt tokens

4096

Output tokens

512

Prefill tokens

4096

Context length

262144

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

2

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.94

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Apexintelligence-AI/Apex-35B-V1.0@5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Extra flags

--max-model-len 262144 --language-model-only

Notes

Untuned out-of-the-box baseline (same server config as our LMX matrix for this model). Spec decode: none. KV cache: auto (bf16). 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image. Depth run: 4096 tokens of context (4k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 4096. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 130.0-130.2 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 8:45 AM

Last edited

—

Model

Apexintelligence-AI/Apex-35B-V1.0

Display name

Apex-35B-V1.0

Base model

Qwen3.6-35B-A3B

Revision

5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

130 tok/s

Prefill

9,088 tok/s

Total

616 tok/s

TTFT

225 ms

Peak VRAM

120.4GB

Power draw

273.3W

Hardware cost

—

Prompt tokens

2047

Output tokens

512

Prefill tokens

2047

Context length

262144

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

2

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.94

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Apexintelligence-AI/Apex-35B-V1.0@5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Extra flags

--max-model-len 262144 --language-model-only

Notes

Untuned out-of-the-box baseline (same server config as our LMX matrix for this model). Spec decode: none. KV cache: auto (bf16). 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image. Depth run: 2047 tokens of context (2k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 2047. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 130.2-130.4 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 8:45 AM

Last edited

—

Model

Apexintelligence-AI/Apex-35B-V1.0

Display name

Apex-35B-V1.0

Base model

Qwen3.6-35B-A3B

Revision

5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

131 tok/s

Prefill

3,387 tok/s

Total

187 tok/s

TTFT

68.8 ms

Peak VRAM

120.4GB

Power draw

260.4W (126.7 + 133.7W)

Hardware cost

—

Prompt tokens

233

Output tokens

512

Prefill tokens

0

Context length

262144

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

2

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.94

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Apexintelligence-AI/Apex-35B-V1.0@5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Extra flags

--max-model-len 262144 --language-model-only

Notes

Untuned out-of-the-box baseline (same server config as our LMX matrix for this model). Spec decode: none. KV cache: auto (bf16). Layout: PP2 (stock vLLM flag). 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image. Method: LocalMaxxing canonical prompt code-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking off. Median of 5 runs (range 130.5-130.5 tok/s). tokSOut = (completion - 1) / time after first token.

Reactions

—

Submitted

Nov 4, 2026, 8:41 AM

Last edited

—

Model

Apexintelligence-AI/Apex-35B-V1.0

Display name

Apex-35B-V1.0

Base model

Qwen3.6-35B-A3B

Revision

5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68

Family

Qwen

Parameters

35B

Active params

3B

MoE

yes

Output

131 tok/s

Prefill

2,146 tok/s

Total

199 tok/s

TTFT

139 ms

Peak VRAM

120.4GB

Power draw

260.4W (126.7 + 133.7W)

Hardware cost

—

Prompt tokens

298

Output tokens

512

Prefill tokens

0

Context length

262144

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a

Quantization

BF16

Backend

cuda

Tensor parallel

1

Pipeline parallel

2

GPU layers

—

Split mode

—

KV cache dtype

auto

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

0.94

Max running seqs

128

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Apexintelligence-AI/Apex-35B-V1.0@5105f8f9ed8fd8212a72da546bcbb4f01d2cfd68 --max-model-len 262144 --gpu-memory-utilization 0.94 --max-num-seqs 128 --enable-prefix-caching --trust-remote-code --pipeline-parallel-size 2 --language-model-only

Extra flags

--max-model-len 262144 --language-model-only

Notes

Untuned out-of-the-box baseline (same server config as our LMX matrix for this model). Spec decode: none. KV cache: auto (bf16). Layout: PP2 (stock vLLM flag). 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04 | vLLM: Morrowmake vllm-cmp170hx image. Method: LocalMaxxing canonical prompt reasoning-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking off. Median of 5 runs (range 130.5-130.5 tok/s). tokSOut = (completion - 1) / time after first token.

Reactions

—

Submitted

Nov 4, 2026, 8:41 AM

Last edited

—

Model

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Display name

DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Base model

DeepSeek-V4.1-Flash-1017

Revision

54229d38d588a66d1c6cff08af7b281b9326c656

Family

Deepseek

Parameters

763B

Active params

—

MoE

yes

Output

32.8 tok/s

Prefill

199 tok/s

Total

197 tok/s

TTFT

1,317,109 ms

Peak VRAM

106.1GB

Power draw

210.9W

Hardware cost

—

Prompt tokens

262120

Output tokens

512

Prefill tokens

262120

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

IQ2_XXS

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf (IQ2_XXS). IST-DASLab GSQ-RCO Coder: DeepSeek-V4.1-Flash-1017 with 192 of 384 routed experts per layer removed (RCO expert search) and the rest at the GSQ IQ2_XXS allocation; 72.1 GB of transformer weights on the GPUs, the 110.6 GB Engram lookup tables memory-mapped from NVMe. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 262120 tokens of context (256k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 262120. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 32.8-32.8 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 7:26 AM

Last edited

—

Model

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Display name

DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Base model

DeepSeek-V4.1-Flash-1017

Revision

54229d38d588a66d1c6cff08af7b281b9326c656

Family

Deepseek

Parameters

763B

Active params

—

MoE

yes

Output

38.6 tok/s

Prefill

274 tok/s

Total

250 tok/s

TTFT

119,688 ms

Peak VRAM

106.1GB

Power draw

223.9W

Hardware cost

—

Prompt tokens

32763

Output tokens

512

Prefill tokens

32763

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

IQ2_XXS

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf (IQ2_XXS). IST-DASLab GSQ-RCO Coder: DeepSeek-V4.1-Flash-1017 with 192 of 384 routed experts per layer removed (RCO expert search) and the rest at the GSQ IQ2_XXS allocation; 72.1 GB of transformer weights on the GPUs, the 110.6 GB Engram lookup tables memory-mapped from NVMe. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 32763 tokens of context (32k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 32763. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 38.6-38.6 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 7:26 AM

Last edited

—

Model

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Display name

DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Base model

DeepSeek-V4.1-Flash-1017

Revision

54229d38d588a66d1c6cff08af7b281b9326c656

Family

Deepseek

Parameters

763B

Active params

—

MoE

yes

Output

39.2 tok/s

Prefill

252 tok/s

Total

216 tok/s

TTFT

65,022 ms

Peak VRAM

106.1GB

Power draw

227.3W

Hardware cost

—

Prompt tokens

16381

Output tokens

512

Prefill tokens

16381

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

IQ2_XXS

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf (IQ2_XXS). IST-DASLab GSQ-RCO Coder: DeepSeek-V4.1-Flash-1017 with 192 of 384 routed experts per layer removed (RCO expert search) and the rest at the GSQ IQ2_XXS allocation; 72.1 GB of transformer weights on the GPUs, the 110.6 GB Engram lookup tables memory-mapped from NVMe. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 16381 tokens of context (16k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 16381. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 39.1-39.2 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 7:26 AM

Last edited

—

Model

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Display name

DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Base model

DeepSeek-V4.1-Flash-1017

Revision

54229d38d588a66d1c6cff08af7b281b9326c656

Family

Deepseek

Parameters

763B

Active params

—

MoE

yes

Output

39.4 tok/s

Prefill

282 tok/s

Total

207 tok/s

TTFT

29,015 ms

Peak VRAM

106.1GB

Power draw

226.9W

Hardware cost

—

Prompt tokens

8191

Output tokens

512

Prefill tokens

8191

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

IQ2_XXS

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf (IQ2_XXS). IST-DASLab GSQ-RCO Coder: DeepSeek-V4.1-Flash-1017 with 192 of 384 routed experts per layer removed (RCO expert search) and the rest at the GSQ IQ2_XXS allocation; 72.1 GB of transformer weights on the GPUs, the 110.6 GB Engram lookup tables memory-mapped from NVMe. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 8191 tokens of context (8k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 8191. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 39.4-39.4 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 7:26 AM

Last edited

—

Model

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Display name

DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Base model

DeepSeek-V4.1-Flash-1017

Revision

54229d38d588a66d1c6cff08af7b281b9326c656

Family

Deepseek

Parameters

763B

Active params

—

MoE

yes

Output

39.5 tok/s

Prefill

278 tok/s

Total

166 tok/s

TTFT

14,751 ms

Peak VRAM

106.1GB

Power draw

229.1W

Hardware cost

—

Prompt tokens

4095

Output tokens

512

Prefill tokens

4095

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

IQ2_XXS

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf (IQ2_XXS). IST-DASLab GSQ-RCO Coder: DeepSeek-V4.1-Flash-1017 with 192 of 384 routed experts per layer removed (RCO expert search) and the rest at the GSQ IQ2_XXS allocation; 72.1 GB of transformer weights on the GPUs, the 110.6 GB Engram lookup tables memory-mapped from NVMe. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 4095 tokens of context (4k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 4095. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 39.5-39.6 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 7:26 AM

Last edited

—

Model

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Display name

DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Base model

DeepSeek-V4.1-Flash-1017

Revision

54229d38d588a66d1c6cff08af7b281b9326c656

Family

Deepseek

Parameters

763B

Active params

—

MoE

yes

Output

39.8 tok/s

Prefill

269 tok/s

Total

125 tok/s

TTFT

7,606 ms

Peak VRAM

106.1GB

Power draw

226.6W

Hardware cost

—

Prompt tokens

2047

Output tokens

512

Prefill tokens

2047

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

IQ2_XXS

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf (IQ2_XXS). IST-DASLab GSQ-RCO Coder: DeepSeek-V4.1-Flash-1017 with 192 of 384 routed experts per layer removed (RCO expert search) and the rest at the GSQ IQ2_XXS allocation; 72.1 GB of transformer weights on the GPUs, the 110.6 GB Engram lookup tables memory-mapped from NVMe. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 2047 tokens of context (2k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 2047. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 39.7-39.9 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 7:26 AM

Last edited

—

Model

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Display name

DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Base model

DeepSeek-V4.1-Flash-1017

Revision

54229d38d588a66d1c6cff08af7b281b9326c656

Family

Deepseek

Parameters

763B

Active params

—

MoE

yes

Output

40.2 tok/s

Prefill

171 tok/s

Total

51.7 tok/s

TTFT

1,218 ms

Peak VRAM

106GB

Power draw

203.2W

Hardware cost

—

Prompt tokens

208

Output tokens

512

Prefill tokens

0

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

IQ2_XXS

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf (IQ2_XXS). IST-DASLab GSQ-RCO Coder: DeepSeek-V4.1-Flash-1017 with 192 of 384 routed experts per layer removed (RCO expert search) and the rest at the GSQ IQ2_XXS allocation; 72.1 GB of transformer weights on the GPUs, the 110.6 GB Engram lookup tables memory-mapped from NVMe. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Method: LocalMaxxing canonical prompt code-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking off. Median of 5 runs (range 40.2-40.3 tok/s). tokSOut = (completion - 1) / time after first token.

Reactions

—

Submitted

Nov 4, 2026, 6:02 AM

Last edited

—

Model

ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Display name

DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-GGUF

Base model

DeepSeek-V4.1-Flash-1017

Revision

54229d38d588a66d1c6cff08af7b281b9326c656

Family

Deepseek

Parameters

763B

Active params

—

MoE

yes

Output

40.0 tok/s

Prefill

196 tok/s

Total

55.0 tok/s

TTFT

1,356 ms

Peak VRAM

106GB

Power draw

203.2W

Hardware cost

—

Prompt tokens

266

Output tokens

512

Prefill tokens

0

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

IQ2_XXS

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: IQ2_XXS/DeepSeek-V4.1-Flash-1017-GSQ-RCO-Coder-IQ2_XXS-00001-of-00006.gguf (IQ2_XXS). IST-DASLab GSQ-RCO Coder: DeepSeek-V4.1-Flash-1017 with 192 of 384 routed experts per layer removed (RCO expert search) and the rest at the GSQ IQ2_XXS allocation; 72.1 GB of transformer weights on the GPUs, the 110.6 GB Engram lookup tables memory-mapped from NVMe. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Method: LocalMaxxing canonical prompt reasoning-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking off. Median of 5 runs (range 40.0-40.1 tok/s). tokSOut = (completion - 1) / time after first token.

Reactions

—

Submitted

Nov 4, 2026, 6:02 AM

Last edited

—

Model

XHToken/Spark-X2.6-4B-GGUF

Display name

Spark-X2.6-4B-GGUF

Base model

Spark-X2.6-4B

Revision

3c33ab38746828196b9b5979c739ee2d0bef894d

Family

—

Parameters

4B

Active params

—

MoE

no

Output

15.3 tok/s

Prefill

872 tok/s

Total

849 tok/s

TTFT

1,200,864 ms

Peak VRAM

19.6GB

Power draw

394.9W

Hardware cost

—

Prompt tokens

1047444

Output tokens

512

Prefill tokens

1047444

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04.5

Power

—

Engine

llama.cpp

Engine version

b12030

Quantization

Q8_0

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Spark-X2.6-4B-Q8_0.gguf -ngl 999 -c 1048576 -ctk q4_0 -ctv q4_0 -np 1 -fa on -fit off

Extra flags

--ctk q4_0 --ctv q4_0 --fit off

Notes

GGUF file: Spark-X2.6-4B-Q8_0.gguf (Q8_0). Spark-X2.6-4B by XHToken (official GGUF). llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q4_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q4_0. 1x RTX 3090 24GB (Zagitta/gpu-unlock), 420 W limit, PCIe Gen3 x16 | Intel Core i7-9700K 8C | 32 GB DDR4 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 1047444 tokens of context (1023k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 1047444. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 15.3-15.3 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.

Reactions

—

Submitted

Nov 4, 2026, 5:32 AM

Last edited

—

Model

XHToken/Spark-X2.6-4B-GGUF

Display name

Spark-X2.6-4B-GGUF

Base model

Spark-X2.6-4B

Revision

3c33ab38746828196b9b5979c739ee2d0bef894d

Family

—

Parameters

4B

Active params

—

MoE

no

Output

99.4 tok/s

Prefill

7,111 tok/s

Total

3,411 tok/s

TTFT

4,607 ms

Peak VRAM

19.6GB

Power draw

410.4W