Sandberg

Model

turboderp/Qwen3.8-27B-exl3

Display name

Qwen3.8-27B-exl3

Base model

Qwen3.8-27B

Revision

SC_5.00bpw_H6

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

160 tok/s

Prefill

937 tok/s

Total

215 tok/s

TTFT

247 ms

Peak VRAM

23.5GB

Power draw

410.5W

Hardware cost

—

Prompt tokens

231

Output tokens

512

Prefill tokens

0

Context length

5632

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a (Morrowmake/vllm-cmp170hx)

Quantization

EXL3 5.0 bpw (SC_5.00bpw_H6)

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

bfloat16

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

512

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

exl3

SGLang quant

—

GPU mem util

0.985

Max running seqs

1

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

DFlash2

Spec model

r0b0tlab/Qwen3.8-27B-DFlash2-EXL3-4.00bpw

Spec draft model

r0b0tlab/Qwen3.8-27B-DFlash2-EXL3-4.00bpw

Spec tokens

10

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve turboderp/Qwen3.8-27B-exl3:SC_5.00bpw_H6 --trust-remote-code --quantization exl3 --load-format exl3 --dtype bfloat16 --max-model-len 5632 --gpu-memory-utilization 0.985 --max-num-seqs 1 --max-num-batched-tokens 512 --skip-mm-profiling --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' --speculative-config '{"method":"dflash","model":"r0b0tlab/Qwen3.8-27B-DFlash2-EXL3-4.00bpw","num_speculative_tokens":10}'

Extra flags

--load-format exl3 --max-model-len 5632 --skip-mm-profiling --compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY"}

Notes

Rig: Phantex: Intel Core i7-9700K, 32 GB DDR4, Ubuntu 24.04.5, driver 580.178.04, PCIe Gen3 x16. 1x RTX 3090 24 GB (sm_86), power limit 420 W. Engine: Morrowmake's vllm-cmp170hx image (vLLM 0.32.0rc1.dev249) @sha256:5b45c4d6feaf, used as published; its EXL3 kernels run on Ampere sm_86 through a local integration layer. BF16/FP16 tensor-core rate unlocked with the open-source gpu-unlock kernel module (Zagitta/gpu-unlock; outputs bit-identical to stock in our checks). Weights: Turboderp's EXL3 5.0 bpw build as published. Speculative decoding: DFlash2 drafter r0b0tlab/Qwen3.8-27B-DFlash2-EXL3-4.00bpw (EXL3 build of incoai's DFlash2), 10 draft tokens. Target + drafter fill 24 GB, so max-model-len is 5,632 (KV pool ~5.9k tokens, bf16) and one sequence: a single-user short-context config. Prefix caching on, cudagraphs FULL_DECODE_ONLY. Thanks to Turboderp for EXL3/exllamav3 and the quant, Morrowmake for the vLLM EXL3 build, the Qwen team, and incoai and r0b0tlab for the drafter. Method: LocalMaxxing canonical prompt code-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cold prefill, thinking off. Median of 5 runs (range 157.7-163.9 tok/s). tokSOut = (completion - 1) / time after first token. gpuPowerWatts: mean per-card board power over the config's runs (nvidia-smi 2 s). peakVramGb: sum of per-card peaks.

Reactions

—

Submitted

Nov 8, 2026, 9:46 PM

Last edited

—

Model

turboderp/Qwen3.8-27B-exl3

Display name

Qwen3.8-27B-exl3

Base model

Qwen3.8-27B

Revision

SC_5.00bpw_H6

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

168 tok/s

Prefill

968 tok/s

Total

242 tok/s

TTFT

306 ms

Peak VRAM

23.5GB

Power draw

410.5W

Hardware cost

—

Prompt tokens

296

Output tokens

512

Prefill tokens

0

Context length

5632

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a (Morrowmake/vllm-cmp170hx)

Quantization

EXL3 5.0 bpw (SC_5.00bpw_H6)

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

bfloat16

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

512

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

exl3

SGLang quant

—

GPU mem util

0.985

Max running seqs

1

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

DFlash2

Spec model

r0b0tlab/Qwen3.8-27B-DFlash2-EXL3-4.00bpw

Spec draft model

r0b0tlab/Qwen3.8-27B-DFlash2-EXL3-4.00bpw

Spec tokens

10

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve turboderp/Qwen3.8-27B-exl3:SC_5.00bpw_H6 --trust-remote-code --quantization exl3 --load-format exl3 --dtype bfloat16 --max-model-len 5632 --gpu-memory-utilization 0.985 --max-num-seqs 1 --max-num-batched-tokens 512 --skip-mm-profiling --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' --speculative-config '{"method":"dflash","model":"r0b0tlab/Qwen3.8-27B-DFlash2-EXL3-4.00bpw","num_speculative_tokens":10}'

Extra flags

--load-format exl3 --max-model-len 5632 --skip-mm-profiling --compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY"}

Notes

Rig: Phantex: Intel Core i7-9700K, 32 GB DDR4, Ubuntu 24.04.5, driver 580.178.04, PCIe Gen3 x16. 1x RTX 3090 24 GB (sm_86), power limit 420 W. Engine: Morrowmake's vllm-cmp170hx image (vLLM 0.32.0rc1.dev249) @sha256:5b45c4d6feaf, used as published; its EXL3 kernels run on Ampere sm_86 through a local integration layer. BF16/FP16 tensor-core rate unlocked with the open-source gpu-unlock kernel module (Zagitta/gpu-unlock; outputs bit-identical to stock in our checks). Weights: Turboderp's EXL3 5.0 bpw build as published. Speculative decoding: DFlash2 drafter r0b0tlab/Qwen3.8-27B-DFlash2-EXL3-4.00bpw (EXL3 build of incoai's DFlash2), 10 draft tokens. Target + drafter fill 24 GB, so max-model-len is 5,632 (KV pool ~5.9k tokens, bf16) and one sequence: a single-user short-context config. Prefix caching on, cudagraphs FULL_DECODE_ONLY. Thanks to Turboderp for EXL3/exllamav3 and the quant, Morrowmake for the vLLM EXL3 build, the Qwen team, and incoai and r0b0tlab for the drafter. Method: LocalMaxxing canonical prompt reasoning-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cold prefill, thinking off. Median of 5 runs (range 160.4-168.7 tok/s). tokSOut = (completion - 1) / time after first token. gpuPowerWatts: mean per-card board power over the config's runs (nvidia-smi 2 s). peakVramGb: sum of per-card peaks.

Reactions

—

Submitted

Nov 8, 2026, 9:46 PM

Last edited

—

Model

unsloth/Qwen3.8-27B-GGUF

Display name

Qwen3.8-27B-GGUF

Base model

Qwen3.8-27B

Revision

main

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

141 tok/s

Prefill

838 tok/s

Total

190 tok/s

TTFT

276 ms

Peak VRAM

23GB

Power draw

390.4W

Hardware cost

—

Prompt tokens

231

Output tokens

512

Prefill tokens

0

Context length

131072

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04

Power

—

Engine

llama.cpp

Engine version

dspine fork, commit d493f84 (Ajay9o9/llama.cpp)

Quantization

GGUF UD-Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

99

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

yes

Spec method

DSpine (draft-dflash)

Spec model

Qwen3.8-27B-DSpine-b16-Q8_0-bf16bridge.gguf

Spec draft model

tencent/Qwen3.8-27B-DSpine-b16 (Q8_0, BF16 bridges)

Spec tokens

15

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf -md Qwen3.8-27B-DSpine-b16-Q8_0-bf16bridge.gguf --spec-type draft-dflash --spec-draft-n-max 15 -ngl 99 -c 131072 -np 1 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --jinja

Extra flags

--cache-type-k q4_0 --cache-type-v q4_0 --jinja

Notes

Rig: Phantex: Intel Core i7-9700K, 32 GB DDR4, Ubuntu 24.04.5, driver 580.178.04, PCIe Gen3 x16. 1x RTX 3090 24 GB (sm_86), power limit 420 W. Engine: llama.cpp built from Ajay9o9's dspine branch (commit d493f84, one commit on upstream llama.cpp adding the DSpine drafter graph), CUDA 12.9, sm_86. BF16/FP16 tensor-core rate unlocked with the open-source gpu-unlock kernel module (Zagitta/gpu-unlock; outputs bit-identical to stock in our checks). Target: unsloth's Qwen3.8-27B UD-Q4_K_M GGUF, all layers on GPU, flash attention, q4_0 K/V cache, 131,072-token context, 1 slot. Speculative decoding: Tencent's DSpine drafter (tencent/Qwen3.8-27B-DSpine-b16, Apache-2.0; a DFlash-style block drafter, 5 layers, block 16), converted with the fork's convert_hf_to_gguf.py and quantized to Q8_0 with llama-quantize, which keeps the dspine_* bridge projections in BF16; up to 15 draft tokens per step. Thanks to Tencent for DSpine, Ajay9o9 for the llama.cpp port, unsloth for the GGUF and the ggml-org llama.cpp team. Method: LocalMaxxing canonical prompt code-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cold prefill, thinking off. Median of 5 runs (range 136.3-146.9 tok/s). tokSOut = (completion - 1) / time after first token. gpuPowerWatts: mean per-card board power over the config's runs (nvidia-smi 2 s). peakVramGb: sum of per-card peaks.

Reactions

—

Submitted

Nov 8, 2026, 9:41 PM

Last edited

—

Model

unsloth/Qwen3.8-27B-GGUF

Display name

Qwen3.8-27B-GGUF

Base model

Qwen3.8-27B

Revision

main

Family

Qwen

Parameters

27B

Active params

—

MoE

no

Output

133 tok/s

Prefill

930 tok/s

Total

194 tok/s

TTFT

318 ms

Peak VRAM

23GB

Power draw

390.4W

Hardware cost

—

Prompt tokens

296

Output tokens

512

Prefill tokens

0

Context length

131072

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04

Power

—

Engine

llama.cpp

Engine version

dspine fork, commit d493f84 (Ajay9o9/llama.cpp)

Quantization

GGUF UD-Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

99

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

yes

Spec method

DSpine (draft-dflash)

Spec model

Qwen3.8-27B-DSpine-b16-Q8_0-bf16bridge.gguf

Spec draft model

tencent/Qwen3.8-27B-DSpine-b16 (Q8_0, BF16 bridges)

Spec tokens

15

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf -md Qwen3.8-27B-DSpine-b16-Q8_0-bf16bridge.gguf --spec-type draft-dflash --spec-draft-n-max 15 -ngl 99 -c 131072 -np 1 -fa on --cache-type-k q4_0 --cache-type-v q4_0 --jinja

Extra flags

--cache-type-k q4_0 --cache-type-v q4_0 --jinja

Notes

Rig: Phantex: Intel Core i7-9700K, 32 GB DDR4, Ubuntu 24.04.5, driver 580.178.04, PCIe Gen3 x16. 1x RTX 3090 24 GB (sm_86), power limit 420 W. Engine: llama.cpp built from Ajay9o9's dspine branch (commit d493f84, one commit on upstream llama.cpp adding the DSpine drafter graph), CUDA 12.9, sm_86. BF16/FP16 tensor-core rate unlocked with the open-source gpu-unlock kernel module (Zagitta/gpu-unlock; outputs bit-identical to stock in our checks). Target: unsloth's Qwen3.8-27B UD-Q4_K_M GGUF, all layers on GPU, flash attention, q4_0 K/V cache, 131,072-token context, 1 slot. Speculative decoding: Tencent's DSpine drafter (tencent/Qwen3.8-27B-DSpine-b16, Apache-2.0; a DFlash-style block drafter, 5 layers, block 16), converted with the fork's convert_hf_to_gguf.py and quantized to Q8_0 with llama-quantize, which keeps the dspine_* bridge projections in BF16; up to 15 draft tokens per step. Thanks to Tencent for DSpine, Ajay9o9 for the llama.cpp port, unsloth for the GGUF and the ggml-org llama.cpp team. Method: LocalMaxxing canonical prompt reasoning-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cold prefill, thinking off. Median of 5 runs (range 131.7-139.5 tok/s). tokSOut = (completion - 1) / time after first token. gpuPowerWatts: mean per-card board power over the config's runs (nvidia-smi 2 s). peakVramGb: sum of per-card peaks.

Reactions

—

Submitted

Nov 8, 2026, 9:41 PM

Last edited

—

Model

Nanbeige/Nanbeige4.3-33B

Display name

Nanbeige4.3-33B

Base model

Nanbeige4.3-33B-Base

Revision

main

Family

Llama

Parameters

33B

Active params

—

MoE

no

Output

70.4 tok/s

Prefill

2,208 tok/s

Total

104 tok/s

TTFT

116 ms

Peak VRAM

22.9GB

Power draw

417.8W

Hardware cost

—

Prompt tokens

256

Output tokens

512

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04.5

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a (Morrowmake/vllm-cmp170hx)

Quantization

EXL3 4.00bpw H6

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

bfloat16

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

exl3

SGLang quant

—

GPU mem util

0.98

Max running seqs

1

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

DFlash

Spec model

nb33-dflash-2L

Spec draft model

own DFlash drafter, 2 layers (unpublished)

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Nanbeige4.3-33B-exl3-4.00bpw_H6 --trust-remote-code --quantization exl3 --load-format exl3 --dtype bfloat16 --max-model-len 4096 --gpu-memory-utilization 0.98 --max-num-seqs 1 --speculative-config '{"method":"dflash","model":"nb33-dflash-2L","num_speculative_tokens":4,"quantization":"fp8"}' --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

Extra flags

--load-format exl3 --max-model-len 4096 --compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY"}

Notes

Nanbeige4.3-33B (dense, 52 layers) quantized to EXL3 4.00 bpw with 6-bit head (H6) using Turboderp's exllamav3 convert.py; served by vLLM 0.32.0rc1 from Morrowmake's vllm-cmp170hx image (sha256:5b45c4d6feaf), whose EXL3 kernels we run on Ampere sm_86 through a local integration layer. Thanks to Turboderp for EXL3/exllamav3 and to Morrowmake for the vLLM EXL3 build. Rig Phantex: i7-9700K, 32 GB DDR4, Ubuntu 24.04.5, driver 580.178.04. 1x RTX 3090 24 GB, PCIe Gen3 x16, power limit 420 W (BF16/FP16 tensor-core rate unlocked with the open-source gpu-unlock kernel module, Zagitta/gpu-unlock; outputs bit-identical to stock in our checks). Speculative decoding: our own DFlash drafter (2 layers initialised from target layers 50-51, trained with vllm-project/speculators on ~8.3k self-distilled chat/code responses; weight-only FP8 when served), 4 draft tokens. KV pool 13,083 tokens (bf16), prefix caching on, cudagraphs FULL_DECODE_ONLY. Method: LocalMaxxing canonical prompt code-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cold prefill (nonce), thinking off. Median of 5 runs (range 68.7-71.7 tok/s). tokSOut = (completion - 1) / time after first token. gpuPowerWatts: mean board power over the config's runs (nvidia-smi 2 s, loaded samples). Weights not published.

Reactions

—

Submitted

Nov 8, 2026, 9:34 PM

Last edited

—

Model

Nanbeige/Nanbeige4.3-33B

Display name

Nanbeige4.3-33B

Base model

Nanbeige4.3-33B-Base

Revision

main

Family

Llama

Parameters

33B

Active params

—

MoE

no

Output

81.5 tok/s

Prefill

2,735 tok/s

Total

130 tok/s

TTFT

117 ms

Peak VRAM

22.9GB

Power draw

417.8W

Hardware cost

—

Prompt tokens

321

Output tokens

512

Prefill tokens

0

Context length

4096

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

RTX 3090 24GB

GPU slots

—

GPU count

1

VRAM

24GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

Intel Core i7-9700K (8C)

RAM

32GB

OS

Ubuntu 24.04.5

Power

—

Engine

vllm

Engine version

0.32.0rc1.dev249+g18305948a (Morrowmake/vllm-cmp170hx)

Quantization

EXL3 4.00bpw H6

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

bfloat16

KV cache size

—

Prefix caching

yes

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

exl3

SGLang quant

—

GPU mem util

0.98

Max running seqs

1

Scheduler delay

—

Num parallel

—

Concurrency

1

Spec decoding

yes

Spec method

DFlash

Spec model

nb33-dflash-2L

Spec draft model

own DFlash drafter, 2 layers (unpublished)

Spec tokens

4

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

vllm serve Nanbeige4.3-33B-exl3-4.00bpw_H6 --trust-remote-code --quantization exl3 --load-format exl3 --dtype bfloat16 --max-model-len 4096 --gpu-memory-utilization 0.98 --max-num-seqs 1 --speculative-config '{"method":"dflash","model":"nb33-dflash-2L","num_speculative_tokens":4,"quantization":"fp8"}' --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

Extra flags

--load-format exl3 --max-model-len 4096 --compilation-config {"cudagraph_mode":"FULL_DECODE_ONLY"}

Notes

Nanbeige4.3-33B (dense, 52 layers) quantized to EXL3 4.00 bpw with 6-bit head (H6) using Turboderp's exllamav3 convert.py; served by vLLM 0.32.0rc1 from Morrowmake's vllm-cmp170hx image (sha256:5b45c4d6feaf), whose EXL3 kernels we run on Ampere sm_86 through a local integration layer. Thanks to Turboderp for EXL3/exllamav3 and to Morrowmake for the vLLM EXL3 build. Rig Phantex: i7-9700K, 32 GB DDR4, Ubuntu 24.04.5, driver 580.178.04. 1x RTX 3090 24 GB, PCIe Gen3 x16, power limit 420 W (BF16/FP16 tensor-core rate unlocked with the open-source gpu-unlock kernel module, Zagitta/gpu-unlock; outputs bit-identical to stock in our checks). Speculative decoding: our own DFlash drafter (2 layers initialised from target layers 50-51, trained with vllm-project/speculators on ~8.3k self-distilled chat/code responses; weight-only FP8 when served), 4 draft tokens. KV pool 13,083 tokens (bf16), prefix caching on, cudagraphs FULL_DECODE_ONLY. Method: LocalMaxxing canonical prompt reasoning-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cold prefill (nonce), thinking off. Median of 5 runs (range 78.8-82.5 tok/s). tokSOut = (completion - 1) / time after first token. gpuPowerWatts: mean board power over the config's runs (nvidia-smi 2 s, loaded samples). Weights not published.

Reactions

—

Submitted

Nov 8, 2026, 9:34 PM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

13.4 tok/s

Prefill

487 tok/s

Total

158 tok/s

TTFT

16,815 ms

Peak VRAM

114.9GB

Power draw

269.7W

Hardware cost

—

Prompt tokens

8189

Output tokens

512

Prefill tokens

8189

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q8_0

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q8_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q8_0-00001-of-00002.gguf -ngl 999 -c 1048576 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off

Extra flags

--ctk q8_0 --ctv q8_0 --fit off

Notes

GGUF file: Ripple-65B-Preview-Q8_0-00001-of-00002.gguf (Q8_0). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, Q8_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q8_0. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 8189 tokens of context (8k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 8189. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 13.3-13.4 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:36 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

13.6 tok/s

Prefill

436 tok/s

Total

97.8 tok/s

TTFT

9,389 ms

Peak VRAM

114.9GB

Power draw

282.2W

Hardware cost

—

Prompt tokens

4095

Output tokens

512

Prefill tokens

4095

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q8_0

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q8_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q8_0-00001-of-00002.gguf -ngl 999 -c 1048576 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off

Extra flags

--ctk q8_0 --ctv q8_0 --fit off

Notes

GGUF file: Ripple-65B-Preview-Q8_0-00001-of-00002.gguf (Q8_0). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, Q8_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q8_0. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 4095 tokens of context (4k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 4095. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 13.6-13.6 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:36 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

12.0 tok/s

Prefill

504 tok/s

Total

309 tok/s

TTFT

65,061 ms

Peak VRAM

114.9GB

Power draw

281.3W

Hardware cost

—

Prompt tokens

32762

Output tokens

512

Prefill tokens

32762

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q8_0

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q8_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q8_0-00001-of-00002.gguf -ngl 999 -c 1048576 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off

Extra flags

--ctk q8_0 --ctv q8_0 --fit off

Notes

GGUF file: Ripple-65B-Preview-Q8_0-00001-of-00002.gguf (Q8_0). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, Q8_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q8_0. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 32762 tokens of context (32k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 32762. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 12.0-12.0 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

13.7 tok/s

Prefill

357 tok/s

Total

59.4 tok/s

TTFT

5,734 ms

Peak VRAM

114.9GB

Power draw

288.1W

Hardware cost

—

Prompt tokens

2047

Output tokens

512

Prefill tokens

2047

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q8_0

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q8_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q8_0-00001-of-00002.gguf -ngl 999 -c 1048576 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off

Extra flags

--ctk q8_0 --ctv q8_0 --fit off

Notes

GGUF file: Ripple-65B-Preview-Q8_0-00001-of-00002.gguf (Q8_0). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, Q8_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q8_0. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 2047 tokens of context (2k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 2047. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 13.7-13.7 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

12.9 tok/s

Prefill

507 tok/s

Total

234 tok/s

TTFT

32,293 ms

Peak VRAM

114.9GB

Power draw

275W

Hardware cost

—

Prompt tokens

16380

Output tokens

512

Prefill tokens

16380

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q8_0

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q8_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q8_0-00001-of-00002.gguf -ngl 999 -c 1048576 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off

Extra flags

--ctk q8_0 --ctv q8_0 --fit off

Notes

GGUF file: Ripple-65B-Preview-Q8_0-00001-of-00002.gguf (Q8_0). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, Q8_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q8_0. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 16380 tokens of context (16k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 16380. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 12.9-12.9 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

8.4 tok/s

Prefill

413 tok/s

Total

348 tok/s

TTFT

317,030 ms

Peak VRAM

114.9GB

Power draw

282W

Hardware cost

—

Prompt tokens

131055

Output tokens

512

Prefill tokens

131055

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q8_0

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q8_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q8_0-00001-of-00002.gguf -ngl 999 -c 1048576 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off

Extra flags

--ctk q8_0 --ctv q8_0 --fit off

Notes

GGUF file: Ripple-65B-Preview-Q8_0-00001-of-00002.gguf (Q8_0). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, Q8_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q8_0. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 131055 tokens of context (128k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 131055. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 8.4-8.4 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

14.1 tok/s

Prefill

204 tok/s

Total

20.8 tok/s

TTFT

1,321 ms

Peak VRAM

114.9GB

Power draw

281.8W

Hardware cost

—

Prompt tokens

269

Output tokens

512

Prefill tokens

0

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q8_0

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q8_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q8_0-00001-of-00002.gguf -ngl 999 -c 1048576 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off

Extra flags

--ctk q8_0 --ctv q8_0 --fit off

Notes

GGUF file: Ripple-65B-Preview-Q8_0-00001-of-00002.gguf (Q8_0). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, Q8_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q8_0. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Method: LocalMaxxing canonical prompt reasoning-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 5 runs (range 14.1-14.1 tok/s). tokSOut = (completion - 1) / time after first token. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

14.0 tok/s

Prefill

225 tok/s

Total

19.2 tok/s

TTFT

919 ms

Peak VRAM

114.9GB

Power draw

281.8W

Hardware cost

—

Prompt tokens

207

Output tokens

512

Prefill tokens

0

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q8_0

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q8_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q8_0-00001-of-00002.gguf -ngl 999 -c 1048576 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off

Extra flags

--ctk q8_0 --ctv q8_0 --fit off

Notes

GGUF file: Ripple-65B-Preview-Q8_0-00001-of-00002.gguf (Q8_0). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, Q8_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q8_0. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Method: LocalMaxxing canonical prompt code-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking left on (Ripple's chat template has no thinking switch): on this code prompt the model went straight to code in all 5 runs, without reasoning first. Median of 5 runs (range 14.0-14.1 tok/s). tokSOut = (completion - 1) / time after first token. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

17.8 tok/s

Prefill

478 tok/s

Total

189 tok/s

TTFT

17,128 ms

Peak VRAM

109.7GB

Power draw

289.2W

Hardware cost

—

Prompt tokens

8189

Output tokens

512

Prefill tokens

8189

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q4_K_M.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: Ripple-65B-Preview-Q4_K_M.gguf (Q4_K_M). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 8189 tokens of context (8k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 8189. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 17.8-17.8 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

17.9 tok/s

Prefill

418 tok/s

Total

120 tok/s

TTFT

9,809 ms

Peak VRAM

109.7GB

Power draw

288.8W

Hardware cost

—

Prompt tokens

4095

Output tokens

512

Prefill tokens

4095

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q4_K_M.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: Ripple-65B-Preview-Q4_K_M.gguf (Q4_K_M). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 4095 tokens of context (4k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 4095. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 17.8-17.9 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

17.3 tok/s

Prefill

537 tok/s

Total

367 tok/s

TTFT

61,001 ms

Peak VRAM

109.7GB

Power draw

295.7W

Hardware cost

—

Prompt tokens

32762

Output tokens

512

Prefill tokens

32762

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q4_K_M.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: Ripple-65B-Preview-Q4_K_M.gguf (Q4_K_M). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 32762 tokens of context (32k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 32762. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 17.3-17.3 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

17.9 tok/s

Prefill

321 tok/s

Total

73.2 tok/s

TTFT

6,373 ms

Peak VRAM

109.7GB

Power draw

266.7W

Hardware cost

—

Prompt tokens

2047

Output tokens

512

Prefill tokens

2047

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q4_K_M.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: Ripple-65B-Preview-Q4_K_M.gguf (Q4_K_M). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 2047 tokens of context (2k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 2047. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 17.9-17.9 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

17.6 tok/s

Prefill

522 tok/s

Total

279 tok/s

TTFT

31,401 ms

Peak VRAM

109.7GB

Power draw

285.5W

Hardware cost

—

Prompt tokens

16380

Output tokens

512

Prefill tokens

16380

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q4_K_M.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: Ripple-65B-Preview-Q4_K_M.gguf (Q4_K_M). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 16380 tokens of context (16k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 16380. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 17.6-17.6 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

15.6 tok/s

Prefill

452 tok/s

Total

408 tok/s

TTFT

290,030 ms

Peak VRAM

109.7GB

Power draw

294.9W

Hardware cost

—

Prompt tokens

131055

Output tokens

512

Prefill tokens

131055

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q4_K_M.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: Ripple-65B-Preview-Q4_K_M.gguf (Q4_K_M). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 131055 tokens of context (128k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 131055. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 15.6-15.6 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:35 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

18.0 tok/s

Prefill

183 tok/s

Total

26.2 tok/s

TTFT

1,469 ms

Peak VRAM

109.7GB

Power draw

290.6W

Hardware cost

—

Prompt tokens

269

Output tokens

512

Prefill tokens

0

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q4_K_M.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: Ripple-65B-Preview-Q4_K_M.gguf (Q4_K_M). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Method: LocalMaxxing canonical prompt reasoning-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 5 runs (range 18.0-18.1 tok/s). tokSOut = (completion - 1) / time after first token. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:34 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

17.9 tok/s

Prefill

194 tok/s

Total

24.3 tok/s

TTFT

1,067 ms

Peak VRAM

109.7GB

Power draw

290.6W

Hardware cost

—

Prompt tokens

207

Output tokens

512

Prefill tokens

0

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB ×2

GPU slots

—

GPU count

2

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

f16

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q4_K_M.gguf -ngl 999 -c 1048576 -np 1 -fa on -fit off

Extra flags

--fit off

Notes

GGUF file: Ripple-65B-Preview-Q4_K_M.gguf (Q4_K_M). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload (layer split over 2 GPUs), FlashAttention on, F16 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: f16. 2x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Method: LocalMaxxing canonical prompt code-v1, batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), cache-bust nonce so every prefill is cold, thinking left on (Ripple's chat template has no thinking switch): on this code prompt the model went straight to code in all 5 runs, without reasoning first. Median of 5 runs (range 17.9-17.9 tok/s). tokSOut = (completion - 1) / time after first token. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:34 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

14.9 tok/s

Prefill

373 tok/s

Total

154 tok/s

TTFT

21,969 ms

Peak VRAM

59.5GB

Power draw

178.5W

Hardware cost

—

Prompt tokens

8189

Output tokens

512

Prefill tokens

8189

Context length

1048576

Batch size

1

Hardware class

DISCRETE_GPU

Hardware

CMP 170HX 64GB

GPU slots

—

GPU count

1

VRAM

64GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

2x Intel Xeon E5-2630 v3

RAM

64GB

OS

Ubuntu 26.04.1

Power

—

Engine

llama.cpp

Engine version

b12119-pr31692

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

1

Pipeline parallel

—

GPU layers

999

Split mode

—

KV cache dtype

q4_0

KV cache size

—

Prefix caching

no

Attention backend

—

Flash attention

yes

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

1

Scheduler delay

—

Num parallel

1

Concurrency

1

Spec decoding

no

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

Spec draft window

—

MTP enabled

no

MTP draft layers

—

Temperature

0

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

llama-server -m Ripple-65B-Preview-Q4_K_M.gguf -ngl 999 -c 1048576 -ctk q4_0 -ctv q4_0 -np 1 -fa on -fit off

Extra flags

--ctk q4_0 --ctv q4_0 --fit off

Notes

GGUF file: Ripple-65B-Preview-Q4_K_M.gguf (Q4_K_M). Ripple architecture is new: these numbers use the unmerged llama.cpp support from PR #31692 by notforu777, with GGUFs by AtomicChat; they may change once support is merged. llama.cpp llama-server built from ggml-org/llama.cpp PR #31692 (Ripple architecture support, not yet merged) at commit 57713fb on top of b12119, CUDA 12.8 server image built from the repo's .devops/cuda.Dockerfile for sm_80, full GPU offload, FlashAttention on, Q4_0 KV, 1 slot, native 1048576 context, prompt cache off per request. Spec decode: none. KV cache: q4_0. 1x NVIDIA CMP 170HX 64GB (cmpunlocker), 180 W limit, PCIe Gen2 x4 | 2x Intel Xeon E5-2630 v3 | 64 GB DDR4 | Ubuntu 26.04.1, NVIDIA driver 610.57.04. Depth run: 8189 tokens of context (8k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 8189. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking left on (Ripple's chat template has no thinking switch): the model reasoned first in every run, so the 512 measured tokens include its reasoning text. Median of 3 runs (range 14.9-14.9 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT. Reposted 2026-11-07 with corrected thinking wording; measured 2026-11-04.

Reactions

—

Submitted

Nov 8, 2026, 2:34 AM

Last edited

—

Model

AtomicChat/Ripple-65B-Preview-GGUF

Display name

Ripple-65B-Preview-GGUF

Base model

Ripple-65B-Preview

Revision

57066fdb2348a53f2261f096e0a793b9554b4106

Family

Llama

Parameters

65B

Active params

—

MoE

no

Output

15.2 tok/s

Prefill

360 tok/s

Total

102 tok/s

TTFT

11,392 ms

Peak VRAM

59.5GB

Power draw

177.4W