cheezz

@cheezz · Member since 2026

← Leaderboard

Mainly llama.cpp 8 approved runs

Submissions

Models

Avg out

115

Avg prefill

4133.5

Avg total

514.9

Avg TTFT

145.8ms

HW configs

Personal bests7

Fastest approved result for each model.

Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M

JetBrains/Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M

265.5 tok/s

3B MoE1 run

Output265.5 tok/s

Prefill7615.8 tok/s

TTFT67ms

Depth—

RTX 5070 11.9GB

llama.cpp · Q4_K_M · 6d ago

Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M

JetBrains/Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M

261.4 tok/s

3B MoE1 run

Output261.4 tok/s

Prefill7526.3 tok/s

TTFT68ms

Depth—

RTX 5070 11.9GB

llama.cpp · Q4_K_M · 6d ago

gemma-4-12B-it-GGUF

ggml-org/gemma-4-12B-it-GGUF

72.1 tok/s

12B1 run

Output72.1 tok/s

Prefill3173.2 tok/s

TTFT161ms

Depth—

RTX 5070 11.9GB

llama.cpp · Q4_K_M · 6d ago

Gemmable-4-12B-MTP-GGUF

Mia-AiLab/Gemmable-4-12B-MTP-GGUF

71.2 tok/s

12B1 run

Output71.2 tok/s

Prefill3138.2 tok/s

TTFT163ms

Depth—

RTX 5070 11.9GB

llama.cpp · Q4_K_M · 6d ago

Grug-12B-GGUF

kai-os/Grug-12B-GGUF

69.2 tok/s

12B1 run

Output69.2 tok/s

Prefill3045.7 tok/s

TTFT168ms

Depth—

RTX 5070 11.9GB

llama.cpp · Q4_K_M · 6d ago

Qwen3-14B-GGUF

unsloth/Qwen3-14B-GGUF

66.6 tok/s

14B1 run

Output66.6 tok/s

Prefill2749.3 tok/s

TTFT186ms

Depth—

RTX 5070 11.9GB

llama.cpp · Q4_K_M · 6d ago

gemma-4-12b-it-GGUF

unsloth/gemma-4-12b-it-GGUF

62.8 tok/s

12B2 runs

Output62.8 tok/s

Prefill3050.4 tok/s

TTFT168ms

Depth—

RTX 5070 11.9GB

llama.cpp · UD-Q5_K_XL · 6d ago

All runs8

Full approved benchmark history.

Gemmable-4-12B-MTP-GGUF

12B · Gemma

71.2

tok/s

Hardware

RTX 5070 11.9GB

Engine

llama.cpp · Q4_K_M

TTFT

163ms

Context

2k · 6d ago

Show all run details

Model

Mia-AiLab/Gemmable-4-12B-MTP-GGUF

Display name

Gemmable-4-12B-MTP-GGUF

Base model

—

Revision

main

Family

Gemma

Parameters

12B

Active params

—

MoE

Output tok/s

71.2

Prefill tok/s

3138.2

Total tok/s

326.5

TTFT

163.1ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

512

Output tokens

128

Prefill tokens

—

Context length

2048

Batch size

Hardware class

DISCRETE_GPU

Hardware

RTX 5070 11.9GB

GPU slots

—

GPU count

VRAM

11.9GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen 7 9800X3D 8-Core Processor

RAM

—

linux

Power

—

Engine

llama.cpp

Engine version

—

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

/home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/huggingface/Mia-AiLab--Gemmable-4-12B-MTP-GGUF/gemmable-4-12b-Q4_K_M.gguf' -p 512 -n 128

Extra flags

--p 512

Notes

—

Reactions

—

Submitted

Jul 5, 2026, 12:10 AM

Last edited

—

Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M

3B MoE · Llama

265.5

tok/s

Hardware

RTX 5070 11.9GB

Engine

llama.cpp · Q4_K_M

TTFT

67ms

Context

2k · 6d ago

Show all run details

Model

JetBrains/Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M

Display name

Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M

Base model

Mellum2-12B-A2.5B-Thinking

Revision

main

Family

Llama

Parameters

12B

Active params

MoE

yes

Output tok/s

265.5

Prefill tok/s

7615.8

Total tok/s

1164.9

TTFT

67.2ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

512

Output tokens

128

Prefill tokens

—

Context length

2048

Batch size

Hardware class

DISCRETE_GPU

Hardware

RTX 5070 11.9GB

GPU slots

—

GPU count

VRAM

11.9GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen 7 9800X3D 8-Core Processor

RAM

—

linux

Power

—

Engine

llama.cpp

Engine version

—

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

/home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/huggingface/JetBrains--Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M/Mellum2-12B-A2.5B-Thinking-Q4_K_M.gguf' -p 512 -n 128

Extra flags

--p 512

Notes

—

Reactions

—

Submitted

Jul 5, 2026, 12:10 AM

Last edited

—

gemma-4-12b-it-GGUF

12B · Gemma

51.3

tok/s

Hardware

RTX 5070 11.9GB

Engine

llama.cpp · UD-Q6_K_XL

TTFT

185ms

Context

2k · 6d ago

Show all run details

Model

unsloth/gemma-4-12b-it-GGUF

Display name

gemma-4-12b-it-GGUF

Base model

gemma-4-12B-it

Revision

main

Family

Gemma

Parameters

12B

Active params

—

MoE

Output tok/s

51.3

Prefill tok/s

2768.8

Total tok/s

238.6

TTFT

184.9ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

512

Output tokens

128

Prefill tokens

—

Context length

2048

Batch size

Hardware class

DISCRETE_GPU

Hardware

RTX 5070 11.9GB

GPU slots

—

GPU count

VRAM

11.9GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen 7 9800X3D 8-Core Processor

RAM

—

linux

Power

—

Engine

llama.cpp

Engine version

—

Quantization

UD-Q6_K_XL

Backend

cuda

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

/home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/gguf-quants/gemma-4-12b-it-unsloth/gemma-4-12b-it-UD-Q6_K_XL.gguf' -p 512 -n 128

Extra flags

--p 512

Notes

—

Reactions

—

Submitted

Jul 5, 2026, 12:06 AM

Last edited

—

gemma-4-12b-it-GGUF

12B · Gemma

62.8

tok/s

Hardware

RTX 5070 11.9GB

Engine

llama.cpp · UD-Q5_K_XL

TTFT

168ms

Context

2k · 6d ago

Show all run details

Model

unsloth/gemma-4-12b-it-GGUF

Display name

gemma-4-12b-it-GGUF

Base model

gemma-4-12B-it

Revision

main

Family

Gemma

Parameters

12B

Active params

—

MoE

Output tok/s

62.8

Prefill tok/s

3050.4

Total tok/s

289.9

TTFT

167.8ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

512

Output tokens

128

Prefill tokens

—

Context length

2048

Batch size

Hardware class

DISCRETE_GPU

Hardware

RTX 5070 11.9GB

GPU slots

—

GPU count

VRAM

11.9GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen 7 9800X3D 8-Core Processor

RAM

—

linux

Power

—

Engine

llama.cpp

Engine version

—

Quantization

UD-Q5_K_XL

Backend

cuda

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

/home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/gguf-quants/gemma-4-12b-it-unsloth/gemma-4-12b-it-UD-Q5_K_XL.gguf' -p 512 -n 128

Extra flags

--p 512

Notes

—

Reactions

—

Submitted

Jul 5, 2026, 12:03 AM

Last edited

—

Qwen3-14B-GGUF

14B · Qwen

66.6

tok/s

Hardware

RTX 5070 11.9GB

Engine

llama.cpp · Q4_K_M

TTFT

186ms

Context

2k · 6d ago

Show all run details

Model

unsloth/Qwen3-14B-GGUF

Display name

Qwen3-14B-GGUF

Base model

Qwen3-14B

Revision

main

Family

Qwen

Parameters

14B

Active params

—

MoE

Output tok/s

66.6

Prefill tok/s

2749.3

Total tok/s

303.5

TTFT

186.2ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

512

Output tokens

128

Prefill tokens

—

Context length

2048

Batch size

Hardware class

DISCRETE_GPU

Hardware

RTX 5070 11.9GB

GPU slots

—

GPU count

VRAM

11.9GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen 7 9800X3D 8-Core Processor

RAM

—

linux

Power

—

Engine

llama.cpp

Engine version

—

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

/home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/llama.cpp-models/qwen3-14b/Qwen3-14B-Q4_K_M.gguf' -p 512 -n 128

Extra flags

--p 512

Notes

—

Reactions

—

Submitted

Jul 4, 2026, 11:51 PM

Last edited

—

Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M

3B MoE · Llama

261.4

tok/s

Hardware

RTX 5070 11.9GB

Engine

llama.cpp · Q4_K_M

TTFT

68ms

Context

2k · 6d ago

Show all run details

Model

JetBrains/Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M

Display name

Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M

Base model

Mellum2-12B-A2.5B-Instruct

Revision

main

Family

Llama

Parameters

12B

Active params

MoE

yes

Output tok/s

261.4

Prefill tok/s

7526.3

Total tok/s

1147.6

TTFT

68ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

512

Output tokens

128

Prefill tokens

—

Context length

2048

Batch size

Hardware class

DISCRETE_GPU

Hardware

RTX 5070 11.9GB

GPU slots

—

GPU count

VRAM

11.9GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen 7 9800X3D 8-Core Processor

RAM

—

linux

Power

—

Engine

llama.cpp

Engine version

—

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

/home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/huggingface/JetBrains--Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M/Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf' -p 512 -n 128

Extra flags

--p 512

Notes

—

Reactions

—

Submitted

Jul 4, 2026, 11:48 PM

Last edited

—

Grug-12B-GGUF

12B · Llama

69.2

tok/s

Hardware

RTX 5070 11.9GB

Engine

llama.cpp · Q4_K_M

TTFT

168ms

Context

2k · 6d ago

Show all run details

Model

kai-os/Grug-12B-GGUF

Display name

Grug-12B-GGUF

Base model

Grug-12B

Revision

main

Family

Llama

Parameters

12B

Active params

—

MoE

Output tok/s

69.2

Prefill tok/s

3045.7

Total tok/s

317.3

TTFT

168.1ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

512

Output tokens

128

Prefill tokens

—

Context length

2048

Batch size

Hardware class

DISCRETE_GPU

Hardware

RTX 5070 11.9GB

GPU slots

—

GPU count

VRAM

11.9GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen 7 9800X3D 8-Core Processor

RAM

—

linux

Power

—

Engine

llama.cpp

Engine version

—

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

/home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/huggingface/kai-os--Grug-12B-GGUF/Grug-12B-Q4_K_M.gguf' -p 512 -n 128

Extra flags

--p 512

Notes

—

Reactions

—

Submitted

Jul 4, 2026, 11:48 PM

Last edited

—

gemma-4-12B-it-GGUF

12B · Gemma

72.1

tok/s

Hardware

RTX 5070 11.9GB

Engine

llama.cpp · Q4_K_M

TTFT

161ms

Context

2k · 6d ago

Show all run details

Model

ggml-org/gemma-4-12B-it-GGUF

Display name

gemma-4-12B-it-GGUF

Base model

gemma-4-12B-it

Revision

main

Family

Gemma

Parameters

12B

Active params

—

MoE

Output tok/s

72.1

Prefill tok/s

3173.2

Total tok/s

330.5

TTFT

161.4ms

Peak VRAM

—

Power draw

—

Hardware cost

—

Prompt tokens

512

Output tokens

128

Prefill tokens

—

Context length

2048

Batch size

Hardware class

DISCRETE_GPU

Hardware

RTX 5070 11.9GB

GPU slots

—

GPU count

VRAM

11.9GB

Chip vendor

—

Chip family

—

Chip variant

—

Unified memory

—

NPU TOPS

—

CPU

AMD Ryzen 7 9800X3D 8-Core Processor

RAM

—

linux

Power

—

Engine

llama.cpp

Engine version

—

Quantization

Q4_K_M

Backend

cuda

Tensor parallel

—

Pipeline parallel

—

GPU layers

—

Split mode

—

KV cache dtype

—

KV cache size

—

Prefix caching

—

Attention backend

—

Flash attention

—

Chunked prefill

—

Prefill chunk

—

Continuous batching

—

CPU offload

—

CPU layers

—

Rope scaling

—

Rope scale

—

Yarn ext factor

—

Engine quant

—

SGLang quant

—

GPU mem util

—

Max running seqs

—

Scheduler delay

—

Num parallel

—

Concurrency

—

Spec decoding

Spec method

—

Spec model

—

Spec draft model

—

Spec tokens

—

Spec ngram

—

Spec draft TP

—

MTP enabled

MTP draft layers

—

Temperature

—

Top P

—

Top K

—

Min P

—

Repeat penalty

—

Mirostat

—

Command

/home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/gguf-quants/gemma-4-12b-it-ggml/gemma-4-12B-it-Q4_K_M.gguf' -p 512 -n 128

Extra flags

--p 512

Notes

—

Reactions

—

Submitted

Jul 4, 2026, 11:45 PM

Last edited

—

Model	Hardware	Engine	tok/s out	prefill	tok/s total	TTFT	Depth
Gemmable-4-12B-MTP-GGUF 12B · Gemma	RTX 5070 11.9GB	llama.cppQ4_K_M	71.2	3138.2	326.5	163ms	—
Show all details for Gemmable-4-12B-MTP-GGUF Model Mia-AiLab/Gemmable-4-12B-MTP-GGUF Display name Gemmable-4-12B-MTP-GGUF Base model — Revision main Family Gemma Parameters 12B Active params — MoE no Output tok/s 71.2 Prefill tok/s 3138.2 Total tok/s 326.5 TTFT 163.1ms Peak VRAM — Power draw — Hardware cost — Prompt tokens 512 Output tokens 128 Prefill tokens — Context length 2048 Batch size 1 Hardware class DISCRETE_GPU Hardware RTX 5070 11.9GB GPU slots — GPU count 1 VRAM 11.9GB Chip vendor — Chip family — Chip variant — Unified memory — NPU TOPS — CPU AMD Ryzen 7 9800X3D 8-Core Processor RAM — OS linux Power — Engine llama.cpp Engine version — Quantization Q4_K_M Backend cuda Tensor parallel — Pipeline parallel — GPU layers — Split mode — KV cache dtype — KV cache size — Prefix caching — Attention backend — Flash attention — Chunked prefill — Prefill chunk — Continuous batching — CPU offload — CPU layers — Rope scaling — Rope scale — Yarn ext factor — Engine quant — SGLang quant — GPU mem util — Max running seqs — Scheduler delay — Num parallel — Concurrency — Spec decoding no Spec method — Spec model — Spec draft model — Spec tokens — Spec ngram — Spec draft TP — MTP enabled no MTP draft layers — Temperature — Top P — Top K — Min P — Repeat penalty — Mirostat — Command /home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/huggingface/Mia-AiLab--Gemmable-4-12B-MTP-GGUF/gemmable-4-12b-Q4_K_M.gguf' -p 512 -n 128 Extra flags --p 512 Notes — Reactions — Submitted Jul 5, 2026, 12:10 AM Last edited —
Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M 3B MoE · Llama	RTX 5070 11.9GB	llama.cppQ4_K_M	265.5	7615.8	1164.9	67ms	—
Show all details for Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M Model JetBrains/Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M Display name Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M Base model Mellum2-12B-A2.5B-Thinking Revision main Family Llama Parameters 12B Active params 3B MoE yes Output tok/s 265.5 Prefill tok/s 7615.8 Total tok/s 1164.9 TTFT 67.2ms Peak VRAM — Power draw — Hardware cost — Prompt tokens 512 Output tokens 128 Prefill tokens — Context length 2048 Batch size 1 Hardware class DISCRETE_GPU Hardware RTX 5070 11.9GB GPU slots — GPU count 1 VRAM 11.9GB Chip vendor — Chip family — Chip variant — Unified memory — NPU TOPS — CPU AMD Ryzen 7 9800X3D 8-Core Processor RAM — OS linux Power — Engine llama.cpp Engine version — Quantization Q4_K_M Backend cuda Tensor parallel — Pipeline parallel — GPU layers — Split mode — KV cache dtype — KV cache size — Prefix caching — Attention backend — Flash attention — Chunked prefill — Prefill chunk — Continuous batching — CPU offload — CPU layers — Rope scaling — Rope scale — Yarn ext factor — Engine quant — SGLang quant — GPU mem util — Max running seqs — Scheduler delay — Num parallel — Concurrency — Spec decoding no Spec method — Spec model — Spec draft model — Spec tokens — Spec ngram — Spec draft TP — MTP enabled no MTP draft layers — Temperature — Top P — Top K — Min P — Repeat penalty — Mirostat — Command /home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/huggingface/JetBrains--Mellum2-12B-A2.5B-Thinking-GGUF-Q4_K_M/Mellum2-12B-A2.5B-Thinking-Q4_K_M.gguf' -p 512 -n 128 Extra flags --p 512 Notes — Reactions — Submitted Jul 5, 2026, 12:10 AM Last edited —
gemma-4-12b-it-GGUF 12B · Gemma	RTX 5070 11.9GB	llama.cppUD-Q6_K_XL	51.3	2768.8	238.6	185ms	—
Show all details for gemma-4-12b-it-GGUF Model unsloth/gemma-4-12b-it-GGUF Display name gemma-4-12b-it-GGUF Base model gemma-4-12B-it Revision main Family Gemma Parameters 12B Active params — MoE no Output tok/s 51.3 Prefill tok/s 2768.8 Total tok/s 238.6 TTFT 184.9ms Peak VRAM — Power draw — Hardware cost — Prompt tokens 512 Output tokens 128 Prefill tokens — Context length 2048 Batch size 1 Hardware class DISCRETE_GPU Hardware RTX 5070 11.9GB GPU slots — GPU count 1 VRAM 11.9GB Chip vendor — Chip family — Chip variant — Unified memory — NPU TOPS — CPU AMD Ryzen 7 9800X3D 8-Core Processor RAM — OS linux Power — Engine llama.cpp Engine version — Quantization UD-Q6_K_XL Backend cuda Tensor parallel — Pipeline parallel — GPU layers — Split mode — KV cache dtype — KV cache size — Prefix caching — Attention backend — Flash attention — Chunked prefill — Prefill chunk — Continuous batching — CPU offload — CPU layers — Rope scaling — Rope scale — Yarn ext factor — Engine quant — SGLang quant — GPU mem util — Max running seqs — Scheduler delay — Num parallel — Concurrency — Spec decoding no Spec method — Spec model — Spec draft model — Spec tokens — Spec ngram — Spec draft TP — MTP enabled no MTP draft layers — Temperature — Top P — Top K — Min P — Repeat penalty — Mirostat — Command /home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/gguf-quants/gemma-4-12b-it-unsloth/gemma-4-12b-it-UD-Q6_K_XL.gguf' -p 512 -n 128 Extra flags --p 512 Notes — Reactions — Submitted Jul 5, 2026, 12:06 AM Last edited —
gemma-4-12b-it-GGUF 12B · Gemma	RTX 5070 11.9GB	llama.cppUD-Q5_K_XL	62.8	3050.4	289.9	168ms	—
Show all details for gemma-4-12b-it-GGUF Model unsloth/gemma-4-12b-it-GGUF Display name gemma-4-12b-it-GGUF Base model gemma-4-12B-it Revision main Family Gemma Parameters 12B Active params — MoE no Output tok/s 62.8 Prefill tok/s 3050.4 Total tok/s 289.9 TTFT 167.8ms Peak VRAM — Power draw — Hardware cost — Prompt tokens 512 Output tokens 128 Prefill tokens — Context length 2048 Batch size 1 Hardware class DISCRETE_GPU Hardware RTX 5070 11.9GB GPU slots — GPU count 1 VRAM 11.9GB Chip vendor — Chip family — Chip variant — Unified memory — NPU TOPS — CPU AMD Ryzen 7 9800X3D 8-Core Processor RAM — OS linux Power — Engine llama.cpp Engine version — Quantization UD-Q5_K_XL Backend cuda Tensor parallel — Pipeline parallel — GPU layers — Split mode — KV cache dtype — KV cache size — Prefix caching — Attention backend — Flash attention — Chunked prefill — Prefill chunk — Continuous batching — CPU offload — CPU layers — Rope scaling — Rope scale — Yarn ext factor — Engine quant — SGLang quant — GPU mem util — Max running seqs — Scheduler delay — Num parallel — Concurrency — Spec decoding no Spec method — Spec model — Spec draft model — Spec tokens — Spec ngram — Spec draft TP — MTP enabled no MTP draft layers — Temperature — Top P — Top K — Min P — Repeat penalty — Mirostat — Command /home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/gguf-quants/gemma-4-12b-it-unsloth/gemma-4-12b-it-UD-Q5_K_XL.gguf' -p 512 -n 128 Extra flags --p 512 Notes — Reactions — Submitted Jul 5, 2026, 12:03 AM Last edited —
Qwen3-14B-GGUF 14B · Qwen	RTX 5070 11.9GB	llama.cppQ4_K_M	66.6	2749.3	303.5	186ms	—
Show all details for Qwen3-14B-GGUF Model unsloth/Qwen3-14B-GGUF Display name Qwen3-14B-GGUF Base model Qwen3-14B Revision main Family Qwen Parameters 14B Active params — MoE no Output tok/s 66.6 Prefill tok/s 2749.3 Total tok/s 303.5 TTFT 186.2ms Peak VRAM — Power draw — Hardware cost — Prompt tokens 512 Output tokens 128 Prefill tokens — Context length 2048 Batch size 1 Hardware class DISCRETE_GPU Hardware RTX 5070 11.9GB GPU slots — GPU count 1 VRAM 11.9GB Chip vendor — Chip family — Chip variant — Unified memory — NPU TOPS — CPU AMD Ryzen 7 9800X3D 8-Core Processor RAM — OS linux Power — Engine llama.cpp Engine version — Quantization Q4_K_M Backend cuda Tensor parallel — Pipeline parallel — GPU layers — Split mode — KV cache dtype — KV cache size — Prefix caching — Attention backend — Flash attention — Chunked prefill — Prefill chunk — Continuous batching — CPU offload — CPU layers — Rope scaling — Rope scale — Yarn ext factor — Engine quant — SGLang quant — GPU mem util — Max running seqs — Scheduler delay — Num parallel — Concurrency — Spec decoding no Spec method — Spec model — Spec draft model — Spec tokens — Spec ngram — Spec draft TP — MTP enabled no MTP draft layers — Temperature — Top P — Top K — Min P — Repeat penalty — Mirostat — Command /home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/llama.cpp-models/qwen3-14b/Qwen3-14B-Q4_K_M.gguf' -p 512 -n 128 Extra flags --p 512 Notes — Reactions — Submitted Jul 4, 2026, 11:51 PM Last edited —
Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M 3B MoE · Llama	RTX 5070 11.9GB	llama.cppQ4_K_M	261.4	7526.3	1147.6	68ms	—
Show all details for Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M Model JetBrains/Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M Display name Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M Base model Mellum2-12B-A2.5B-Instruct Revision main Family Llama Parameters 12B Active params 3B MoE yes Output tok/s 261.4 Prefill tok/s 7526.3 Total tok/s 1147.6 TTFT 68ms Peak VRAM — Power draw — Hardware cost — Prompt tokens 512 Output tokens 128 Prefill tokens — Context length 2048 Batch size 1 Hardware class DISCRETE_GPU Hardware RTX 5070 11.9GB GPU slots — GPU count 1 VRAM 11.9GB Chip vendor — Chip family — Chip variant — Unified memory — NPU TOPS — CPU AMD Ryzen 7 9800X3D 8-Core Processor RAM — OS linux Power — Engine llama.cpp Engine version — Quantization Q4_K_M Backend cuda Tensor parallel — Pipeline parallel — GPU layers — Split mode — KV cache dtype — KV cache size — Prefix caching — Attention backend — Flash attention — Chunked prefill — Prefill chunk — Continuous batching — CPU offload — CPU layers — Rope scaling — Rope scale — Yarn ext factor — Engine quant — SGLang quant — GPU mem util — Max running seqs — Scheduler delay — Num parallel — Concurrency — Spec decoding no Spec method — Spec model — Spec draft model — Spec tokens — Spec ngram — Spec draft TP — MTP enabled no MTP draft layers — Temperature — Top P — Top K — Min P — Repeat penalty — Mirostat — Command /home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/huggingface/JetBrains--Mellum2-12B-A2.5B-Instruct-GGUF-Q4_K_M/Mellum2-12B-A2.5B-Instruct-Q4_K_M.gguf' -p 512 -n 128 Extra flags --p 512 Notes — Reactions — Submitted Jul 4, 2026, 11:48 PM Last edited —
Grug-12B-GGUF 12B · Llama	RTX 5070 11.9GB	llama.cppQ4_K_M	69.2	3045.7	317.3	168ms	—
Show all details for Grug-12B-GGUF Model kai-os/Grug-12B-GGUF Display name Grug-12B-GGUF Base model Grug-12B Revision main Family Llama Parameters 12B Active params — MoE no Output tok/s 69.2 Prefill tok/s 3045.7 Total tok/s 317.3 TTFT 168.1ms Peak VRAM — Power draw — Hardware cost — Prompt tokens 512 Output tokens 128 Prefill tokens — Context length 2048 Batch size 1 Hardware class DISCRETE_GPU Hardware RTX 5070 11.9GB GPU slots — GPU count 1 VRAM 11.9GB Chip vendor — Chip family — Chip variant — Unified memory — NPU TOPS — CPU AMD Ryzen 7 9800X3D 8-Core Processor RAM — OS linux Power — Engine llama.cpp Engine version — Quantization Q4_K_M Backend cuda Tensor parallel — Pipeline parallel — GPU layers — Split mode — KV cache dtype — KV cache size — Prefix caching — Attention backend — Flash attention — Chunked prefill — Prefill chunk — Continuous batching — CPU offload — CPU layers — Rope scaling — Rope scale — Yarn ext factor — Engine quant — SGLang quant — GPU mem util — Max running seqs — Scheduler delay — Num parallel — Concurrency — Spec decoding no Spec method — Spec model — Spec draft model — Spec tokens — Spec ngram — Spec draft TP — MTP enabled no MTP draft layers — Temperature — Top P — Top K — Min P — Repeat penalty — Mirostat — Command /home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/huggingface/kai-os--Grug-12B-GGUF/Grug-12B-Q4_K_M.gguf' -p 512 -n 128 Extra flags --p 512 Notes — Reactions — Submitted Jul 4, 2026, 11:48 PM Last edited —
gemma-4-12B-it-GGUF 12B · Gemma	RTX 5070 11.9GB	llama.cppQ4_K_M	72.1	3173.2	330.5	161ms	—
Show all details for gemma-4-12B-it-GGUF Model ggml-org/gemma-4-12B-it-GGUF Display name gemma-4-12B-it-GGUF Base model gemma-4-12B-it Revision main Family Gemma Parameters 12B Active params — MoE no Output tok/s 72.1 Prefill tok/s 3173.2 Total tok/s 330.5 TTFT 161.4ms Peak VRAM — Power draw — Hardware cost — Prompt tokens 512 Output tokens 128 Prefill tokens — Context length 2048 Batch size 1 Hardware class DISCRETE_GPU Hardware RTX 5070 11.9GB GPU slots — GPU count 1 VRAM 11.9GB Chip vendor — Chip family — Chip variant — Unified memory — NPU TOPS — CPU AMD Ryzen 7 9800X3D 8-Core Processor RAM — OS linux Power — Engine llama.cpp Engine version — Quantization Q4_K_M Backend cuda Tensor parallel — Pipeline parallel — GPU layers — Split mode — KV cache dtype — KV cache size — Prefix caching — Attention backend — Flash attention — Chunked prefill — Prefill chunk — Continuous batching — CPU offload — CPU layers — Rope scaling — Rope scale — Yarn ext factor — Engine quant — SGLang quant — GPU mem util — Max running seqs — Scheduler delay — Num parallel — Concurrency — Spec decoding no Spec method — Spec model — Spec draft model — Spec tokens — Spec ngram — Spec draft TP — MTP enabled no MTP draft layers — Temperature — Top P — Top K — Min P — Repeat penalty — Mirostat — Command /home/cheez/Projects/local-llm/llama.cpp/build-cuda-sm120/bin/llama-bench -m '/home/cheez/Projects/local-llm/models/gguf-quants/gemma-4-12b-it-ggml/gemma-4-12B-it-Q4_K_M.gguf' -p 512 -n 128 Extra flags --p 512 Notes — Reactions — Submitted Jul 4, 2026, 11:45 PM Last edited —