Verified 4w ago Mainly llama.cpp 200 approved runs
Submissions
3,189
200 approved public runs
Models
11
unique models
Avg output
82.9tok/s
Best 275 tok/s
Avg prefill
3,227tok/s
Best 18,370 tok/s
Avg total
943tok/s
Best 7,929 tok/s
Avg TTFT
38,338ms
Best 31.2 ms
Hardware
4
distinct setups
Matrix runs
351
speed test matrices
Matrix runs351+
Matrix runs 351
A matrix run measures prompt reading (prefill) and generation (decode) speed in one go, across several context lengths and numbers of simultaneous requests. Showing the newest 10.
llama-server -m TwIL-8B-Q8_0-verbose.gguf -ngl 999 -c 32768 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off
Extra flags
--ctk q8_0
--ctv q8_0
--fit off
Notes
GGUF file: TwIL-8B-Q8_0-verbose.gguf (Q8_0). TwIL-8B by webAI (Granite-4.2-8B post-trained with SFT + GRPO/DAPO), verbose variant: webAI publishes Q8_0 and F16 GGUFs only for verbose; same architecture as balanced. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q8_0 KV, 1 slot, 32768 context (native 131072 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q8_0. 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 16383 tokens of context (16k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 16383. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 24.4-24.4 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.
llama-server -m TwIL-8B-Q8_0-verbose.gguf -ngl 999 -c 32768 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off
Extra flags
--ctk q8_0
--ctv q8_0
--fit off
Notes
GGUF file: TwIL-8B-Q8_0-verbose.gguf (Q8_0). TwIL-8B by webAI (Granite-4.2-8B post-trained with SFT + GRPO/DAPO), verbose variant: webAI publishes Q8_0 and F16 GGUFs only for verbose; same architecture as balanced. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q8_0 KV, 1 slot, 32768 context (native 131072 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q8_0. 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 8192 tokens of context (8k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 8192. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 28.7-28.7 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.
llama-server -m TwIL-8B-Q8_0-verbose.gguf -ngl 999 -c 32768 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off
Extra flags
--ctk q8_0
--ctv q8_0
--fit off
Notes
GGUF file: TwIL-8B-Q8_0-verbose.gguf (Q8_0). TwIL-8B by webAI (Granite-4.2-8B post-trained with SFT + GRPO/DAPO), verbose variant: webAI publishes Q8_0 and F16 GGUFs only for verbose; same architecture as balanced. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q8_0 KV, 1 slot, 32768 context (native 131072 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q8_0. 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 4096 tokens of context (4k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 4096. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 31.4-31.4 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.
llama-server -m TwIL-8B-Q8_0-verbose.gguf -ngl 999 -c 32768 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off
Extra flags
--ctk q8_0
--ctv q8_0
--fit off
Notes
GGUF file: TwIL-8B-Q8_0-verbose.gguf (Q8_0). TwIL-8B by webAI (Granite-4.2-8B post-trained with SFT + GRPO/DAPO), verbose variant: webAI publishes Q8_0 and F16 GGUFs only for verbose; same architecture as balanced. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q8_0 KV, 1 slot, 32768 context (native 131072 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q8_0. 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 2048 tokens of context (2k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 2048. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 33.0-33.0 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.
llama-server -m TwIL-8B-Q8_0-verbose.gguf -ngl 999 -c 32768 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off
Extra flags
--ctk q8_0
--ctv q8_0
--fit off
Notes
GGUF file: TwIL-8B-Q8_0-verbose.gguf (Q8_0). TwIL-8B by webAI (Granite-4.2-8B post-trained with SFT + GRPO/DAPO), verbose variant: webAI publishes Q8_0 and F16 GGUFs only for verbose; same architecture as balanced. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q8_0 KV, 1 slot, 32768 context (native 131072 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q8_0. 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 16383 tokens of context (16k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 16383. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 24.4-24.4 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.
llama-server -m TwIL-8B-Q8_0-verbose.gguf -ngl 999 -c 32768 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off
Extra flags
--ctk q8_0
--ctv q8_0
--fit off
Notes
GGUF file: TwIL-8B-Q8_0-verbose.gguf (Q8_0). TwIL-8B by webAI (Granite-4.2-8B post-trained with SFT + GRPO/DAPO), verbose variant: webAI publishes Q8_0 and F16 GGUFs only for verbose; same architecture as balanced. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q8_0 KV, 1 slot, 32768 context (native 131072 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q8_0. 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 8192 tokens of context (8k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 8192. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 28.7-28.7 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.
llama-server -m TwIL-8B-Q8_0-verbose.gguf -ngl 999 -c 32768 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off
Extra flags
--ctk q8_0
--ctv q8_0
--fit off
Notes
GGUF file: TwIL-8B-Q8_0-verbose.gguf (Q8_0). TwIL-8B by webAI (Granite-4.2-8B post-trained with SFT + GRPO/DAPO), verbose variant: webAI publishes Q8_0 and F16 GGUFs only for verbose; same architecture as balanced. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q8_0 KV, 1 slot, 32768 context (native 131072 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q8_0. 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 4096 tokens of context (4k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 4096. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 31.4-31.4 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.
llama-server -m TwIL-8B-Q8_0-verbose.gguf -ngl 999 -c 32768 -ctk q8_0 -ctv q8_0 -np 1 -fa on -fit off
Extra flags
--ctk q8_0
--ctv q8_0
--fit off
Notes
GGUF file: TwIL-8B-Q8_0-verbose.gguf (Q8_0). TwIL-8B by webAI (Granite-4.2-8B post-trained with SFT + GRPO/DAPO), verbose variant: webAI publishes Q8_0 and F16 GGUFs only for verbose; same architecture as balanced. llama.cpp llama-server b12030 (official ghcr.io/ggml-org/llama.cpp server-cuda12 image), full GPU offload, FlashAttention on, Q8_0 KV, 1 slot, 32768 context (native 131072 does not fit with Q4_0 KV), prompt cache off per request. Spec decode: none. KV cache: q8_0. 1x RTX 3060 12GB, stock 170 W limit, PCIe Gen3 x16 | Intel Xeon E5-2680 v4 14C | 32 GB DDR4-2400 | Ubuntu 24.04.5, NVIDIA driver 580.178.04. Depth run: 2048 tokens of context (2k target). Method: system message = cache-bust nonce + filler (CPython 3.12 standard-library source, cut to length with the model's own tokenizer), then LocalMaxxing canonical prompt reasoning-v1 verbatim as the final user turn; the whole conversation is a cold prefill, so promptTokens = prefillTokens = 2048. Batch 1, streaming, temperature 0, 512 output tokens (ignore_eos), thinking off. Median of 3 runs (range 33.0-33.0 tok/s). tokSOut = (completion - 1) / time after first token; tokSPrefill = promptTokens / TTFT.
