Eval Suites

The canonical HuggingFace Open LLM Leaderboard suite: MMLU, ARC Challenge, HellaSwag, WinoGrande, TruthfulQA MC2, and GSM8K with official few-shot settings. Weighted mean aggregate.

Discrete Reasoning Over Paragraphs. Reading-comprehension benchmark requiring numerical and symbolic reasoning over passages.

A collection of challenging BIG-Bench tasks selected because prior models performed poorly. Covers symbolic reasoning, algorithmic reasoning, and language understanding.

Graduate-level Google-proof Q&A benchmark focused on biology, physics, and chemistry. The Diamond split is the highest-quality expert-validated subset.

Large-scale Winograd schema challenge for commonsense reasoning. Fill-in-the-blank pronoun resolution requiring world knowledge.

Sentence completion benchmark testing grounded commonsense inference. Models must pick the most plausible continuation of an activity description.

AI2 Reasoning Challenge (Challenge set) — grade-school science questions that require reasoning beyond simple retrieval. Harder subset of ARC.

Massive Multitask Language Understanding — 57-subject academic exam covering STEM, humanities, social sciences, and more. The gold-standard broad-knowledge benchmark.

A lightweight 10-question sanity check for locally served models. Designed for the trusted /api/evals/execute path.