GSM8K
Official shard evalGSM8K deterministic disjoint eval shards from openai/gsm8k/main:test. Running all 13 shards covers the full 1,319-question dataset; Postgres stores only shard metadata and pass/fail results.
Category: ReasoningEval type: Shard evalQuestions: 1,319Shards: 13Runs: 290
Dry-run first:
lmx eval shard gsm8k --base-url http://localhost:8000 --questions 298 --dry-runThen submit with a real model and hardware profile:
lmx eval shard gsm8k --base-url http://localhost:8000 --questions 298 --model <hfId> --hardware hardware.json --submitScores are pooled by unique question_id; the leaderboard is ranked by Wilson 95% lower bound.
Leaderboard
pearsonkyle/Qwen3.8-27B-GPTQ-W4A16 · 13 runs · 13/13 shards · harness-scoped
97.8%
95% CI 96.9–98.5%
1219/1246 correct · 94.5% coverage
2
nvidia/Gemma-4-31B-IT-NVFP4 · 20 runs · 13/13 shards · harness-scoped
97.6%
95% CI 96.7–98.3%
1288/1319 correct · 100.0% coverage
3
local-inference-lab/Qwen3.8-Flash-Next-NVFP4 · 1 run · 1/13 shards · harness-scoped
100.0%
95% CI 96.2–100.0%
98/98 correct · 7.4% coverage
4
zai-org/GLM-5.3-Flash · 3 runs · 3/13 shards · harness-scoped
98.3%
95% CI 96.2–99.3%
298/303 correct · 23.0% coverage
5
unsloth/Qwen3.8-27B-GGUF · 11 runs · 11/13 shards · harness-scoped
97.2%
95% CI 96.1–98.0%
1085/1116 correct · 84.6% coverage
6
Qwen/Qwen3.6-35B-A3B · 13 runs · 13/13 shards · harness-scoped
97.0%
95% CI 95.9–97.8%
1239/1277 correct · 96.8% coverage
7
unsloth/Qwen3.6-35B-A3B-GGUF · 14 runs · 13/13 shards · harness-scoped
97.0%
95% CI 95.9–97.8%
1279/1319 correct · 100.0% coverage
8
unsloth/Qwen3.6-35B-A3B-GGUF · 13 runs · 13/13 shards · harness-scoped
96.8%
95% CI 95.7–97.6%
1277/1319 correct · 100.0% coverage
9
unsloth/Qwen3.8-27B-GGUF · 9 runs · 9/13 shards · harness-scoped
96.9%
95% CI 95.6–97.9%
884/912 correct · 69.1% coverage
10
Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF · 13 runs · 13/13 shards · harness-scoped
96.6%
95% CI 95.5–97.4%
1274/1319 correct · 100.0% coverage
11
HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive · 15 runs · 13/13 shards · harness-scoped
96.5%
95% CI 95.4–97.4%
1273/1319 correct · 100.0% coverage
12
HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive · 13 runs · 13/13 shards · harness-scoped
96.4%
95% CI 95.2–97.2%
1271/1319 correct · 100.0% coverage
13
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 · 9 runs · 9/13 shards · harness-scoped
96.5%
95% CI 95.1–97.5%
881/913 correct · 69.2% coverage
14
nvidia/Qwen3.8-Flash-Next-NVFP4 · 2 runs · 2/13 shards · harness-scoped
98.0%
95% CI 95.0–99.2%
196/200 correct · 15.2% coverage
15
nvidia/Qwen3.6-27B-NVFP4 · 14 runs · 13/13 shards · harness-scoped
96.1%
95% CI 95.0–97.0%
1268/1319 correct · 100.0% coverage
16
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 · 13 runs · 13/13 shards · harness-scoped
95.9%
95% CI 94.7–96.8%
1265/1319 correct · 100.0% coverage
17
unsloth/gemma-4-31B-it-GGUF · 1 run · 1/13 shards · harness-scoped
99.0%
95% CI 94.7–99.8%
101/102 correct · 7.7% coverage
18
Qwen/Qwen3.6-35B-A3B · 13 runs · 13/13 shards · harness-scoped
95.9%
95% CI 94.6–96.8%
1253/1307 correct · 99.1% coverage
19
HauhauCS/Qwen3.5-27B-Uncensored-HauhauCS-Aggressive · 1 run · 1/13 shards · harness-scoped
99.0%
95% CI 94.6–99.8%
100/101 correct · 7.7% coverage
20
Jackrong/Qwopus3.6-27B-Coder-MTP-GGUF · 1 run · 1/13 shards · harness-scoped
99.0%
95% CI 94.6–99.8%
100/101 correct · 7.7% coverage
21
Qwen/Qwen3.6-35B-A3B · 13 runs · 13/13 shards · harness-scoped
95.3%
95% CI 94.0–96.3%
1245/1307 correct · 99.1% coverage
22
nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4 · 1 run · 1/13 shards · harness-scoped
100.0%
95% CI 93.8–100.0%
58/58 correct · 4.4% coverage
23
brandonmusic/GLM-5.3-Flash-tr3-4bpw · 1 run · 1/13 shards · harness-scoped
98.0%
95% CI 93.1–99.5%
100/102 correct · 7.7% coverage
24
unsloth/Qwen3.6-27B-GGUF · 1 run · 1/13 shards · harness-scoped
98.0%
95% CI 93.1–99.5%
100/102 correct · 7.7% coverage
25
Qwen/Qwen3.6-35B-A3B · 1 run · 1/13 shards · harness-scoped
98.0%
95% CI 93.0–99.4%
98/100 correct · 7.6% coverage
26
unsloth/Qwen3.8-27B-GGUF · 1 run · 1/13 shards · harness-scoped
98.0%
95% CI 93.0–99.4%
98/100 correct · 7.6% coverage
27
unsloth/gemma-4-26B-A4B-it-qat-GGUF · 1 run · 1/13 shards · harness-scoped
98.0%
95% CI 92.9–99.4%
96/98 correct · 7.4% coverage
28
rdtand/Ornith-1.0-35B-PrismaAURA-4.75bit-vllm-MTP · 4 runs · 3/13 shards · harness-scoped
95.7%
95% CI 92.8–97.5%
290/303 correct · 23.0% coverage
29
palmfuture/Qwen3.6-35B-A3B-GPTQ-Int4 · 9 runs · 7/13 shards · harness-scoped
94.6%
95% CI 92.7–96.1%
672/710 correct · 53.8% coverage
30
zai-org/GLM-5.3-Flash · 2 runs · 2/13 shards · harness-scoped
96.1%
95% CI 92.4–98.0%
195/203 correct · 15.4% coverage
31
sakamakismile/Qwen3.8-27B-MTP-NVFP4 · 1 run · 1/13 shards · harness-scoped
97.1%
95% CI 91.7–99.0%
99/102 correct · 7.7% coverage
32
unsloth/Qwen3.5-27B-GGUF · 1 run · 1/13 shards · harness-scoped
97.1%
95% CI 91.7–99.0%
99/102 correct · 7.7% coverage
33
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 · 1 run · 1/13 shards · harness-scoped
97.1%
95% CI 91.7–99.0%
99/102 correct · 7.7% coverage
34
cyankiwi/Ornith-1.0-35B-AWQ-INT4 · 1 run · 1/13 shards · harness-scoped
97.0%
95% CI 91.6–99.0%
98/101 correct · 7.7% coverage
35
ggml-org/gpt-oss-20b-GGUF · 9 runs · 9/13 shards · harness-scoped
93.4%
95% CI 91.6–94.9%
852/912 correct · 69.1% coverage
36
unsloth/gemma-4-26B-A4B-it-qat-GGUF · 1 run · 1/13 shards · harness-scoped
97.0%
95% CI 91.5–99.0%
97/100 correct · 7.6% coverage
37
unsloth/Qwen3.8-27B-GGUF · 1 run · 1/13 shards · harness-scoped
97.0%
95% CI 91.5–99.0%
97/100 correct · 7.6% coverage
38
peculiar-ragdoll/Unsloth-Ornith-1.5-35B-A3B · 8 runs · 8/13 shards · harness-scoped
93.8%
95% CI 91.0–95.8%
377/402 correct · 30.5% coverage
39
ornith-ai/Ornith-1.5-35B-A3B-NVFP4 · 1 run · 1/13 shards · harness-scoped
96.1%
95% CI 90.3–98.5%
98/102 correct · 7.7% coverage
40
deepseek-ai/DeepSeek-V4-Flash-0731 · 1 run · 1/13 shards · harness-scoped
96.1%
95% CI 90.3–98.5%
98/102 correct · 7.7% coverage
41
unsloth/Muse-Glimmer-30B-GGUF · 1 run · 1/13 shards · harness-scoped
96.1%
95% CI 90.3–98.5%
98/102 correct · 7.7% coverage
42
philbert440/Qwen3.8-27B-W4A16-AWQ · 1 run · 1/13 shards · harness-scoped
96.1%
95% CI 90.3–98.5%
98/102 correct · 7.7% coverage
43
deepseek-ai/DeepSeek-V4-Flash-0731 · 1 run · 1/13 shards · harness-scoped
96.0%
95% CI 90.3–98.4%
97/101 correct · 7.7% coverage
44
poolside/Laguna-XS-2.1-INT4 · 1 run · 1/13 shards · harness-scoped
96.0%
95% CI 90.3–98.4%
97/101 correct · 7.7% coverage
45
unsloth/gemma-4-26B-A4B-it-GGUF · 1 run · 1/13 shards · harness-scoped
96.0%
95% CI 90.3–98.4%
97/101 correct · 7.7% coverage
46
Ar4ikov/KAT-Coder-V2.5-Dev-AWQ-W4A16-ASYM · 1 run · 1/13 shards · harness-scoped
96.0%
95% CI 90.1–98.4%
95/99 correct · 7.5% coverage
47
poolside/Laguna-XS-2.1-GGUF · 1 run · 1/13 shards · harness-scoped
95.9%
95% CI 89.9–98.4%
93/97 correct · 7.4% coverage
48
unsloth/Muse-Glimmer-30B-GGUF · 1 run · 1/13 shards · harness-scoped
95.0%
95% CI 88.9–97.9%
96/101 correct · 7.7% coverage
49
prism-ml/Bonsai-27B-gguf · 1 run · 1/13 shards · harness-scoped
95.0%
95% CI 88.8–97.8%
95/100 correct · 7.6% coverage
50
unsloth/gemma-4-12b-it-GGUF · 1 run · 1/13 shards · harness-scoped
95.0%
95% CI 88.8–97.8%
95/100 correct · 7.6% coverage
51
bartowski/Ling-3.0-tiny-GGUF · 6 runs · 6/13 shards · harness-scoped
91.3%
95% CI 88.8–93.3%
556/609 correct · 46.2% coverage
52
RedHatAI/Muse-Glimmer-30B-NVFP4 · 1 run · 1/13 shards · harness-scoped
94.1%
95% CI 87.8–97.3%
96/102 correct · 7.7% coverage
53
unsloth/Qwen3.8-27B-GGUF · 8 runs · 8/13 shards · harness-scoped
89.4%
95% CI 87.1–91.3%
726/812 correct · 61.6% coverage
54
poolside/Laguna-XS-2.1-NVFP4 · 1 run · 1/13 shards · harness-scoped
93.1%
95% CI 86.5–96.6%
95/102 correct · 7.7% coverage
55
Qwen/Qwen3-Coder-30B-A3B-Instruct · 1 run · 1/13 shards · harness-scoped
92.1%
95% CI 85.1–95.9%
93/101 correct · 7.7% coverage
56
google/gemma-4-E4B-it · 4 runs · 4/13 shards · harness-scoped
94.0%
95% CI 83.8–97.9%
47/50 correct · 3.8% coverage
57
AEON-7/Qwen3.6-27B-AEON-Ultimate-Uncensored-Text-NVFP4-MTP-XS · 1 run · 1/13 shards · harness-scoped
91.0%
95% CI 83.8–95.2%
91/100 correct · 7.6% coverage
58
google/gemma-4-26B-A4B-it-qat-q4_0-gguf · 1 run · 1/13 shards · harness-scoped
89.2%
95% CI 81.7–93.9%
91/102 correct · 7.7% coverage
59
ornith-ai/Ornith-1.0-9B-GGUF · 2 runs · 2/13 shards · harness-scoped
88.2%
95% CI 76.6–94.5%
45/51 correct · 3.9% coverage
60
Qwen/Qwen3.6-35B-A3B · 1 run · 1/13 shards · harness-scoped
72.0%
95% CI 62.5–79.9%
72/100 correct · 7.6% coverage
61
tencent/Hy-MT2-7B · 1 run · 1/13 shards · harness-scoped
68.0%
95% CI 54.2–79.2%
34/50 correct · 3.8% coverage
62
openbmb/MiniCPM5-1B-GGUF · 1 run · 1/13 shards · harness-scoped
80.0%
95% CI 49.0–94.3%
8/10 correct · 0.8% coverage
63
Jackrong/DeepSeek-V4-Pro-Qwen3.5-9B-MTP-GGUF · 1 run · 1/13 shards · harness-scoped
100.0%
95% CI 34.2–100.0%
2/2 correct · 0.2% coverage
64
ornith-ai/Ornith-1.0-9B · 1 run · 1/13 shards · harness-scoped
75.0%
95% CI 30.1–95.4%
3/4 correct · 0.3% coverage
Stability— historical rerun transparency
Leaderboard rank uses the canonical latest approved answer per question_id. These metrics include historical submissions too, so reruns and changed answers are visible but do not drive rank.
13 historical runs · 1,246 unique questions
Canonical
97.8%
Row avg
97.8%
Run avg
98.0%
Repeated
0
Changed
0
20 historical runs · 1,319 unique questions
Canonical
97.6%
Row avg
97.4%
Run avg
97.4%
Repeated
711
Changed
4
1 historical run · 98 unique questions
Canonical
100.0%
Row avg
100.0%
Run avg
100.0%
Repeated
0
Changed
0
3 historical runs · 303 unique questions
Canonical
98.3%
Row avg
98.3%
Run avg
98.4%
Repeated
0
Changed
0
11 historical runs · 1,116 unique questions
Canonical
97.2%
Row avg
97.2%
Run avg
97.2%
Repeated
0
Changed
0
13 historical runs · 1,277 unique questions
Canonical
97.0%
Row avg
97.0%
Run avg
97.0%
Repeated
0
Changed
0
14 historical runs · 1,319 unique questions
Canonical
97.0%
Row avg
97.0%
Run avg
97.0%
Repeated
101
Changed
0
13 historical runs · 1,319 unique questions
Canonical
96.8%
Row avg
96.8%
Run avg
96.8%
Repeated
0
Changed
0
9 historical runs · 912 unique questions
Canonical
96.9%
Row avg
96.9%
Run avg
96.9%
Repeated
0
Changed
0
13 historical runs · 1,319 unique questions
Canonical
96.6%
Row avg
96.6%
Run avg
96.6%
Repeated
0
Changed
0
15 historical runs · 1,319 unique questions
Canonical
96.5%
Row avg
96.7%
Run avg
96.7%
Repeated
202
Changed
0
13 historical runs · 1,319 unique questions
Canonical
96.4%
Row avg
96.4%
Run avg
96.4%
Repeated
0
Changed
0
9 historical runs · 913 unique questions
Canonical
96.5%
Row avg
96.5%
Run avg
96.5%
Repeated
0
Changed
0
2 historical runs · 200 unique questions
Canonical
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
14 historical runs · 1,319 unique questions
Canonical
96.1%
Row avg
95.4%
Run avg
95.4%
Repeated
102
Changed
10
13 historical runs · 1,319 unique questions
Canonical
95.9%
Row avg
95.9%
Run avg
95.9%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
99.0%
Row avg
99.0%
Run avg
99.0%
Repeated
0
Changed
0
13 historical runs · 1,307 unique questions
Canonical
95.9%
Row avg
95.9%
Run avg
95.9%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Canonical
99.0%
Row avg
99.0%
Run avg
99.0%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Canonical
99.0%
Row avg
99.0%
Run avg
99.0%
Repeated
0
Changed
0
13 historical runs · 1,307 unique questions
Canonical
95.3%
Row avg
95.3%
Run avg
95.3%
Repeated
0
Changed
0
1 historical run · 58 unique questions
Canonical
100.0%
Row avg
100.0%
Run avg
100.0%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
1 historical run · 100 unique questions
Canonical
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
1 historical run · 100 unique questions
Canonical
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
1 historical run · 98 unique questions
Canonical
98.0%
Row avg
98.0%
Run avg
98.0%
Repeated
0
Changed
0
4 historical runs · 303 unique questions
Canonical
95.7%
Row avg
96.0%
Run avg
96.0%
Repeated
98
Changed
1
9 historical runs · 710 unique questions
Canonical
94.6%
Row avg
94.7%
Run avg
94.7%
Repeated
203
Changed
7
2 historical runs · 203 unique questions
Canonical
96.1%
Row avg
96.1%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
97.1%
Row avg
97.1%
Run avg
97.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
97.1%
Row avg
97.1%
Run avg
97.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
97.1%
Row avg
97.1%
Run avg
97.1%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Canonical
97.0%
Row avg
97.0%
Run avg
97.0%
Repeated
0
Changed
0
9 historical runs · 912 unique questions
Canonical
93.4%
Row avg
93.4%
Run avg
93.4%
Repeated
0
Changed
0
0 historical runs · 100 unique questions
Canonical
97.0%
Row avg
—
Run avg
—
Repeated
0
Changed
0
1 historical run · 100 unique questions
Canonical
97.0%
Row avg
97.0%
Run avg
97.0%
Repeated
0
Changed
0
8 historical runs · 402 unique questions
Canonical
93.8%
Row avg
93.8%
Run avg
82.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
96.1%
Row avg
96.1%
Run avg
96.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
96.1%
Row avg
96.1%
Run avg
96.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
96.1%
Row avg
96.1%
Run avg
96.1%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
96.1%
Row avg
96.1%
Run avg
96.1%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Canonical
96.0%
Row avg
96.0%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Canonical
96.0%
Row avg
96.0%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Canonical
96.0%
Row avg
96.0%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 99 unique questions
Canonical
96.0%
Row avg
96.0%
Run avg
96.0%
Repeated
0
Changed
0
1 historical run · 97 unique questions
Canonical
95.9%
Row avg
95.9%
Run avg
95.9%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Canonical
95.0%
Row avg
95.0%
Run avg
95.0%
Repeated
0
Changed
0
1 historical run · 100 unique questions
Canonical
95.0%
Row avg
95.0%
Run avg
95.0%
Repeated
0
Changed
0
0 historical runs · 100 unique questions
Canonical
95.0%
Row avg
—
Run avg
—
Repeated
0
Changed
0
6 historical runs · 609 unique questions
Canonical
91.3%
Row avg
91.3%
Run avg
91.3%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
94.1%
Row avg
94.1%
Run avg
94.1%
Repeated
0
Changed
0
8 historical runs · 812 unique questions
Canonical
89.4%
Row avg
89.4%
Run avg
89.4%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
93.1%
Row avg
93.1%
Run avg
93.1%
Repeated
0
Changed
0
1 historical run · 101 unique questions
Canonical
92.1%
Row avg
92.1%
Run avg
92.1%
Repeated
0
Changed
0
4 historical runs · 50 unique questions
Canonical
94.0%
Row avg
94.0%
Run avg
92.2%
Repeated
0
Changed
0
1 historical run · 100 unique questions
Canonical
91.0%
Row avg
91.0%
Run avg
91.0%
Repeated
0
Changed
0
1 historical run · 102 unique questions
Canonical
89.2%
Row avg
89.2%
Run avg
89.2%
Repeated
0
Changed
0
2 historical runs · 51 unique questions
Canonical
88.2%
Row avg
88.2%
Run avg
82.2%
Repeated
0
Changed
0
0 historical runs · 100 unique questions
Canonical
72.0%
Row avg
—
Run avg
—
Repeated
0
Changed
0
1 historical run · 50 unique questions
Canonical
68.0%
Row avg
68.0%
Run avg
68.0%
Repeated
0
Changed
0
1 historical run · 10 unique questions
Canonical
80.0%
Row avg
80.0%
Run avg
80.0%
Repeated
0
Changed
0
1 historical run · 2 unique questions
Canonical
100.0%
Row avg
100.0%
Run avg
100.0%
Repeated
0
Changed
0
1 historical run · 4 unique questions
Canonical
75.0%
Row avg
75.0%
Run avg
75.0%
Repeated
0
Changed
0
