Archived leaderboard

Corpus 307-bf8e3bdb8cf5, revision 2026-11-08, published 2026-11-08, grader 2026-10-31b. This snapshot is immutable — cite this URL.

Rank Model Track k Score/100q Correct Incorrect Abstained No answer Truncated Reasoning Grounded Hallucination Cost
1–5 claude-fable-5.1 closed 1 +25.9 (±10.5) 149/305 35 121 0 0 100%
[92–100]
40%
[35–46]
25%
[18–33]
$15.37
1–5 claude-opus-5.5 closed 1 +25.2 (±10.4) 145/305 34 126 0 0 100%
[92–100]
39%
[33–45]
25%
[18–33]
$5.06
1–5 grok-4.7 closed 1 +24.9 (±6.0) 86/305 5 214 0 0 93%
[81–98]
18%
[13–23]
6%
[2–17]
$3.34
1–5 claude-sonnet-5.5 closed 1 +18.4 (±10.6) 132/305 38 135 0 0 100%
[92–100]
34%
[29–40]
30%
[23–38]
$2.44
1–5 claude-haiku-5.5 closed 1 +8.5 (±9.2) 86/305 30 189 0 0 100%
[92–100]
16%
[12–21]
41%
[31–53]
$0.14
6–8 gemma4-12b closed 1 -68.7 (±14.1) 40/262 110 112 0 43 68%
[49–82]
9%
[6–13]
83%
[75–88]
$0.00
6–8 deepseek-r1-14b closed 1 -82.8 (±13.3) 42/302 146 114 0 3 58%
[42–71]
7%
[5–11]
87%
[81–92]
$0.00
6–8 gpt-oss-20b closed 1 -90.8 (±13.5) 45/303 160 98 0 2 71%
[56–83]
6%
[4–9]
91%
[85–94]
$0.00
1–2 claude-opus-5.5 mcp 3 +98.7 (+98…+99) 908/912 4 0 0 0 100%
[97–100]
99%
[99–100]
1%
[0–1]
$155.93
1–2 claude-fable-5.1 mcp 3 +98.6 (+98…+100) 907/912 4 0 1 0 100%
[97–100]
99%
[99–100]
1%
[0–1]
$368.93
3–4 claude-sonnet-5.5 mcp 3 +93.3 (+92…+95) 887/912 18 6 1 0 95%
[90–98]
98%
[96–99]
2%
[1–3]
$29.80
3–4 grok-4.7 mcp 3 +92.8 (+92…+93) 876/912 15 21 0 0 96%
[91–98]
96%
[94–97]
2%
[1–3]
$73.43
5 claude-haiku-5.5 mcp 3 +85.3 (+85…+86) 840/912 31 39 2 0 94%
[88–97]
92%
[90–94]
4%
[3–5]
$3.70
6 gemma4-12b mcp 3 +42.0 (+38…+49) 649/889 138 101 1 23 74%
[66–81]
73%
[70–76]
18%
[15–21]
$0.00
7 gpt-oss-20b mcp 3 +30.5 (+24…+34) 558/910 140 17 195 2 75%
[66–83]
78%
[75–82]
19%
[16–23]
$0.00

Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.

What tools actually buy: trust vs. accuracy, closed-book → MCP-grounded
Trust — Score/100q Grounded accuracy — % -50 0 50 0 25 50 75 100 gpt-oss-20b +122 +73pp gemma4-12b +110 +64pp claude-haiku-5.5 +77 +76pp claude-sonnet-5.5 +75 +64pp claude-opus-5.5 +73 +61pp claude-fable-5.1 +73 +59pp grok-4.7 +68 +78pp

closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions