Archived leaderboard

Corpus 278-8bc1c4f8addd, revision 2026-10-17, published 2026-10-16, grader 2026-10-10. This snapshot is immutable — cite this URL.

Rank Model Track k Score/100q Correct Incorrect Abstained No answer Truncated Reasoning Grounded Hallucination Cost
1–4 claude-opus-5.5 closed 1 +29.7 (±10.2) 132/276 25 119 0 0 98%
[88–100]
39%
[33–45]
21%
[15–29]
$4.49
1–4 claude-fable-5.1 closed 1 +28.6 (±10.9) 139/276 30 107 0 0 100%
[92–100]
41%
[35–48]
24%
[17–32]
$13.57
1–4 grok-4.7 closed 1 +27.2 (±6.5) 85/276 5 186 0 0 91%
[78–96]
20%
[15–25]
4%
[1–14]
$2.98
1–4 claude-sonnet-5.5 closed 1 +13.8 (±12.4) 128/276 45 103 0 0 100%
[92–100]
36%
[31–43]
35%
[27–43]
$2.21
5 claude-haiku-5.5 closed 1 +4.7 (±10.2) 77/276 32 167 0 0 95%
[85–99]
15%
[11–21]
45%
[34–57]
$0.13
6–8 gemma4-12b closed 1 -73.4 (±14.6) 30/237 102 105 0 39 73%
[56–86]
4%
[2–7]
92%
[85–96]
$0.00
6–8 gpt-oss-20b closed 1 -90.9 (±14.1) 40/275 145 88 2 1 68%
[52–80]
6%
[3–9]
91%
[85–95]
$0.00
6–8 deepseek-r1-14b closed 1 -93.1 (±13.8) 34/275 145 96 0 1 52%
[38–67]
5%
[3–9]
91%
[85–95]
$0.00
1–2 claude-opus-5.5 mcp 3 +83.5 (+82…+85) 717/828 13 37 61 0 98%
[93–99]
93%
[90–94]
2%
[1–3]
$109.31
1–2 claude-fable-5.1 mcp 2 +82.1 (+82…+82) 489/552 18 31 14 0 98%
[92–99]
90%
[86–92]
4%
[2–6]
$305.61
3–4 claude-sonnet-5.5 mcp 3 +72.8 (+71…+74) 699/828 48 47 34 0 94%
[88–97]
87%
[84–89]
6%
[5–9]
$43.64
3–4 grok-4.7 mcp 2 +71.4 (+71…+72) 436/552 21 95 0 0 93%
[86–97]
76%
[72–80]
5%
[3–7]
$54.31
5 claude-haiku-5.5 mcp 3 +62.7 (+61…+65) 629/828 55 99 45 0 92%
[86–96]
78%
[75–81]
9%
[7–11]
$4.19
6 gpt-oss-20b mcp 3 +4.5 (+3…+6) 383/828 173 44 228 0 74%
[65–81]
62%
[57–66]
32%
[28–37]
$0.00
7 gemma4-12b mcp 3 -3.5 (-7…+2) 400/802 214 186 2 26 70%
[62–78]
46%
[43–50]
38%
[34–42]
$0.00

Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.

What tools actually buy: trust vs. accuracy, closed-book → MCP-grounded
Trust — Score/100q Grounded accuracy — % -50 0 50 0 25 50 75 100 gpt-oss-20b +95 +56pp gemma4-12b +70 +43pp claude-sonnet-5.5 +59 +50pp claude-haiku-5.5 +58 +63pp claude-opus-5.5 +54 +54pp claude-fable-5.1 +53 +48pp grok-4.7 +44 +57pp

closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions