Archived leaderboard
Corpus 307-bf8e3bdb8cf5, revision 2026-11-08, published 2026-11-08, grader 2026-10-31b. This snapshot is immutable — cite this URL.
| Rank | Model | Track | k | Score/100q | Correct | Incorrect | Abstained | No answer | Truncated | Reasoning | Grounded | Hallucination | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1–5 | claude-fable-5.1 | closed | 1 | +25.9 (±10.5) | 149/305 | 35 | 121 | 0 | 0 | 100% [92–100] | 40% [35–46] | 25% [18–33] | $15.37 |
| 1–5 | claude-opus-5.5 | closed | 1 | +25.2 (±10.4) | 145/305 | 34 | 126 | 0 | 0 | 100% [92–100] | 39% [33–45] | 25% [18–33] | $5.06 |
| 1–5 | grok-4.7 | closed | 1 | +24.9 (±6.0) | 86/305 | 5 | 214 | 0 | 0 | 93% [81–98] | 18% [13–23] | 6% [2–17] | $3.34 |
| 1–5 | claude-sonnet-5.5 | closed | 1 | +18.4 (±10.6) | 132/305 | 38 | 135 | 0 | 0 | 100% [92–100] | 34% [29–40] | 30% [23–38] | $2.44 |
| 1–5 | claude-haiku-5.5 | closed | 1 | +8.5 (±9.2) | 86/305 | 30 | 189 | 0 | 0 | 100% [92–100] | 16% [12–21] | 41% [31–53] | $0.14 |
| 6–8 | gemma4-12b | closed | 1 | -68.7 (±14.1) | 40/262 | 110 | 112 | 0 | 43 | 68% [49–82] | 9% [6–13] | 83% [75–88] | $0.00 |
| 6–8 | deepseek-r1-14b | closed | 1 | -82.8 (±13.3) | 42/302 | 146 | 114 | 0 | 3 | 58% [42–71] | 7% [5–11] | 87% [81–92] | $0.00 |
| 6–8 | gpt-oss-20b | closed | 1 | -90.8 (±13.5) | 45/303 | 160 | 98 | 0 | 2 | 71% [56–83] | 6% [4–9] | 91% [85–94] | $0.00 |
| 1–2 | claude-opus-5.5 | mcp | 3 | +98.7 (+98…+99) | 908/912 | 4 | 0 | 0 | 0 | 100% [97–100] | 99% [99–100] | 1% [0–1] | $155.93 |
| 1–2 | claude-fable-5.1 | mcp | 3 | +98.6 (+98…+100) | 907/912 | 4 | 0 | 1 | 0 | 100% [97–100] | 99% [99–100] | 1% [0–1] | $368.93 |
| 3–4 | claude-sonnet-5.5 | mcp | 3 | +93.3 (+92…+95) | 887/912 | 18 | 6 | 1 | 0 | 95% [90–98] | 98% [96–99] | 2% [1–3] | $29.80 |
| 3–4 | grok-4.7 | mcp | 3 | +92.8 (+92…+93) | 876/912 | 15 | 21 | 0 | 0 | 96% [91–98] | 96% [94–97] | 2% [1–3] | $73.43 |
| 5 | claude-haiku-5.5 | mcp | 3 | +85.3 (+85…+86) | 840/912 | 31 | 39 | 2 | 0 | 94% [88–97] | 92% [90–94] | 4% [3–5] | $3.70 |
| 6 | gemma4-12b | mcp | 3 | +42.0 (+38…+49) | 649/889 | 138 | 101 | 1 | 23 | 74% [66–81] | 73% [70–76] | 18% [15–21] | $0.00 |
| 7 | gpt-oss-20b | mcp | 3 | +30.5 (+24…+34) | 558/910 | 140 | 17 | 195 | 2 | 75% [66–83] | 78% [75–82] | 19% [16–23] | $0.00 |
Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.
closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions