Archived leaderboard
Corpus 308-97da788a5741, revision 2026-11-05, published 2026-11-05, grader 2026-10-31b. This snapshot is immutable — cite this URL.
| Rank | Model | Track | k | Score/100q | Correct | Incorrect | Abstained | No answer | Truncated | Reasoning | Grounded | Hallucination | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1–3 | grok-4.7 | closed | 1 | +28.0 (±5.5) | 90/307 | 2 | 215 | 0 | 0 | 95% [85–99] | 19% [14–24] | 4% [1–13] | $3.16 |
| 1–3 | claude-opus-5.5 | closed | 1 | +24.8 (±10.0) | 138/307 | 31 | 138 | 0 | 0 | 100% [92–100] | 36% [30–42] | 25% [18–33] | $4.86 |
| 1–3 | claude-fable-5.1 | closed | 1 | +18.6 (±10.9) | 139/307 | 41 | 127 | 0 | 0 | 100% [92–100] | 36% [31–42] | 30% [23–38] | $14.91 |
| 4–5 | claude-sonnet-5.5 | closed | 1 | +9.4 (±11.5) | 129/307 | 50 | 128 | 0 | 0 | 100% [92–100] | 33% [27–38] | 37% [29–45] | $2.44 |
| 4–5 | claude-haiku-5.5 | closed | 1 | +6.2 (±9.1) | 81/307 | 31 | 195 | 0 | 0 | 100% [92–100] | 14% [11–19] | 45% [34–57] | $0.14 |
| 6–8 | gemma4-12b | closed | 1 | -66.1 (±13.8) | 43/274 | 112 | 119 | 0 | 33 | 69% [51–82] | 9% [6–13] | 83% [75–89] | $0.00 |
| 6–8 | deepseek-r1-14b | closed | 1 | -81.6 (±13.1) | 41/305 | 145 | 118 | 1 | 2 | 60% [45–74] | 6% [4–10] | 88% [82–93] | $0.00 |
| 6–8 | gpt-oss-20b | closed | 1 | -86.6 (±13.4) | 45/306 | 155 | 104 | 2 | 1 | 60% [44–73] | 8% [5–11] | 87% [81–92] | $0.00 |
| 1–2 | claude-opus-5.5 | mcp | 3 | +98.7 (+98…+100) | 910/918 | 2 | 2 | 4 | 0 | 100% [97–100] | 99% [99–100] | 0% [0–1] | $76.93 |
| 1–2 | claude-fable-5.1 | mcp | 3 | +98.1 (+98…+98) | 909/918 | 4 | 3 | 2 | 0 | 100% [97–100] | 99% [98–100] | 1% [0–1] | $378.90 |
| 3–4 | claude-sonnet-5.5 | mcp | 3 | +93.8 (+92…+96) | 897/918 | 18 | 1 | 2 | 0 | 95% [90–98] | 98% [97–99] | 2% [1–3] | $29.91 |
| 3–4 | grok-4.7 | mcp | 3 | +92.3 (+91…+94) | 879/918 | 16 | 23 | 0 | 0 | 95% [90–98] | 96% [94–97] | 1% [1–3] | $70.88 |
| 5 | claude-haiku-5.5 | mcp | 3 | +85.0 (+84…+87) | 848/918 | 34 | 35 | 1 | 0 | 95% [89–97] | 92% [90–94] | 4% [3–6] | $3.81 |
| 6 | gemma4-12b | mcp | 3 | +39.9 (+36…+43) | 649/895 | 146 | 97 | 3 | 23 | 69% [60–77] | 73% [70–76] | 18% [16–21] | $0.00 |
| 7 | gpt-oss-20b | mcp | 3 | +26.2 (+23…+31) | 544/916 | 152 | 29 | 191 | 2 | 75% [66–82] | 75% [72–78] | 21% [18–25] | $0.00 |
Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.
closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions