Archived leaderboard

Corpus 83-81e122a887e2, revision 2026-07-24, published 2026-07-24, grader 2026-07-20. This snapshot is immutable — cite this URL.

Revisions of this corpus: 2026-07-24 (68 runs) · 2026-07-22 (20 runs)

Rank Model Track k Score/100q Correct Incorrect Abstained No answer Truncated Reasoning Grounded Hallucination Cost
1–5 claude-opus-4.8 closed 3 +72.7 (+70…+75) 183/249 1 65 0 0 99%
[97–100]
23%
[15–33]
0%
[0–17]
$0.71
1–5 gemini-3.5-flash closed 3 +72.3 (+71…+73) 180/249 0 69 0 0 100%
[98–100]
18%
[11–27]
0%
[0–20]
$1.55
1–5 claude-fable-5 closed 3 +71.9 (+70…+73) 187/249 4 58 0 0 100%
[98–100]
26%
[18–36]
15%
[6–34]
$3.29
1–5 gemini-3.1-pro closed 3 +69.5 (+69…+70) 173/249 0 76 0 0 100%
[98–100]
10%
[5–18]
0/8
$2.24
1–5 claude-sonnet-5 closed 3 +66.3 (+63…+71) 177/249 6 66 0 0 98%
[95–99]
18%
[11–27]
17%
[6–39]
$0.33
6–8 grok-4.5 closed 3 +66.3 (+65…+67) 167/249 1 75 6 0 99%
[97–100]
11%
[6–19]
0/9
$0.22
6–8 kimi-k2.6 closed 3 +64.3 (+63…+65) 160/249 0 75 14 0 100%
[98–100]
6%
[3–14]
0/5
$1.58
6–8 qwen3-max closed 3 +64.3 (+57…+70) 176/249 8 65 0 0 99%
[97–100]
14%
[8–23]
37%
[19–59]
$0.23
9–11 deepseek-v4-pro closed 3 +57.4 (+51…+63) 171/249 14 60 4 0 97%
[93–99]
15%
[9–24]
43%
[24–63]
$0.16
9–11 llama-4-maverick closed 3 +52.2 (+51…+53) 162/249 16 71 0 0 94%
[89–97]
8%
[4–16]
46%
[23–71]
$0.04
9–11 claude-haiku-4.5 closed 3 +45.4 (+40…+51) 157/249 22 70 0 0 91%
[86–94]
8%
[4–16]
50%
[27–73]
$0.32
12–13 gpt-5.4 closed 3 +41.0 (+39…+43) 190/249 44 15 0 0 99%
[96–100]
32%
[23–43]
61%
[49–72]
$0.25
12–13 gpt-5.4-mini closed 3 +41.0 (+34…+47) 174/249 36 39 0 0 96%
[92–98]
19%
[12–29]
65%
[51–77]
$0.06
1–3 gemini-3.5-flash mcp 3 +73.5 (+70…+78) 197/249 7 41 4 0 100%
[98–100]
40%
[30–51]
18%
[9–33]
$11.61
1–3 gemini-3.1-pro mcp 1 +71.1 (±14.0) 65/83 3 15 0 0 100%
[93–100]
36%
[21–54]
23%
[8–50]
$6.25
1–3 claude-opus-4.8 mcp 1 +67.5 (±16.4) 66/83 5 12 0 0 100%
[93–100]
39%
[24–58]
31%
[14–56]
$2.38
4–7 claude-fable-5 mcp 3 +62.7 (+57…+66) 208/249 26 15 0 0 98%
[95–99]
55%
[44–65]
33%
[23–45]
$41.92
4–7 grok-4.5 mcp 3 +59.8 (+52…+71) 171/249 11 66 1 0 82%
[75–87]
44%
[34–55]
20%
[11–33]
$15.99
4–7 claude-sonnet-5 mcp 3 +59.0 (+57…+61) 193/249 23 33 0 0 99%
[96–100]
36%
[26–46]
41%
[29–55]
$7.91
4–7 deepseek-v4-pro mcp 1 +55.4 (±16.8) 56/83 5 20 2 0 89%
[78–95]
30%
[16–48]
38%
[18–64]
$1.50
8–9 gpt-5.4 mcp 3 +51.8 (+45…+55) 161/249 16 72 0 0 85%
[79–90]
24%
[16–34]
38%
[23–55]
$4.47
8–9 kimi-k2.6 mcp 1 +42.2 (±13.4) 39/83 2 36 6 0 64%
[50–75]
18%
[7–39]
2/6
$2.83
10 qwen3-max mcp 3 +42.2 (+41…+43) 165/249 30 52 2 0 88%
[83–92]
25%
[17–35]
58%
[44–71]
$4.44
11 claude-haiku-4.5 mcp 3 +36.1 (+34…+39) 128/249 19 102 0 0 64%
[56–71]
27%
[19–38]
39%
[26–55]
$3.87
12–13 gpt-5.4-mini mcp 3 +27.3 (+27…+29) 122/249 27 97 3 0 67%
[60–74]
15%
[9–25]
57%
[39–73]
$1.03
12–13 llama-4-maverick mcp 1 +10.8 (±25.4) 45/83 18 12 8 0 86%
[73–93]
12%
[4–29]
82%
[59–94]
$0.12

Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.

What tools actually buy: trust vs. accuracy, closed-book → MCP-grounded
Trust — Score/100q Grounded accuracy — % 0 25 50 0 25 50 75 100 gpt-5.4 +11 -8pp gemini-3.1-pro +2 +26pp gemini-3.5-flash +1 +22pp deepseek-v4-pro -2 +15pp claude-opus-4.8 -5 +17pp grok-4.5 -6 +33pp claude-sonnet-5 -7 +18pp claude-haiku-4.5 -9 +19pp claude-fable-5 -9 +29pp gpt-5.4-mini -14 -4pp kimi-k2.6 -22 +12pp qwen3-max -22 +11pp llama-4-maverick -41 +3pp

closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions