Full leaderboard
Every run in full. Use the buttons to show one track, and click Score, Correct or Cost to sort. What each column means is below the table.
| Rank | Model | Track | k | Score/100q | Correct | Incorrect | Abstained | No answer | Truncated | Reasoning | Grounded | Hallucination | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1–5 | claude-fable-5.1 | closed | 1 | +25.9 (±10.5) | 149/305 | 35 | 121 | 0 | 0 | 100% [92–100] | 40% [35–46] | 25% [18–33] | $15.37 |
| 1–5 | claude-opus-5.5 | closed | 1 | +25.2 (±10.4) | 145/305 | 34 | 126 | 0 | 0 | 100% [92–100] | 39% [33–45] | 25% [18–33] | $5.06 |
| 1–5 | grok-4.7 | closed | 1 | +24.9 (±6.0) | 86/305 | 5 | 214 | 0 | 0 | 93% [81–98] | 18% [13–23] | 6% [2–17] | $3.34 |
| 1–5 | claude-sonnet-5.5 | closed | 1 | +18.4 (±10.6) | 132/305 | 38 | 135 | 0 | 0 | 100% [92–100] | 34% [29–40] | 30% [23–38] | $2.44 |
| 1–5 | claude-haiku-5.5 | closed | 1 | +8.5 (±9.2) | 86/305 | 30 | 189 | 0 | 0 | 100% [92–100] | 16% [12–21] | 41% [31–53] | $0.14 |
| 6–8 | gemma4-12b | closed | 1 | -68.7 (±14.1) | 40/262 | 110 | 112 | 0 | 43 | 68% [49–82] | 9% [6–13] | 83% [75–88] | $0.00 |
| 6–8 | deepseek-r1-14b | closed | 1 | -82.8 (±13.3) | 42/302 | 146 | 114 | 0 | 3 | 58% [42–71] | 7% [5–11] | 87% [81–92] | $0.00 |
| 6–8 | gpt-oss-20b | closed | 1 | -90.8 (±13.5) | 45/303 | 160 | 98 | 0 | 2 | 71% [56–83] | 6% [4–9] | 91% [85–94] | $0.00 |
| 1–2 | claude-opus-5.5 | mcp | 3 | +98.7 (+98…+99) | 908/912 | 4 | 0 | 0 | 0 | 100% [97–100] | 99% [99–100] | 1% [0–1] | $155.93 |
| 1–2 | claude-fable-5.1 | mcp | 3 | +98.6 (+98…+100) | 907/912 | 4 | 0 | 1 | 0 | 100% [97–100] | 99% [99–100] | 1% [0–1] | $368.93 |
| 3–4 | claude-sonnet-5.5 | mcp | 3 | +93.3 (+92…+95) | 887/912 | 18 | 6 | 1 | 0 | 95% [90–98] | 98% [96–99] | 2% [1–3] | $29.80 |
| 3–4 | grok-4.7 | mcp | 3 | +92.8 (+92…+93) | 876/912 | 15 | 21 | 0 | 0 | 96% [91–98] | 96% [94–97] | 2% [1–3] | $73.43 |
| 5 | claude-haiku-5.5 | mcp | 3 | +85.3 (+85…+86) | 840/912 | 31 | 39 | 2 | 0 | 94% [88–97] | 92% [90–94] | 4% [3–5] | $3.70 |
| 6 | gemma4-12b | mcp | 3 | +42.0 (+38…+49) | 649/889 | 138 | 101 | 1 | 23 | 74% [66–81] | 73% [70–76] | 18% [15–21] | $0.00 |
| 7 | gpt-oss-20b | mcp | 3 | +30.5 (+24…+34) | 558/910 | 140 | 17 | 195 | 2 | 75% [66–83] | 78% [75–82] | 19% [16–23] | $0.00 |
Score/100q = (correct − 2×incorrect) per 100 gradeable questions; abstention is free. The −2 penalty encodes a 67% confidence threshold. Gradeable = total minus transport errors and truncated completions, and the Correct column is shown over that same denominator. The small figure beside each score is its uncertainty: for k>1 the lowest and highest score/100q observed across those repeat runs, and for k=1 a ±95% interval — repeats re-ask the same questions, so treating them as independent trials would understate the spread. Shared ranks (e.g. 1–10) group models whose interval overlaps the group leader's. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.
What the columns mean
- Track
- closed: the AI answers on its own. mcp: the same AI connected to Orbit Sentinel through MCP tools.
- k
- How many repeat runs are pooled in the row. Tool tracks ran three times; closed-book once.
- Score/100q
- Right answers minus twice the wrong ones, per 100 gradeable questions. +100 is perfect; abstaining scores 0, so answering only pays when the AI is more than two-thirds sure.
- Band or ±
- Beside each score: the lowest and highest score across repeat runs where k > 1, or the 95% interval where k = 1. Ranks are shared when this figure overlaps the group leader.
- Correct
- Right answers out of gradeable ones (total minus provider errors and replies cut off at the length cap).
- Incorrect
- Wrong answers. Each costs 2 points.
- Abstained
- The AI declined to answer ("UNKNOWN"). Free.
- No answer
- An empty reply or one that broke the answer format, such as running out of tool turns. Counts as gradeable and earns 0.
- Truncated
- Reply cut off at the 8,192-token cap; left out of the denominator.
- Errors
- Provider faults. The column appears only when a published run has any; every error in this release was rerun.
- Reasoning / Grounded
- Accuracy on the 43 reasoning questions (history, orbital mechanics) and on the data questions, with Wilson 95% intervals.
- Hallucination
- Wrong answers as a share of answers given to data questions: how often an answer, once given, is wrong.
- Cost
- List-price cost of the run (API-equivalent for Claude on a subscription; xAI's billed figure for Grok; local models are $0).
closed-book MCP-grounded amber Δ = accuracy and trust moved in opposite directions
Each line runs from the AI on its own (grey) to the same AI with the tools (blue). Left: the calibrated score. Right: accuracy on data questions.
closed-book MCP-grounded
Higher is better, further left is cheaper. Local models cost nothing to run and are not shown.
Data and citation
Raw data
The JSON bundle behind this page: leaderboard, corpus and all 29 runs, CC BY 4.0.
Downloads and versions →Cite
@misc{spacebench2026,
title = {SpaceBench: Can you trust AI for space work?},
author = {Anthony Vincent Caracappa},
year = {2026},
note = {Corpus 307-bf8e3bdb8cf5, revision 2026-11-08},
url = {https://spacebench.net/}
}