Can you trust AI for space work?

Not on its own. The top AI, Claude Opus 5.5, got 1 in 4 of the space-data answers it gave wrong from memory. Connected to live data, about 1 in 200.

SpaceBench is an open benchmark for AI models (LLMs) in the space industry. It asks 8 AIs 307 questions a space professional actually asks, about satellites, debris, licensing and ground stations, and grades every answer against a published key.

AIs tested
8
Questions
307
Graded answers
8,824

Full leaderboard → How we test Get the data

By , Founder and Self-Directed Space Researcher, Viventine Space Systems · updated

How often the answer given was wrong (hallucination rate)

Data questions only (counts, dates, IDs, filings), on the questions both runs answered. Declining to answer is not counted as wrong.

Each bar is drawn on a 0% to 100% scale. Alone: the AI answers from what it learned in training. With data: the same AI connected to Orbit Sentinel's live satellite and regulatory database through MCP tools.

Scores

Every AI answered every question twice. Scores with the data pool three repeat runs; scores on its own are one run.

With Orbit Sentinel data On its own
Rank AI Score Wrong Score Wrong
1–2 Claude Opus 5.5 Anthropic +98.7 0.5% +25.2 25%
1–2 Claude Fable 5.1 Anthropic +98.6 0.5% +25.9 25%
3–4 Claude Sonnet 5.5 Anthropic +93.3 1.5% +18.4 30%
3–4 Grok 4.7 xAI +92.8 1.6% +24.9 6.1%
5 Claude Haiku 5.5 Anthropic +85.3 3.7% +8.5 41%
6 Gemma 4 12B Google +42.0 18% -68.7 83%
7 gpt-oss 20B OpenAI +30.5 19% -90.8 91%
— DeepSeek-R1 14B DeepSeek not run — -82.8 87%

Score: right answers minus twice the wrong ones, per 100 questions, so +100 is perfect and guessing does not pay. Saying "I don't know" scores 0. Ranks are shared where AIs cannot be told apart across repeat runs.

Every count, cost and confidence interval →

Which AI can you rely on for which job?

Green, trust it: wrong 5% of the time or less and right 80% or more. Red, don't use it: wrong more than 20% of the time. Amber: check its answers. Each cell shows % right / % wrong; click a job for its questions.

Model CatalogConstellationsDebrisRegulatoryMarketGroundBaseline
Grok 4.7
xAI
alone
42 / 0
16 / 0
14 / 2
28 / 0
22 / 5
13 / 5
96 / 0
tools
100 / 0
95 / 5
94 / 2
96 / 1
100 / 0
93 / 3
97 / 0
Claude Fable 5.1
Anthropic
alone
64 / 7
53 / 14
45 / 14
38 / 5
49 / 15
28 / 28
100 / 0
tools
100 / 0
99 / 1
99 / 0
98 / 2
100 / 0
99 / 1
100 / 0
Claude Opus 5.5
Anthropic
alone
76 / 9
63 / 7
40 / 24
33 / 5
34 / 20
23 / 18
100 / 0
tools
100 / 0
100 / 0
99 / 1
97 / 3
100 / 0
100 / 0
100 / 0
Claude Sonnet 5.5
Anthropic
alone
60 / 11
42 / 7
40 / 21
31 / 8
37 / 24
28 / 18
100 / 0
tools
100 / 0
99 / 1
96 / 2
93 / 4
99 / 0
96 / 4
99 / 1
Claude Haiku 5.5
Anthropic
alone
27 / 24
26 / 16
19 / 10
23 / 8
27 / 10
13 / 3
100 / 0
tools
100 / 0
87 / 12
93 / 0
82 / 3
94 / 2
89 / 6
97 / 0
gpt-oss 20B
OpenAI · open
alone
7 / 67
12 / 62
5 / 69
10 / 56
2 / 66
5 / 45
96 / 4
tools
92 / 1
36 / 28
58 / 13
48 / 18
70 / 11
46 / 20
85 / 5
Gemma 4 12B
Google · open
alone
9 / 60
9 / 53
13 / 36
14 / 47
6 / 44
10 / 36
85 / 15
tools
99 / 1
61 / 26
61 / 21
68 / 18
74 / 14
61 / 24
85 / 1
DeepSeek-R1 14B
DeepSeek · open
alone
7 / 60
19 / 49
5 / 67
8 / 51
5 / 46
13 / 43
68 / 32
tools
—
—
—
—
—
—
—

Each cell: % right, then % wrong (the rest declined to answer). Color is the verdict: Trust it · Verify its answers · Don't use it. Hover for the 95% range.

How we test

  1. 1. Same questions, asked twice

    Each AI answers on its own, then again connected to Orbit Sentinel's satellite and regulatory database through MCP tools.

  2. 2. Every key is checkable

    Each data answer is frozen from a published SQL query on a stamped date and checked against public sources. See the questions.

  3. 3. Wrong costs double

    A right answer earns 1, a wrong one costs 2, "I don't know" costs nothing, so a confident guess does not pay. Methodology.

For researchers

Dataset

Every question, graded answer and verdict, with viewer-ready tables.

Hugging Face →

Raw data

The JSON bundle behind this page: leaderboard, corpus and all 29 runs, CC BY 4.0.

Downloads and versions →

Cite

@misc{spacebench2026,
  title  = {SpaceBench: Can you trust AI for space work?},
  author = {Anthony Vincent Caracappa},
  year   = {2026},
  note   = {Corpus 307-bf8e3bdb8cf5, revision 2026-11-08},
  url    = {https://spacebench.net/}
}

Results by job

Results by AI

Limits, harness differences and corrections: methodology. Earlier API leaderboards: round 2 · round 1.