Can you trust AI for space work?
Not on its own. The top AI, Claude Opus 5.5, got 1 in 4 of the space-data answers it gave wrong from memory. Connected to live data, about 1 in 200.
SpaceBench is an open benchmark for AI models (LLMs) in the space industry. It asks 8 AIs 307 questions a space professional actually asks, about satellites, debris, licensing and ground stations, and grades every answer against a published key.
- AIs tested
- 8
- Questions
- 307
- Graded answers
- 8,824
Full leaderboard → How we test Get the data
By Anthony Vincent Caracappa, Founder and Self-Directed Space Researcher, Viventine Space Systems · updated
Data questions only (counts, dates, IDs, filings), on the questions both runs answered. Declining to answer is not counted as wrong.
- Claude Opus 5.5Anthropicalone 25%with data 0.5%
- Claude Fable 5.1Anthropicalone 25%with data 0.5%
- Claude Sonnet 5.5Anthropicalone 30%with data 1.5%
- Grok 4.7xAIalone 6.1%with data 1.6%
- Claude Haiku 5.5Anthropicalone 41%with data 3.7%
- Gemma 4 12BGoogle · open weightsalone 83%with data 18%
- gpt-oss 20BOpenAI · open weightsalone 91%with data 19%
Each bar is drawn on a 0% to 100% scale. Alone: the AI answers from what it learned in training. With data: the same AI connected to Orbit Sentinel's live satellite and regulatory database through MCP tools.
Scores
Every AI answered every question twice. Scores with the data pool three repeat runs; scores on its own are one run.
| With Orbit Sentinel data | On its own | ||||
|---|---|---|---|---|---|
| Rank | AI | Score | Wrong | Score | Wrong |
| 1–2 | Claude Opus 5.5 Anthropic | +98.7 | 0.5% | +25.2 | 25% |
| 1–2 | Claude Fable 5.1 Anthropic | +98.6 | 0.5% | +25.9 | 25% |
| 3–4 | Claude Sonnet 5.5 Anthropic | +93.3 | 1.5% | +18.4 | 30% |
| 3–4 | Grok 4.7 xAI | +92.8 | 1.6% | +24.9 | 6.1% |
| 5 | Claude Haiku 5.5 Anthropic | +85.3 | 3.7% | +8.5 | 41% |
| 6 | Gemma 4 12B Google | +42.0 | 18% | -68.7 | 83% |
| 7 | gpt-oss 20B OpenAI | +30.5 | 19% | -90.8 | 91% |
| — | DeepSeek-R1 14B DeepSeek | not run | — | -82.8 | 87% |
Score: right answers minus twice the wrong ones, per 100 questions, so +100 is perfect and guessing does not pay. Saying "I don't know" scores 0. Ranks are shared where AIs cannot be told apart across repeat runs.
Which AI can you rely on for which job?
Green, trust it: wrong 5% of the time or less and right 80% or more. Red, don't use it: wrong more than 20% of the time. Amber: check its answers. Each cell shows % right / % wrong; click a job for its questions.
| Model | Catalog | Constellations | Debris | Regulatory | Market | Ground | Baseline | |
|---|---|---|---|---|---|---|---|---|
| Grok 4.7 xAI | alone | 42 / 0 | 16 / 0 | 14 / 2 | 28 / 0 | 22 / 5 | 13 / 5 | 96 / 0 |
| tools | 100 / 0 | 95 / 5 | 94 / 2 | 96 / 1 | 100 / 0 | 93 / 3 | 97 / 0 | |
| Claude Fable 5.1 Anthropic | alone | 64 / 7 | 53 / 14 | 45 / 14 | 38 / 5 | 49 / 15 | 28 / 28 | 100 / 0 |
| tools | 100 / 0 | 99 / 1 | 99 / 0 | 98 / 2 | 100 / 0 | 99 / 1 | 100 / 0 | |
| Claude Opus 5.5 Anthropic | alone | 76 / 9 | 63 / 7 | 40 / 24 | 33 / 5 | 34 / 20 | 23 / 18 | 100 / 0 |
| tools | 100 / 0 | 100 / 0 | 99 / 1 | 97 / 3 | 100 / 0 | 100 / 0 | 100 / 0 | |
| Claude Sonnet 5.5 Anthropic | alone | 60 / 11 | 42 / 7 | 40 / 21 | 31 / 8 | 37 / 24 | 28 / 18 | 100 / 0 |
| tools | 100 / 0 | 99 / 1 | 96 / 2 | 93 / 4 | 99 / 0 | 96 / 4 | 99 / 1 | |
| Claude Haiku 5.5 Anthropic | alone | 27 / 24 | 26 / 16 | 19 / 10 | 23 / 8 | 27 / 10 | 13 / 3 | 100 / 0 |
| tools | 100 / 0 | 87 / 12 | 93 / 0 | 82 / 3 | 94 / 2 | 89 / 6 | 97 / 0 | |
| gpt-oss 20B OpenAI · open | alone | 7 / 67 | 12 / 62 | 5 / 69 | 10 / 56 | 2 / 66 | 5 / 45 | 96 / 4 |
| tools | 92 / 1 | 36 / 28 | 58 / 13 | 48 / 18 | 70 / 11 | 46 / 20 | 85 / 5 | |
| Gemma 4 12B Google · open | alone | 9 / 60 | 9 / 53 | 13 / 36 | 14 / 47 | 6 / 44 | 10 / 36 | 85 / 15 |
| tools | 99 / 1 | 61 / 26 | 61 / 21 | 68 / 18 | 74 / 14 | 61 / 24 | 85 / 1 | |
| DeepSeek-R1 14B DeepSeek · open | alone | 7 / 60 | 19 / 49 | 5 / 67 | 8 / 51 | 5 / 46 | 13 / 43 | 68 / 32 |
| tools | — | — | — | — | — | — | — |
Each cell: % right, then % wrong (the rest declined to answer). Color is the verdict: Trust it · Verify its answers · Don't use it. Hover for the 95% range.
How we test
-
1. Same questions, asked twice
Each AI answers on its own, then again connected to Orbit Sentinel's satellite and regulatory database through MCP tools.
-
2. Every key is checkable
Each data answer is frozen from a published SQL query on a stamped date and checked against public sources. See the questions.
-
3. Wrong costs double
A right answer earns 1, a wrong one costs 2, "I don't know" costs nothing, so a confident guess does not pay. Methodology.
For researchers
Raw data
The JSON bundle behind this page: leaderboard, corpus and all 29 runs, CC BY 4.0.
Downloads and versions →Cite
@misc{spacebench2026,
title = {SpaceBench: Can you trust AI for space work?},
author = {Anthony Vincent Caracappa},
year = {2026},
note = {Corpus 307-bf8e3bdb8cf5, revision 2026-11-08},
url = {https://spacebench.net/}
} Results by job
Limits, harness differences and corrections: methodology. Earlier API leaderboards: round 2 · round 1.