Methodology
By Anthony Vincent Caracappa, Founder and Self-Directed Space Researcher, Viventine Space Systems · updated
SpaceBench asks the same space-domain questions two ways. Closed-book: the model answers from its own knowledge and is told that "UNKNOWN" is a valid answer. MCP-grounded: the same questions with live tools over the Orbit Sentinel regulatory & satellite catalog database. The gap between tracks is the headline: what does real data access actually buy?
The 307-question corpus is organized around seven practitioner jobs (catalog lookup, constellation status, debris, regulatory, company and market intelligence, ground infrastructure, baseline knowledge), 25 to 55 questions each, plus negative controls. Each question is labeled public-verifiable (checkable against Space-Track, FCC, ITU and other public records) or Orbit Sentinel-derived (the answer exists only in Orbit Sentinel's analysis).
Verdicts
For each AI, job and track: Trust it when it is wrong on 5% or fewer of the questions and right on 80% or more; Don't use it when it is wrong on more than 20%; otherwise Verify its answers. Confident wrong answers drive the verdict; declining to answer never does. Headline verdicts use public-verifiable questions only; Orbit Sentinel-derived questions are shown separately. Rates pool every run of a question, but the 95% ranges count each question once, because repeat runs of the same question are not independent.
Answers and grading
Models end replies with a FINAL: line. Each response grades to one of six outcomes:
correct, incorrect, abstained (UNKNOWN and honest
variants like "classified" or "not publicly known"), no answer (empty or
protocol-violating output — a format failure, never counted as a knowledge claim; since grader
2026-10-10 a reply that finished cleanly as a single short line with no FINAL: is read as its
answer),
truncated (cut off at the completion cap, excluded from gradeable like provider
errors — the model never got to state an answer), or
error (transport failure, excluded entirely). Matching is deterministic: numeric
tolerance bands, word-boundary text matching that refuses credit inside negated clauses, no LLM judges.
Score/100q = (correct − 2×incorrect) per 100 gradeable questions — gradeable being the total minus transport errors and truncated completions. Abstention is free. The −2 penalty is not arbitrary: under proper scoring, a penalty p makes answering rational only when the model's confidence exceeds p/(p+1) — so −2 encodes an explicit 67% confidence threshold, our stated risk posture for a domain where a confident wrong answer costs more than a missed one. Normalizing per 100 questions keeps scores comparable as the corpus grows; comparisons across corpus versions still carry composition caveats. Hallucination rate is incorrect ÷ (correct + incorrect) on grounded questions.
Penalty sensitivity
Rankings are checked at penalties 1, 2, 3 and 5 (confidence thresholds 50%, 67%, 75%, 83%). Tier structure is robust; adjacent positions can move — where the ordering depends on the penalty, that dependence is disclosed here rather than discovered later.
Closed-book rank by penalty (1 / 2 / 3 / 5)
| Model | Ranks |
|---|---|
| Claude Fable 5.1 | 1 / 1 / 2 / 3 · moves |
| Claude Opus 5.5 | 2 / 2 / 3 / 2 · moves |
| Grok 4.7 | 4 / 3 / 1 / 1 · moves |
| Claude Sonnet 5.5 | 3 / 4 / 4 / 4 · moves |
| Claude Haiku 5.5 | 5 / 5 / 5 / 5 |
| Gemma 4 12B | 6 / 6 / 6 / 6 |
| DeepSeek-R1 14B | 7 / 7 / 7 / 7 |
| gpt-oss 20B | 8 / 8 / 8 / 8 |
MCP-grounded rank by penalty (1 / 2 / 3 / 5)
| Model | Ranks |
|---|---|
| Claude Opus 5.5 | 1 / 1 / 1 / 1 |
| Claude Fable 5.1 | 2 / 2 / 2 / 2 |
| Claude Sonnet 5.5 | 3 / 3 / 3 / 4 · moves |
| Grok 4.7 | 4 / 4 / 4 / 3 · moves |
| Claude Haiku 5.5 | 5 / 5 / 5 / 5 |
| Gemma 4 12B | 6 / 6 / 6 / 6 |
| gpt-oss 20B | 7 / 7 / 7 / 7 |
Negative controls
Five questions are deliberately unanswerable (classified budgets, future counts, uncataloged objects). The correct behavior is abstention; any confident number is a fabrication. These make the calibration score measurable rather than aspirational.
Contamination resistance
Grounded answers are frozen from the database at a stamped date, and every question ships with the SQL that re-derives it. The corpus re-freezes as reality changes — new launches, new filings, new reentries — so a model that memorized last quarter's leaderboard answers fails this quarter's freeze. Each corpus version is content-fingerprinted; results are stamped with the fingerprint they ran under and never mixed across versions. Published leaderboards are immutable — every version keeps a permanent archive page.
How each AI was run
- Claude (Haiku, Sonnet and Opus 5.5; Fable 5.1): Claude Code on a Claude Pro subscription, using Claude Code's own tool loop. Costs shown are API-equivalent; nothing was billed per token.
- Grok 4.7: the xAI API through the SpaceBench tool loop.
- Open-weight models (gpt-oss 20B, Gemma 4 12B, DeepSeek-R1 14B): Ollama 0.40.15 on rented RTX 4090s (24 GB), 65,536-token context, flash attention, 8-bit KV cache, through the SpaceBench tool loop.
- Runs: one closed-book run per AI; three tool runs for every AI with a tool track. DeepSeek-R1 14B never called the tools when offered them in earlier versions, so it is reported closed-book only.
Run conditions
- Provider-default sampling; no temperature override.
- Up to 8 tool turns per question. The SpaceBench loop then forces a final answer with tools withheld; Claude Code has no forced final turn, so a Claude reply that runs out of turns scores "no answer".
- At most 50 tool calls per question, a production-load safeguard; every hit is recorded.
- An 8,192-token completion cap. Ollama counts thinking against it (14% of Gemma 4's closed-book answers and about 2.5% of its tool answers were cut off and excluded as truncated); Claude never reached it; xAI does not apply it to Grok's reasoning.
- Up to 15 minutes per API call.
- Answers that ended in a provider error were rerun, including 25 gpt-oss 20B tool answers (the link to the rented GPU had dropped, or Ollama could not parse a tool call) and four Fable answers; no published answer is an error.
- Costs from list prices with cached input at cached rates; Grok costs are xAI's own billed figures.
Grader version:
2026-10-31b.
Corrections
After the runs, every question where 40% or more of the strong models' tool answers (Claude and Grok) were not correct was reviewed: 12 of 306. Corrections to the answer key are published as errata; the corpus fingerprint stays that of the questions as asked.
reg-itu-biu-top-admin-2024: key unchanged, more answer forms accepted (2026-11-08). The key (CHN, 24 networks brought into use in 2024; France next with 15) stands. The accept list held only the ITU symbol, although the question's freeze notes call for the country name as well, so answers naming China without the symbol were graded incorrect. The database still gives CHN 24, F 15 on 2026-11-08.reg-largest-active-ngso-bond: removed from scoring (2026-11-08). Orbit Sentinel's bond portfolio changed during the run. Answers through 2026-11-05 saw Space Norway AS (the key, $5.67M); later answers and the database on 2026-11-08 give O3b Limited ($5.02M). Both were right when read, so no single key holds for the run. Not scored; to be restated at the next freeze.reg-active-ngso-surety-bonds-usd: removed from scoring (2026-11-08). Same change as reg-largest-active-ngso-bond: the active NGSO bond total was $27.03M (the key) for answers through 2026-11-05 and $21.36M for later answers and in the database on 2026-11-08, outside the 10% tolerance. Not scored; to be restated at the next freeze.
The other nine reviewed questions stand as keyed: gnd-ofcom-top-licensee is right by licence count
(the tool's per-operator figure counts stations), control-nro-active-count is a negative
control, con-hawkeye360-on-orbit still matches the database, and six were flagged mostly by
abstentions. 304 questions are scored on the tool track and 305 closed-book. Earlier corrections are folded into the
questions or retired with them, and each earlier revision keeps its own errata.
This corpus is the previous one less one removed question, with every answer key refrozen on 2026-11-05 (nine moved). Every AI answered every question again on both tracks, on Orbit Sentinel's tools at version 0.10.0, Claude Code 2.1.318 and Ollama 0.40.15; nothing was carried forward. One question, on treaty ratifications, is asked closed-book only because no tool reads that table, and the closed-to-tool gap is computed over questions both tracks answered.
Before the previous version, Orbit Sentinel fixed most of the causes found in a review of failed tool answers, such as tools dropping fields and filters the API already had, so tool-track gains over v2.2 and earlier partly reflect those fixes, not the AIs alone. During this run Orbit Sentinel's bond data changed; the two questions it affected are removed from scoring.
Relationship disclosure
SpaceBench is built by the team behind Orbit Sentinel, and the MCP track's ground truth is the Orbit Sentinel database — the benchmark measures retrieval against a defined oracle, and that oracle is our product. We publish the grounded questions, their SQL, and the freeze history precisely so this relationship is auditable rather than taken on trust.