Can AI reason about space?
83 space-domain questions, asked two ways: closed-book, and grounded via MCP tools against a living satellite & regulatory database. Scoring rewards calibration — a wrong answer costs twice what a right one earns, and admitting ignorance is free.
| Model | Track | Score ▾ | Correct | Incorrect | Abstained | No answer | Errors | Reasoning | Grounded | Hallucination | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|
| claude-opus-4.8 | closed | +62 | 62/83 | 0 | 21 | 0 | 0 | 100% [93–100] | 25% [13–43] | 0/7 | $0.24 |
| claude-sonnet-5 | closed | +59 | 61/83 | 1 | 21 | 0 | 0 | 100% [93–100] | 21% [10–40] | 1/7 | $0.15 |
| gemini-3.5-flash | closed | +59 | 59/83 | 0 | 24 | 0 | 0 | 100% [93–100] | 14% [6–31] | 0/4 | $0.53 |
| claude-fable-5 | closed | +58 | 62/83 | 2 | 19 | 0 | 0 | 100% [93–100] | 25% [13–43] | 2/9 | $1.11 |
| qwen3-max | closed | +58 | 60/83 | 1 | 22 | 0 | 0 | 100% [93–100] | 18% [8–36] | 1/6 | $0.08 |
| gemini-3.1-pro | closed | +57 | 57/83 | 0 | 26 | 0 | 0 | 100% [93–100] | 7% [2–23] | 0/2 | $0.72 |
| grok-4.5 | closed | +55 | 55/83 | 0 | 25 | 3 | 0 | 100% [93–100] | 11% [4–27] | 0/3 | $0.07 |
| deepseek-v4-pro | closed | +52 | 58/83 | 3 | 19 | 3 | 0 | 96% [87–99] | 23% [11–42] | 1/7 | $0.05 |
| kimi-k2.6 | closed | +52 | 52/83 | 0 | 25 | 6 | 0 | 100% [93–100] | 7% [2–23] | 0/2 | $0.56 |
| llama-4-maverick | closed | +44 | 54/83 | 5 | 24 | 0 | 0 | 95% [85–98] | 7% [2–23] | 2/4 | $0.01 |
| claude-haiku-4.5 | closed | +33 | 51/83 | 9 | 23 | 0 | 0 | 89% [78–95] | 7% [2–23] | 3/5 | $0.11 |
| gpt-5.4 | closed | +32 | 62/83 | 15 | 6 | 0 | 0 | 100% [93–100] | 25% [13–43] | 68% [47–84] | $0.08 |
| gpt-5.4-mini | closed | +28 | 58/83 | 15 | 10 | 0 | 0 | 96% [88–99] | 18% [8–36] | 72% [49–88] | $0.02 |
| grok-4.5 | mcp | +59 | 61/83 | 1 | 21 | 0 | 0 | 84% [72–91] | 54% [36–70] | 6% [1–28] | $5.31 |
| gemini-3.5-flash | mcp | +58 | 64/83 | 3 | 14 | 2 | 0 | 100% [93–100] | 35% [19–54] | 25% [9–53] | $3.99 |
| claude-sonnet-5 | mcp | +47 | 65/83 | 9 | 9 | 0 | 0 | 98% [90–100] | 39% [24–58] | 42% [23–64] | $3.53 |
| gpt-5.4 | mcp | +46 | 52/83 | 3 | 28 | 0 | 0 | 84% [72–91] | 21% [10–40] | 2/8 | $1.35 |
| qwen3-max | mcp | +36 | 54/83 | 9 | 20 | 0 | 0 | 85% [74–92] | 25% [13–43] | 56% [33–77] | $1.59 |
| claude-haiku-4.5 | mcp | +32 | 44/83 | 6 | 33 | 0 | 0 | 65% [52–77] | 29% [15–47] | 38% [18–64] | $1.22 |
| gpt-5.4-mini | mcp | +22 | 38/83 | 8 | 36 | 1 | 0 | 61% [48–73] | 18% [8–36] | 50% [24–76] | $0.36 |
Score = correct +1, incorrect −2, abstained/no-answer 0. Hallucination = incorrect ÷ (correct + incorrect) on grounded questions. Cells with n<10 show raw fractions; percentages carry Wilson 95% intervals.
closed-book MCP-grounded
closed-book MCP-grounded