Can you trust Gemma 4 12B for space work?
By Anthony Vincent Caracappa, Founder and Self-Directed Space Researcher, Viventine Space Systems · updated
On its own, it earns "Trust it" on none of the jobs. Connected to Orbit Sentinel, trust it for catalog lookup, baseline knowledge.
| Job | AloneClosed-book | With dataWith Orbit Sentinel tools | n | ||||
|---|---|---|---|---|---|---|---|
| right / wrong | range | declined | right / wrong | range | declined | ||
| Catalog lookup | 9 / 60 | 3–22 | 31 | 99 / 1 | 90–100 | 0 | 45×3 |
| Constellation & fleet status | 9 / 53 | 3–23 | 38 | 61 / 26 | 46–74 | 12 | 43×3 |
| Debris & conjunction context | 13 / 36 | 6–27 | 51 | 61 / 21 | 46–75 | 17 | 42×3 |
| Regulatory & licensing | 14 / 47 | 6–29 | 39 | 68 / 18 | 53–81 | 14 | 38×3 |
| Company & market intelligence | 6 / 44 | 2–18 | 50 | 74 / 14 | 59–85 | 12 | 41×3 |
| Ground infrastructure | 10 / 36 | 4–24 | 54 | 61 / 24 | 45–74 | 15 | 40×3 |
| Baseline knowledge | 85 / 15 | 58–96 | 0 | 85 / 1 | 66–94 | 13 | 24×3 |
| Orbit Sentinel-only questions | 8 / 32 | 2–25 | 60 | 78 / 9 | 59–90 | 12 | 25×3 |
% right / % wrong per cell; color is the verdict (green trust it, amber verify, red don't use). Range is the 95% interval for % right, counting each question once; n is questions × runs for the tools track.
- How it was run
- Open weights, run locally (Ollama on one RTX 4090)
- Tool loop
- SpaceBench tool loop
- Runs
- 1 on its own, 3 with tools; rates pool all runs, ranges count each question once
- Verdict rule
- Trust it: wrong ≤ 5% and right ≥ 80%. Don't use it: wrong > 20%. Otherwise verify its answers.
Limits, harness differences and corrections: methodology.