← All verdicts

Can you trust Gemma 4 12B for space work?

By , Founder and Self-Directed Space Researcher, Viventine Space Systems · updated

On its own, it earns "Trust it" on none of the jobs. Connected to Orbit Sentinel, trust it for catalog lookup, baseline knowledge.

Job Alone With data
right / wrong declined right / wrong declined
Catalog lookup
9 / 60
31
99 / 1
0
Constellation & fleet status
9 / 53
38
61 / 26
12
Debris & conjunction context
13 / 36
51
61 / 21
17
Regulatory & licensing
14 / 47
39
68 / 18
14
Company & market intelligence
6 / 44
50
74 / 14
12
Ground infrastructure
10 / 36
54
61 / 24
15
Baseline knowledge
85 / 15
0
85 / 1
13
Orbit Sentinel-only questions
8 / 32
60
78 / 9
12

% right / % wrong per cell; color is the verdict (green trust it, amber verify, red don't use). Range is the 95% interval for % right, counting each question once; n is questions × runs for the tools track.

How it was run
Open weights, run locally (Ollama on one RTX 4090)
Tool loop
SpaceBench tool loop
Runs
1 on its own, 3 with tools; rates pool all runs, ranges count each question once
Verdict rule
Trust it: wrong ≤ 5% and right ≥ 80%. Don't use it: wrong > 20%. Otherwise verify its answers.

Limits, harness differences and corrections: methodology.