orbit-agent: can the ground trust the numbers?

The day's class shares

truth self-checking audit + its 90% error bar triage (agent v4) labels only

Show as a table

Error bars that held today

A 90% error bar should contain the truth for about 9 classes in 10.

Self-check, pass by pass

The ground compares the mistakes in the checked photos with the mistakes the model expected. Much bigger (over x) means the model is broken in a way it can't see: switch to safe mode.

Some photos the audit checked

Picked at random (unsure photos a bit more often). Red frame = the model's label was wrong.

How it works

Label every photo onboard, with the model's chance for each class (about 60 bytes).
Check a random sample: send some full photos, picked at random with known chances. Every photo keeps a minimum chance, so confident mistakes can be caught too.
Correct on the ground: the labels' average, plus the average mistake seen in the checked photos. The error bar comes from the checks, not from the model's opinion of itself.
Check the checker: if the checked mistakes are clearly bigger than the model expected, the next passes run in safe mode: checks picked fully at random, and the model's chances recalibrated from past checks.