xlab-calibration(7)
NAME
xlab-calibration — test the lab before trusting its verdict.
SYNOPSIS
xlab selftestDESCRIPTION
Start with the synthetic selftest: known effects through the real planner, runner, and analysis, checking coverage, false verdicts, and power.
Then run A/A on the actual adapter and conditions: the same configuration under different labels. It must come out equivalent. Read the noise floor and blocks-needed estimate. If the question needs more blocks than the available budget can buy, report that constraint.
Finally inject a known handicap larger than the meaningful-effect boundary. It should be called worse. A method that cannot recover a known effect has not earned the right to judge an unknown one.
EXAMPLES
The simulated spec and demo tree provide a quick exercise. They cannot calibrate a model provider or an independent judge. Agent A/A, known-effect, Bernoulli selftests, and judge calibration are P2 work still to be completed. There is no published agent noise table yet.