xlab-analysis(7)

NAME

xlab-analysis — an interval and a decision, from complete paired blocks.

SYNOPSIS

xlab analyze S1-demo
xlab analyze S1-demo --json
xlab report S1-demo

DESCRIPTION

The observation is an arm/control log-ratio within a complete block. Absolute times from different machines or days are not pooled as if interchangeable. The estimate is the exponentiated mean log-ratio, with a paired t interval and a Hodges–Lehmann/Wilcoxon robust check. Both intervals must agree on a verdict.

Verdict	Meaning
better	Both intervals are beyond the meaningful-effect boundary in the good direction
worse	Both intervals are beyond the boundary in the bad direction
equivalent	Both intervals fit inside the declared equivalence region
inconclusive	The available evidence does not resolve the declared question
insufficient	Too few complete blocks to give a verdict

The equivalence region is [1/(1+delta), 1+delta]. The direction comes from the adapter's metric definition. A lower time ratio is good; a higher throughput ratio is good. A statistically detectable change can still be too small to matter.

Screening results are leads. Confirmatory intervals account for the primary comparison family; repeated tree looks widen intervals as well. Pareto fronts show trade-offs rather than declaring a single universal winner.

EXAMPLES

Inspect the control CV and estimated blocks needed before buying another night of data. An inconclusive result is an answer about uncertainty, not permission to change the rubric after measuring. Current positive log-ratio methods do not support agent pass rates; that extension is AG-07.

SEE ALSO

Calibration · Trees · Record