# S1-demo

**Question:** Do the two candidate settings beat the baseline?  
**Hypothesis:** faster takes about 12% less time than base; same is indistinguishable  
Screening experiment on adapter `simulated`. 16 complete blocks from 1 run(s), 64 samples planned. Primary metric `time_s`, control **base**, smallest effect that matters 3%, verdicts need 6 blocks, intervals at 95.00%.

**Samples:** 67 valid, 2 void.

**Void samples, by reason:**

- 2× simulated failure

## Runs

| Run | State | Seed | Started | Ended | Valid / planned | Void | Tree |
|---|---|---|---|---|---|---|---|
| `20261018T153643Z-S1-demo-d51a` | complete | 1 | 2026-10-18T15:36 | 2026-10-18T15:36 | 67 / 64 | 2 |  |

## Comparisons with the control

Each row is one arm against the control in one cell: the geometric-mean ratio over complete blocks with its paired t interval, the Hodges-Lehmann estimate with its distribution-free interval, the sign-flip p-value and the blocks a verdict at delta would need. Only the primary metric is judged; the others are descriptive.

### `time_s` (primary)

Time, s, lower is better.

| Cell | Arm | Ratio | Interval | HL | HL interval | Blocks | p | Blocks needed | Verdict |
|---|---|---|---|---|---|---|---|---|---|
| scale=1 | faster | 0.8966 | [0.8443, 0.9522] | 0.8965 | [0.8393, 0.9579] | 8 | 0.016 | 47 | better |
| scale=1 | same | 0.985 | [0.9621, 1.009] | 0.9833 | [0.962, 1.01] | 8 | 0.18 | 8 | inconclusive |
| scale=1 | slow | 1.089 | [1.055, 1.124] | 1.091 | [1.048, 1.124] | 8 | 0.0078 | 13 | worse |
| scale=4 | faster | 0.8583 | [0.8042, 0.916] | 0.8527 | [0.8033, 0.9269] | 8 | 0.016 | 55 | better |
| scale=4 | same | 0.9668 | [0.9252, 1.01] | 0.9722 | [0.9259, 1.027] | 8 | 0.12 | 25 | inconclusive |
| scale=4 | slow | 1.055 | [1.017, 1.094] | 1.045 | [1.018, 1.092] | 8 | 0.0078 | 18 | inconclusive |

### `throughput` (descriptive)

Throughput, ops/s, higher is better.

| Cell | Arm | Ratio | Interval | HL | HL interval | Blocks | p | Blocks needed | Verdict |
|---|---|---|---|---|---|---|---|---|---|
| scale=1 | faster | 1.115 | [1.05, 1.184] | 1.115 | [1.044, 1.191] | 8 | 0.016 | 47 | (better) |
| scale=1 | same | 1.015 | [0.9915, 1.039] | 1.017 | [0.9897, 1.04] | 8 | 0.18 | 8 | (inconclusive) |
| scale=1 | slow | 0.9183 | [0.8899, 0.9476] | 0.9165 | [0.8897, 0.9543] | 8 | 0.0078 | 13 | (worse) |
| scale=4 | faster | 1.165 | [1.092, 1.244] | 1.173 | [1.079, 1.245] | 8 | 0.016 | 55 | (better) |
| scale=4 | same | 1.034 | [0.9898, 1.081] | 1.029 | [0.9733, 1.08] | 8 | 0.12 | 25 | (inconclusive) |
| scale=4 | slow | 0.9482 | [0.914, 0.9836] | 0.9566 | [0.916, 0.9822] | 8 | 0.0078 | 18 | (inconclusive) |

### `quality` (descriptive)

Quality, score, higher is better.

| Cell | Arm | Ratio | Interval | HL | HL interval | Blocks | p | Blocks needed | Verdict |
|---|---|---|---|---|---|---|---|---|---|
| scale=1 | faster | 0.8379 | [0.8102, 0.8666] | 0.8374 | [0.8012, 0.8679] | 8 | 0.0078 | 15 | (worse) |
| scale=1 | same | 1.093 | [1.069, 1.118] | 1.093 | [1.069, 1.123] | 8 | 0.0078 | 7 | (better) |
| scale=1 | slow | 1.156 | [1.124, 1.19] | 1.155 | [1.117, 1.195] | 8 | 0.0078 | 11 | (better) |
| scale=4 | faster | 0.8383 | [0.8287, 0.8481] | 0.8374 | [0.8276, 0.8491] | 8 | 0.0078 | 2 | (worse) |
| scale=4 | same | 1.109 | [1.085, 1.133] | 1.11 | [1.082, 1.135] | 8 | 0.0078 | 6 | (better) |
| scale=4 | slow | 1.152 | [1.126, 1.178] | 1.153 | [1.119, 1.183] | 8 | 0.0078 | 7 | (better) |

## Absolute values

Geometric means with their intervals over every valid sample (arithmetic means where a metric can be zero or negative).

### `time_s` (s)

| Cell | Arm | Mean | Interval | n | CV | Min | Max |
|---|---|---|---|---|---|---|---|
| scale=1 | base | 1.081 | [1.053, 1.11] | 9 | 0.034 | 1.008 | 1.134 |
| scale=1 | faster | 0.9687 | [0.9103, 1.031] | 8 | 0.071 | 0.8269 | 1.037 |
| scale=1 | same | 1.077 | [1.035, 1.121] | 10 | 0.054 | 0.9618 | 1.134 |
| scale=1 | slow | 1.177 | [1.125, 1.23] | 8 | 0.052 | 1.085 | 1.248 |
| scale=4 | base | 4.397 | [4.146, 4.664] | 8 | 0.068 | 3.805 | 4.775 |
| scale=4 | faster | 3.774 | [3.6, 3.956] | 8 | 0.056 | 3.471 | 4.051 |
| scale=4 | same | 4.251 | [4.044, 4.47] | 8 | 0.06 | 3.921 | 4.668 |
| scale=4 | slow | 4.638 | [4.467, 4.815] | 8 | 0.045 | 4.326 | 5.006 |

### `throughput` (ops/s)

| Cell | Arm | Mean | Interval | n | CV | Min | Max |
|---|---|---|---|---|---|---|---|
| scale=1 | base | 0.9252 | [0.901, 0.95] | 9 | 0.035 | 0.8821 | 0.9925 |
| scale=1 | faster | 1.032 | [0.97, 1.099] | 8 | 0.078 | 0.9647 | 1.209 |
| scale=1 | same | 0.9283 | [0.8923, 0.9658] | 10 | 0.057 | 0.8821 | 1.04 |
| scale=1 | slow | 0.8499 | [0.8128, 0.8886] | 8 | 0.054 | 0.8013 | 0.922 |
| scale=4 | base | 0.2274 | [0.2144, 0.2412] | 8 | 0.073 | 0.2094 | 0.2628 |
| scale=4 | faster | 0.265 | [0.2528, 0.2778] | 8 | 0.057 | 0.2469 | 0.2881 |
| scale=4 | same | 0.2352 | [0.2237, 0.2473] | 8 | 0.06 | 0.2142 | 0.255 |
| scale=4 | slow | 0.2156 | [0.2077, 0.2239] | 8 | 0.045 | 0.1998 | 0.2311 |

### `quality` (score)

| Cell | Arm | Mean | Interval | n | CV | Min | Max |
|---|---|---|---|---|---|---|---|
| scale=1 | base | 49.93 | [49.07, 50.8] | 9 | 0.023 | 48.19 | 51.49 |
| scale=1 | faster | 41.86 | [41.03, 42.71] | 8 | 0.024 | 40.96 | 43.51 |
| scale=1 | same | 54.82 | [54.19, 55.46] | 10 | 0.016 | 53.72 | 56.94 |
| scale=1 | slow | 57.76 | [56.49, 59.06] | 8 | 0.027 | 55.27 | 60.24 |
| scale=4 | base | 49.75 | [49.02, 50.48] | 8 | 0.017 | 48.16 | 50.46 |
| scale=4 | faster | 41.7 | [41.48, 41.92] | 8 | 0.0064 | 41.32 | 42.19 |
| scale=4 | same | 55.17 | [54.37, 55.98] | 8 | 0.017 | 53.39 | 56.17 |
| scale=4 | slow | 57.29 | [56.3, 58.3] | 8 | 0.021 | 54.8 | 58.58 |

## Pareto front: `throughput` against `quality`, cell scale=1

throughput (ops/s, higher is better) and quality (score, higher is better); an arm is on the front when no other arm is at least as good on both and better on one.

| Arm | throughput | quality | On front | n |
|---|---|---|---|---|
| faster | 1.032 | 41.86 | yes | 8 |
| same | 0.9283 | 54.82 | yes | 10 |
| base | 0.9252 | 49.93 |  | 9 |
| slow | 0.8499 | 57.76 | yes | 8 |

## Pareto front: `throughput` against `quality`, cell scale=4

throughput (ops/s, higher is better) and quality (score, higher is better); an arm is on the front when no other arm is at least as good on both and better on one.

| Arm | throughput | quality | On front | n |
|---|---|---|---|---|
| faster | 0.265 | 41.7 | yes | 8 |
| same | 0.2352 | 55.17 | yes | 8 |
| base | 0.2274 | 49.75 |  | 8 |
| slow | 0.2156 | 57.29 | yes | 8 |

## Noise floor

| Cell | Arm | Metric | CV | n |
|---|---|---|---|---|
| scale=1 | base | time_s | 0.034 | 9 |
| scale=4 | base | time_s | 0.068 | 8 |

## Provenance

Spec SHA-256 `a922ff6eb568adf5d553c29c5a6d943d7e4bc707fdd27ef02bf25f605e7ba01f`. Written 2026-10-18T15:36:47.885Z by xlab 0.1.0 on MacBookPro (Darwin 25.6.0 arm64). Each run directory holds the exact spec, the plan with its seed, the host and tool fingerprints, and every sample record; the JSON beside this file holds the whole analysis.