xlab EXPERIMENT INFRASTRUCTURE
THE QUESTION COMES FIRSTLOCAL LAB / OPEN SOURCE

Make every
run count

THE METHOD

A question deserves
more than a pass count

THE LAB, IN MOTION

From question
to evidence

Real terminal output, kept with its source record.
Scrub through the run. Read what actually happened.

The walkthrough uses the simulated adapter and is not a benchmark. The compression recording is a harness check.

xlab / captured sessionRECORDED
Loading captured output…
Loading the original recording…
—
Loading recording provenance…
Original output ↓Source record ↓Report ↓
AN ANSWER YOU CAN AUDIT

The record
stays with you

Inside a run directory

The spec you wrote. The seed that picked the order. The system fingerprint. Every valid sample and every void, with its reason.

Plain files, a local index, and an offline report. Follow a verdict all the way back to the measurements that earned it.

spec.tomlplan.jsonsamples.jsonl
GET STARTED WITH YOUR AGENT

Tell my
agent to…

You bring the question.
Your agent builds the experiment.
You decide what the evidence means.

Read the full getting started guide ↗

You can drive xlab from a conversation with your coding agent. It reads the README, writes the specs, checks the plan, and works the queue. Start by telling it what you want to learn.

New to the lab? Try the simulated walkthrough first ↗

A worked example: a server collects sensor readings all day. Its nightly jobs compress the completed day and aggregate it into a database. Follow the prompts below to compare alternatives while the collector keeps running.

Adapt these example paths and settings to your machine. Review the plan and declared spend before telling your agent to run it.

Set up the lab
TELL MY AGENT TO
Clone https://github.com/s4njee/xlab onto the machine that will take the measurements. Read its README and adapter docstrings, then install it with pip install -e . in a virtual environment using Python 3.11 or newer. Check that this interpreter can import xlab.cli. Make a lab with xlab init ~/labs/telemetry --example compression. From that lab, run xlab doctor and xlab selftest and tell me what passed and what needs fixing. The host collects sensor readings all day, so set max_load_per_core = 0.5 in lab.toml. Check which codecs the example specs require and which are installed. Show me the lab path and configuration. Do not enqueue experiments yet.

What you should getA usable Python environment, the selftest result, a lab directory, and an explicit list of missing tools. A busy host should wait at its load gate.

Read this step in the guide ↗
Freeze and pin the inputs
TELL MY AGENT TO
Sensor readings arrive in /srv/telemetry/YYYY-MM-DD/HH.jsonl, about 2 GB per day. The archive job currently compresses a completed day with gzip -6 into /srv/archive/. Use /srv/telemetry/2026-10-17 as the example completed day; ask me for the real completed date and paths before using this on my host. Never read the live collection directory during a timed sample. Freeze a copy outside it, pack that copy once as a deterministic corpus, and record its SHA-256. Call the corpus telemetry-day and pin the actual digest in the experiment specs so every arm sees the same bytes. Show me the frozen source, corpus definition, size, and digest before planning the comparison.

What you should getA frozen input and a telemetry-day corpus definition pinned to the digest of the packed corpus, with preparation outside the timed path.

Read this step in the guide ↗
Write the question and the tree
TELL MY AGENT TO
Set up a decision tree in ~/labs/telemetry for these questions, in this order. Write the specs and branch rules before measuring anything. Screen zstd levels 1 through 9, lz4, brotli 4, and xz 1 against gzip -6 as the control on the pinned telemetry-day corpus. Use six screening blocks, compress_s as the primary metric, and a 5% smallest effect that matters. A candidate qualifies only if it is faster and loses no more than 3% of the control's compression ratio. Confirm exactly the qualifying settings with twelve fresh blocks in a confirmatory experiment. If a setting is faster only by giving up too much ratio, hold for my decision. If nothing qualifies, stop. Only after a zstd setting is confirmed should the thread follow-up compare 2, 4, and 8 threads. Validate the tree and every concrete spec with xlab check and preflight. Show me the randomized plans, declared spend, budget settings, and the tree's branch order. Explain what remains unknown about the cost of later nodes. Do not start the tree or enqueue a run until I have reviewed the plan.

What you should getValidated specs and a decision tree: screen, confirm with fresh blocks, hold on a trade-off, and run the thread follow-up only after its declared condition.

Read this step in the guide ↗
Keep the lab running
TELL MY AGENT TO
I have reviewed the specs, tree, plans, and declared spend. Start the approved tree in ~/labs/telemetry. Add the fixed sentinel spec as a standing run every six hours so we can see the host's noise floor and drift. Use xlab service to print the launchd or systemd user unit for this machine. Show me the unit and its interpreter and lab paths, then install it so xlab up survives a reboot. Verify that the service is running, and tell me the dashboard address and where the records and reports are stored. When I ask for status, tell me which node is active, which branch the tree took and why, and whether any run is waiting for load, budget, or a decision. If you can monitor this host, tell me when the first tree decision occurs; otherwise explain how I can check it or ask you to read it later.

What you should getOne running lab service, a standing sentinel, a reachable local dashboard, and clear commands for checking the queue and tree.

Read this step in the guide ↗
Add a one-off comparison
TELL MY AGENT TO
The aggregation job is /srv/jobs/aggregate.py --day DAY --out DB. Set up a command-adapter experiment in ~/labs/telemetry: does --parallel 4 make the completed day's aggregation faster than the default? Use the default as the control, eight blocks, wall_s as primary, and a 5% smallest effect that matters. Both arms must use the same frozen input. Give every sample a fresh output database. In the untimed after command, check its row count against an independently established expected result; a failed command or failed verification must be void, never a timing. Keep setup, input preparation, and verification out of the timed command. Validate it, run preflight, and show me the plan, declared spend, and how it fits the lab's current budgets. Wait for me to review those. When I tell you to enqueue it, use --front so it runs before other waiting work.

What you should getA separate command-adapter spec, independent output verification, and a reviewed plan that can be queued ahead of waiting runs.

Read this step in the guide ↗
Read the result and its uncertainty
TELL MY AGENT TO
Analyse the compression screening experiment in ~/labs/telemetry. For each candidate and cell, tell me its ratio to gzip -6, the interval, the number of complete paired blocks, and the verdict. Explain what the metric's direction and the declared smallest effect mean before naming a winner. Tell me which candidates kept the required compression ratio, whether the decision tree moved, and the exact branch condition that caused the move. Show the run ids and distinguish screening leads from confirmation results. Summarize void reasons and any sentinel drift that affects interpretation. For inconclusive results, report the analysis's estimated blocks needed. Use observed sample durations to estimate the extra hours, stating the assumptions and any unknown spend. Do not change the rubric after seeing the results. Write a local report and show me where to find its source records.

What you should getAn evidence-backed explanation with intervals, complete-block counts, run ids, tree decisions, and a report; an inconclusive result remains inconclusive.

Read this step in the guide ↗

A named control. A meaningful effect. Frozen inputs. Independent verification. A verdict with an interval.

Using an agent to operate the lab works today. The adapter for measuring AI agent configurations is still being built.