experiment-brief
Skill4 starsChanged 58 days agoApache-2.0View on GitHub
Pre-register a run before it starts — what is measured, why, which models and provider, what it costs, what would falsify it, and when to abort. Use before any benchmark, α measurement, training run or arm in this repository; the briefing goes before the command, never beside the report.
1 other project uses a version of this.
Adopted by
Most recently taken up by
- ai-analyst-lab/ai-analysttheir own version, first seen Sep 10, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 1
- ai-analyst-lab/ai-analyst★ 304same name
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 12
Create <run-dir>/BRIEF.md before the first token is generated:
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Brief the experiment before running it
A briefing written next to the report is not a briefing. It is a justification.
Write this file first
Create <run-dir>/BRIEF.md before the first token is generated:
# <step id> — <one line>
**Question.** The single thing this run answers.
**Falsification.** The result that would kill the step. Written now, while the
answer is unknown.
**Arms.** Each one, and why it is being bought *now* rather than after.
**Suite.** Split, n, and why this subset carries the signal.
**Models.** Exact tags and the provider actually serving them.
**Parameters.** Temperature, `k`, max tokens, seed.
**Cost.** Estimated calls and dollars, and the ceiling at which this run stops.
**Abort rule.** The condition under which the run is killed early, decided now.
**Redesign count.** How many times this instrument has been reshaped. Three is
the stopping condition.The four rules the brief exists to enforce
- Buy arms in sequence. The arm that can kill the hypothesis runs first. An attribution arm — the control that explains why — is bought only once there is an effect to attribute. A flat result makes every other arm unnecessary, which is exactly when a grid has already spent the money.
- Name the provider, not just the model. A configured model is not necessarily the model that runs: this organisation has shipped a system where every model but one silently fell back to a different one.
- State the falsification before the run. A condition invented after the numbers arrive is a description, not a test.
- Decide the abort rule while it is still cheap. A run already paid for is not an argument for finishing it; compare what remains against what an abort would force you to repeat, and record which you chose.
When the run finishes
Append the outcome to the same BRIEF.md — including "flat", "aborted at case
14", or "the falsification condition fired". Then update the step's row in
docs/PLAN.md, marked [ran]
with the run directory named.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

