agentleFS
Fadi88Fadi88/one-man-startup/.claude/skills/experiment/SKILL.md

experiment

Skill16 starsChanged 16 days agoMITView on GitHub

Run every bet, growth idea or workflow change as a pre-registered experiment - hypothesis, the number that would prove it, a read date, and what result would make it wrong - written before it starts. Use for pricing, copy, onboarding, channels, cost cuts and any \"let's try\".

29 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use cost, eval and 9 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • maxvaegaSkill

    cost

    Estimate what Claude Cowork / Claude Code conversations would cost if run on the pay-per-token Claude API instead of the subscription. Reports token counts and cache-aware API cost in USD and EUR, for the current conversation or for the whole history (cost per conversation). Trigger when the user asks things like "how much would this conversation cost on the API", "API cost of my Cowork conversations", "token usage / token count", "cost per conversation", "/cost", or wants to compare subscription vs API pricing.

    Also in 3 projects that use this

  • qtzx06Skill

    eval

    Evaluate trained YOLO model accuracy with mAP metrics and per-class error analysis. Identifies failure cases for targeted improvement. Use after training.

    Also in 4 projects that use this

  • dirienSkill

    decide

    Three-agent adversarial debate protocol for strategic decisions. Two advocates steelman opposing positions while a moderator identifies risks and synthesizes a binding consensus. Use when choosing between alternatives, evaluating tradeoffs, or making high-stakes decisions.

    Also in 3 projects that use this

  • javatarzSkill

    wiki

    Access and manage the local GitHub Wiki clone. Use this skill when you need to read wiki content (architecture, design docs, guides), update wiki documentation, check wiki status, or push wiki changes. The wiki contains high-level system design and architectural documentation.

    Also in 3 projects that use this

  • HeshamFSSkill

    loop

    Run a command or check on a recurring interval

    Also in 3 projects that use this

  • VercelSkill

    workflow

    Creates durable, resumable workflows using Vercel's Workflow SDK. Use when building workflows that need to survive restarts, pause for external events, retry on failure, or coordinate multi-step operations over time. Triggers on mentions of "workflow", "durable functions", "resumable", "workflow sdk", "queue", "event", "push", "subscribe", or step-based orchestration.

    Also in 3 projects that use this

The file

We split this skill into the rules it gives an agent

3 rules. 0 are common, 1 is distinctive, and 2 are very specific to this project.

  • 0

    Common

    Most of the 27 edited versions we read keep these.

  • 1

    Distinctive

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Write the entry in docs/EXPERIMENTS.md before the change ships.

    line 10
  2. Distinctive:

    When the change is something users see (copy, onboarding, a price page, a feature) and enough of them arrive, test it on a random sample …

    line 38
  3. Very specific:

    No peeking: Read once, at the planned size or the read date.

    line 52

“Common” is measured: kept by at least 6 in 10 of the 27 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md
<!-- Generated by tools/sync_adapters.py from skills/experiment/SKILL.md. Edit the source, not this file. -->

Experiment

Write the entry in docs/EXPERIMENTS.md before the change ships. An experiment written after the result is a story.

The entry

### 7. Annual plan at two months free, from 2026-03-01
- **Why:** monthly churn is 9 percent; annual buyers churn less.
- **Change:** one change only: the annual option on the pricing page.
- **Hypothesis:** some monthly buyers choose annual.
- **Prediction:** at least 15 percent of new paid sign-ups pick annual within 30 days.
- **Wrong if:** under 5 percent pick annual. Then the annual option comes off and we look at
  churn reasons instead.
- **Confounds:** the launch post on 2026-03-03 brings unusual traffic.
- **Read:** 2026-03-31.

And one line in the "At a glance" table at the top of the file.

Rules

  • One change at a time, or the result cannot be attributed.
  • The read date is kept. Reading early because it looks good is how false wins happen.
  • Record what the numbers say, not what was hoped. "Not distinguishable from noise" is a result.
  • A closed experiment moves to docs/EXPERIMENTS_CLOSED.md with its numbers and verdict.

A/B tests: when users can be split

When the change is something users see (copy, onboarding, a price page, a feature) and enough of them arrive, test it on a random sample instead of before and after: half see the current version (A), half the new one (B), at the same time, so a news day or a launch post hits both.

  1. Size first. Pick one primary metric, today's rate and the smallest change worth acting on, then run python tools/ab_size.py size --base <rate> --target <rate>. If that many users will not arrive before the read date, do not run an A/B test: run it before and after, and say the result is weaker.
  2. Assign at random, once. A user's arm comes from a stable hash of their user id and the experiment's name, is logged when assigned, and never changes. Both arms ship behind the flag (TPL-73), so B is switched off in one step.
  3. Guardrails. Name the numbers that must not get worse (errors, refunds, unsubscribes, AI spend per user) and the drop that stops the test early. A guardrail is the only reason to stop before the planned size.
  4. No peeking. Read once, at the planned size or the read date. First run python tools/ab_size.py srm --a <users> --b <users>: a mismatch means assignment is broken and the result is void. Then read for the difference and its p-value.
  5. Ask first. Showing users different prices or terms goes to legal and the founder before it starts; the dashboard shows each arm's count and metrics from one place (skills/single-source/).

An A/B entry adds four lines to the template above, and tools/checks/experiments_check.py refuses one without them:

- **Arms:** A, the current welcome screen; B, the three-step welcome screen. 50/50.
- **Assignment:** hash of user id and "welcome-v2", logged at first open.
- **Sample:** 3,839 per arm (`tools/ab_size.py size --base 0.10 --target 0.12`).
- **Guardrails:** crash rate; stop if B's is double A's.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.