agentleFS
UKGovernmentBEISUKGovernmentBEIS/inspect_evals/.claude/skills/verify-scoring-change/SKILL.md

verify-scoring-change

Skill735 starsChanged 35 days agoMITView on GitHub

Use when reviewing, verifying, or triaging a PR or proposed fix that changes a scorer, a metric, an answer-extraction or verdict-parsing path, or code that consumes a grader model's output. Also use when asked whether a scoring fix is safe to merge or whether reported numbers will move. Do NOT use for reviewing a whole evaluation against every standard (eval-quality-workflow) or for general PR review (ci-maintenance-workflow).

  • Reads credentials
  • Deletes or force-pushes
  • Installs packages

The file

Show its rules and text

We split this skill into the rules it gives an agent

39 rules. About 0 are common, 22 are distinctive, and 17 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈22

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 17

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Every claim about how inspect_ai behaves must be checked against the installed pin before it goes in the verdict.

    line 19
  2. Distinctive:

    Create a worktree on the PR head.

    line 33
  3. Very specific:

    Check whether the eval has extras: grep -n "^<eval>" pyproject.toml under [project.optional-dependencies].

    line 40
  4. Very specific:

    Run uv sync --group dev, adding --extra <name> for each match.

    line 41
  5. Very specific:

    If the scorer imports a model provider module (for example inspect_ai.model._providers.openai), add --group test, which is where the …

    line 42
  6. Distinctive:

    Create the artefact directory in the main checkout, outside the worktree.

    line 46
  7. Distinctive:

    Write NOTES.md there as you go, and VERDICT.md at the end.

    line 51
  8. Distinctive:

    Do not git add this directory.

    line 52
  9. Very specific:

    Read the four non-secret settings from the main checkout's .env, and nothing else from it.

    line 55
  10. Very specific:

    If INSPECT_LOG_DIR is unset, prior logs are in the main checkout's logs/.

    line 60
  11. Distinctive:

    Read the PR body and any linked issue, then set them aside.

    line 62
  12. Distinctive:

    Run the same probe on both trees from one worktree:

    line 101
  13. Very specific:

    Keep any eval() log directory alive while you read the log.

    line 105
  14. Very specific:

    Use a task's local_path override when it has one.

    line 113
  15. Very specific:

    Use get_model("mockllm/openai/gpt-4o") when a proxied model name matters.

    line 115
  16. Very specific:

    Read the @task signature before passing keyword arguments.

    line 116
  17. Distinctive:

    If reproduction needs model API access you do not have, mark this UNVERIFIED.

    line 118
  18. Distinctive:

    Use the same inputs. Record both outputs side by side.

    line 122
  19. Distinctive:

    If the PR fixes a different bug from the one it describes, say which.

    line 123
  20. Distinctive:

    Run one case in each direction yourself.

    line 135
  21. Distinctive:

    When the fix is on the ground-truth side of a comparison, the dataset decides reachability.

    line 139
  22. Distinctive:

    Before trusting any head comparison, re-score prior logs on the reverse-applied tree and check the result equals the stored scores.

    line 141
  23. Very specific:

    If any live path moves a number: N in eval.yaml is bumped, and the README changelog and changelog.d/ fragment name which metrics move and …

    line 145
  24. Very specific:

    If no live path moves a number, N is unchanged.

    line 146
  25. Distinctive:

    Check for open PRs on the same eval:

    line 149
  26. Very specific:

    Use git merge-tree --write-tree <this> <that> for a conflict dry run.

    line 153
  27. Distinctive:

    When a parser gains tolerance for malformed input, probe the shapes it now accepts:

    line 162
  28. Very specific:

    Read pytest's own summary line, and confirm one was printed.

    line 211
  29. Distinctive:

    Read what each new test patches.

    line 213
  30. Distinctive:

    When the revert breaks collection:

    line 217
  31. Distinctive:

    If a new test module fails to import because the PR added a symbol, append a one-line placeholder for that symbol to the reverted file and …

    line 219
  32. Distinctive:

    If the PR added a whole module, write a placeholder module that aliases the new names to the old scorer's behaviour and report both counts.

    line 220
  33. Distinctive:

    Run on the PR head:

    line 227
  34. Very specific:

    If the model under test was instructed to write the file being read, a missing or unparseable file at that path is the model's output …

    line 253
  35. Distinctive:

    Read the eval's prompts before deciding whose failure it is.

    line 255
  36. Very specific:

    Do not post to GitHub. The maintainer decides what goes on the thread.

    line 296
  37. Distinctive:

    Do not fix the PR unless asked.

    line 297
  38. Very specific:

    Write one bullet per item, each tagged with one of the four kinds, and write none under a kind that had nothing.

    line 301
  39. Distinctive:

    If a framework behaviour you checked surprised you, or the tree corrected your first reading, it goes here.

    line 304

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Verify a Scoring Change

A scoring change moves reported numbers. The review has to establish three things: that the bug exists, that the fix removes it, and what else moves. The output is a verdict a maintainer can act on without repeating your work.

The seven questions to answer are the Scoring-Change Gate. The rules behind them are in BEST_PRACTICES.md. Read these four sections first:

  • "Route sample-level failures to the right outcome"
  • "Handle grader failures as instrument failures, not model failures"
  • "Say what a metric does when its denominator is empty"
  • "Dict-valued scores and metric registration"

The receipts rule

Every claim about how inspect_ai behaves must be checked against the installed pin before it goes in the verdict.

  • Find the source with uv run python -c "import inspect_ai, pathlib; print(pathlib.Path(inspect_ai.__file__).parent)".
  • Cite file:line, or run a probe and paste its output.
  • A claim without a receipt is left out.

The same rule applies to the eval. Before judging, read its README changelog, its eval.yaml version, and git log --oneline -- src/inspect_evals/<eval>/<scorer file>. Whether a branch is reachable, whether a zero was a deliberate decision, and who produces the artefact being scored are answered by that history rather than by the diff.

Behaviours that have been cited wrongly from memory

references/framework-receipts.md lists the inspect_ai behaviours that reviewers have got wrong from memory, each with a file:line on the pin it was read on. Open it when a verdict is about to rely on how the framework treats empty inputs, nan, reducers, grader parse misses, redacted reasoning, sandbox exceptions, or mockllm. Line numbers drift between releases, so re-check any row you quote against the current pin.

Setup

  1. Create a worktree on the PR head.

    • git fetch origin pull/<N>/head:pr-<N> && git worktree add .claude/worktrees/pr-<N> pr-<N>
    • Record the head SHA with git rev-parse pr-<N>.
  2. Sync the worktree so the code under review is what runs.

    • Check whether the eval has extras: grep -n "^<eval>" pyproject.toml under [project.optional-dependencies]. Extra names are not one-to-one with directories (bfcl and bfcl_v4 are separate extras), so read the matches.
    • Run uv sync --group dev, adding --extra <name> for each match. A missing extra makes the scorer fail at construction and voids every probe run afterwards.
    • If the scorer imports a model provider module (for example inspect_ai.model._providers.openai), add --group test, which is where the provider packages live.
    • An eval under packages/ has no root extra and its own lockfile. Run its tests through its tox environment (uv run --group dev tox -e <eval> -- <pytest args>; tox appends the arguments to pytest, so a probe cannot go this route) and its probes with uv run --project packages/<eval> python probe.py from the worktree root. For a packages/ eval, receipts cite the isolated lock's inspect_ai pin.
    • Confirm uv run python -c "import inspect_evals; print(inspect_evals.__file__)" prints a path inside the worktree. If it does not, nothing you run is testing the PR.
  3. Create the artefact directory in the main checkout, outside the worktree.

    • Path: agent_artefacts/<eval>_<version>/scoring_review_pr<N>/, using the version the PR head declares.
    • A PR that bumps more than one eval gets one directory per eval, with VERDICT.md in the first and a one-line pointer in the others.
    • A PR to a shared helper that touches many evals and bumps none uses agent_artefacts/utils_<helper>/scoring_review_pr<N>/, with no version component.
    • Write NOTES.md there as you go, and VERDICT.md at the end.
    • Do not git add this directory. It is ignored on main; a local checkout may lag.
    • The worktree directory is removed when the review ends. The verdict reaches the maintainer through the batch runner or, when asked, as a PR comment.
  4. Read the four non-secret settings from the main checkout's .env, and nothing else from it.

    • The file holds live API keys. The only command to run against it is grep -E '^(RUN_SLOW_TESTS|RUN_DATASET_DOWNLOAD_TESTS|INSPECT_LOG_DIR|INSPECT_EVAL_MODEL)=' .env; nothing else opens it.
    • pytest-dotenv reads .env from a nested worktree. RUN_SLOW_TESTS=yes or RUN_DATASET_DOWNLOAD_TESTS=yes run Docker and download tests that CI skips by default. Set them empty on the pytest command line (RUN_SLOW_TESTS= RUN_DATASET_DOWNLOAD_TESTS= ...) when you want CI's selection, and expect local counts to differ from the PR body's when Docker is present.
    • INSPECT_EVAL_MODEL, if set, is loaded too. A scorer test that calls get_model() with no active model then fails building that provider instead of on its assertion. Setting it empty on the command line (INSPECT_EVAL_MODEL= ...) stops the provider build, but get_model() then raises Model name '' should be in the format of <api_name>/<model_name>, where CI, with the variable absent, raises No model specified. pytest-dotenv does not overwrite a variable that is already set, so the absent state cannot be reproduced from the command line; a test that asserts on the message needs monkeypatch.delenv.
    • If INSPECT_LOG_DIR is unset, prior logs are in the main checkout's logs/.
  5. Read the PR body and any linked issue, then set them aside. Treat each claim in the description as an item to check.

Shell traps that have cost runs

  • To run a specific uv version, for example the one the uv== dev dependency pins, use uvx uv@<version> <command>. Write it out in full each time; zsh does not word-split a variable holding it.
  • A bare ===== separator is executed as a command. Use echo "---".
  • An unquoted --include=*.py fails on glob expansion.
  • Backticks inside a double-quoted grep pattern are command substitution.
  • The pipeline status array is pipestatus, not PIPESTATUS.
  • $BASE:tests/x.py applies the :t modifier. Write "${BASE}:tests/x.py".
  • A variable holding several paths reaches a command as one argument unless written ${=FILES}.
  • import inspect_evals.<eval>.<module> as m binds the package's re-exported function when __init__.py exports a same-named symbol. Use importlib.import_module.

Timeouts

Wrap every pytest run in a timeout so a hang is recorded instead of waited out. The command depends on the OS; check once with command -v timeout gtimeout.

  • Linux: timeout 600 <command>.
  • macOS with Homebrew coreutils (brew install coreutils): gtimeout 600 <command>.
  • macOS without coreutils: perl -e 'alarm shift; exec @ARGV or die "exec failed: $!\n"' 600 <command>. The or die makes a failed exec visible, since a missing command inside a pipeline otherwise reads as exit 0.

The commands below are written with timeout 600; substitute the form for your platform. A run the timeout killed exits 124 under timeout or gtimeout and 142 under the perl form, and prints no pytest summary line.

The seven questions

1. Does the bug reproduce on main?

Reproduce the claimed failure in a run of the shipped task wiring, or in a test that exercises it. A stub of the code under review does not count as reproduction.

How to reproduce:

  • For a metric: SampleScore objects shaped like what the scorer emits, fed to the metric directly, or the task wiring with mockllm and a canned grader reply.
  • For a grader path: feed a reply with no parseable verdict and a raised API error, and record each score.
  • For an extraction fix: run the scorer on the output the PR says was misparsed.
  • For a type-guard fix: register a spy metric through Task(metrics=[...]) that records the type of every leaf it receives. The implicit epoch reducer converts leaves to float before list-registered metrics run, so a hand-built int leaf is not a faithful input.
  • For a refactor: there is no bug to reproduce. Run a parity probe that computes old and new side by side on the reverse-applied tree, where both are importable.
  • For a test-only PR: the source diff is empty. Run the merge-base test file against the head test file, across framework versions if the change is version-dependent.
  • For a PR that ports an upstream scoring rule: replay upstream's published prediction files through the scorer objects of both trees, and fetch upstream's scorer to compare rules.

Run the same probe on both trees from one worktree:

  • Reverse-apply the PR's source diff as in question 6, run the probe, restore.
  • The reverse-apply and generate_readmes both change the worktree while probes import from it. Run each as one atomic command (apply, probe, restore) and nothing else in between.
  • Keep any eval() log directory alive while you read the log. Samples load lazily and a removed temporary directory raises FileNotFoundError.

Probe shapes that avoid downloads and keep the code under review in the path being probed:

  • Patch the module's hf_dataset to return a MemoryDataset of hand-built samples around the task constructor.
  • Patch datasets.load_dataset and inspect_ai.dataset._sources.hf.inspect_cache_dir instead, when the framework's shuffle-before-map behaviour matters.
  • Patch the eval's download-or-ensure-data function to return a fixture directory.
  • Pass a mockllm Model as the task's grader parameter when it accepts one, with max_samples=1 so list outputs align with sample order.
  • Use a task's local_path override when it has one.
  • Rebind a sandbox-reading scorer module's sandbox name to a fake with an async read_file.
  • Use get_model("mockllm/openai/gpt-4o") when a proxied model name matters.
  • Read the @task signature before passing keyword arguments.

If reproduction needs model API access you do not have, mark this UNVERIFIED.

2. Does it stop reproducing on the PR head?

  • Use the same inputs. Record both outputs side by side.
  • If the PR fixes a different bug from the one it describes, say which.

3. Do reported numbers move, and is that declared?

Decide reachability first:

  • A leading if not scores guard is unreachable from the framework, so changing it moves nothing.
  • A denominator that empties inside the metric is reachable in a healthy run and does move numbers.
  • A grader path on the default task configuration is live.

Measure how often and in which direction:

  • Run one case in each direction yourself. A "can only decrease" claim has been wrong before.
  • Sweep a list of completion shapes through the old and new comparison functions. A probe on stored samples alone has found only the upward moves, and the shapes the PR body lists have so far been the set where old and new rules coincide.
  • For a metric change, run a seeded random-population sweep through the registered metric object on both trees.
  • When a return becomes a raise, or a rule's reach depends on the shape of dataset records or schemas, run the new branch over the dataset and count how often it fires. A raise that hits 5% of a healthy run is a different change from the one described.
  • When the fix is on the ground-truth side of a comparison, the dataset decides reachability.
  • For an extraction or parsing change the population is model completions. Look in INSPECT_LOG_DIR (from the step 4 grep, else logs/ in the main checkout) for prior .eval logs of the eval. Read log.eval.model from each log header and keep only the eval's runs made with a non-mock model, by model and date; concurrent reviewers write mockllm runs to the same directory. Re-score those completions on both trees.
  • Before trusting any head comparison, re-score prior logs on the reverse-applied tree and check the result equals the stored scores. That proves the revert reproduces shipped behaviour.

Check the declaration:

  • If any live path moves a number: N in eval.yaml is bumped, and the README changelog and changelog.d/ fragment name which metrics move and in which direction.
  • If no live path moves a number, N is unchanged.
  • X is bumped only when the task interface changed, whether or not numbers move. See TASK_VERSIONING.md for both axes.

Check for open PRs on the same eval:

  • Two PRs that both bump 4-A to 5-A merge without a textual conflict and ship a duplicate version.
  • gh pr list --search <path> is full-text search and most hits are false positives. Confirm each with gh pr view N --json files.
  • Use git merge-tree --write-tree <this> <that> for a conflict dry run. It writes objects into the repository's object database and touches no worktree.
  • gh pr view --json mergeable returns UNKNOWN until GitHub has computed it.

4. Does every exclusion sentinel mean one thing?

  • List every path that yields nan, Score.unscored(), or an omitted key, and write down what each means.
  • "The grader failed" and "the grader was not consulted because the agent complied" are different facts. If both produce nan in the same key, a metric that excludes nan drops the second group from its denominator with no record of having done so.
  • Apply the coherence rule. Model-attributed reasons (invalid_response_format, refusal, no_response) belong on scored samples in the denominator. Instrument reasons (grader_failed, scoring_failed) belong on exclusions.

When a parser gains tolerance for malformed input, probe the shapes it now accepts:

  • an unclosed tag or fence followed by bare text
  • an opening fence with no closing fence (a truncated completion)
  • a fence inside a string literal or docstring
  • a closing tag or fence followed by prose
  • a nested block
  • a case variant

Tolerance that lets a refusal count as the harmful action is a new false-positive channel. It appeared in #2294 (a refusal after an unclosed tag), #2131 (a refusal typed inside reasoning) and #2326 (a truncated fence).

5. Is anything swallowed or fallen into?

Grep the changed files and their callees for:

  • except Exception followed by a constant return
  • else 0.0 or if not <list>: return 0.0 in a function that divides
  • == CORRECT on a grader result with no unscored branch
  • metadata={"reason": (the key that does not migrate to Score.reason, which exists from inspect_ai 0.3.261)
  • a bare score.value["key"] in a list-registered metric

Then classify each hit:

  • A zero that a test asserts on purpose is a decision and may stand.
  • A zero that nothing asserts is a defect.
  • A pre-existing defect is a blocker if the PR makes it reachable (for example, a new filter that can empty the list a pre-existing else 0.0 guards). Otherwise it is a follow-up.

6. Do tests pin both sides?

Revert the PR's own source changes against its merge base, then run the PR's tests and count the failures. That count is the number of behaviours pinned.

BASE=$(git merge-base HEAD origin/main)
test -z "$(git status --porcelain -- src/)" &&
  git diff "$BASE" HEAD -- src/ | git apply -R &&
  timeout 600 uv run pytest tests/<eval> -v --tb=line --color=no --continue-on-collection-errors
git checkout HEAD -- src/ && git clean -fd -- src/  # restore the PR head whatever happened above

Why this shape:

  • The merge base rather than main, so unrelated newer changes to the eval are not pulled in.
  • Reverse-apply rather than checking out a directory, so files the PR added are removed and changes to shared helpers such as src/inspect_evals/utils/ are included.
  • The && chain, so pytest runs only if src/ was clean and the reverse-apply succeeded. git apply is atomic: a leftover probe edit or a placeholder line makes it apply nothing and exit 1, and an unguarded pytest would then run against the PR head and report the head's results as the reverted tree's. The restore line runs either way.
  • timeout 600 is the Linux form. See Timeouts above for macOS.
  • Evals under packages/ use the tox command from question 7.

Reading the result:

  • Read pytest's own summary line, and confirm one was printed. A run the timeout killed has no summary line, and pytest's output is colour-coded by default so a grep for FAILED misses without --color=no.
  • Record the summary count, and also list each new test's result on the reverted tree. A fixture that happens to give the same value under both codes pins nothing.
  • Read what each new test patches. A test that monkeypatches the function under review pins nothing and passes on both trees.
  • Also list any pre-existing test that passes on the reverted tree and fails on the PR head. Authors often run only the file they edited.
  • For a refactor the expected count is zero. Run each deleted test against the head from a copy of the merge-base file: an assertion failure is a behaviour change to report, a collection error means the test only named a moved symbol, and a pass means coverage was dropped for no reason. The parity probe from question 1 must pass.

When the revert breaks collection:

  • If a new test module fails to import because the PR added a symbol, append a one-line placeholder for that symbol to the reverted file and rerun.
  • If the PR added a whole module, write a placeholder module that aliases the new names to the old scorer's behaviour and report both counts.
  • The revert covers src/ only, so a test the PR deleted stays deleted. Diff the eval's test files against the merge base for removed test functions, and run any removed test against the head from a copy of the merge-base file.

Then confirm a correct zero still reports 0.0. For a grader path the three cases are: parse miss, raised API error, genuine negative verdict.

7. Are the checks green?

Run on the PR head:

  • uv run ruff check
  • uv run ruff format --check
  • uv run mypy <changed files>
  • uv run --group dev python tools/check_changelog.py
  • uv run --group dev python tools/generate_readmes.py --create-missing-readmes, then uv run --group dev pre-commit run mdformat --all-files and uv run --group dev pre-commit run markdownlint-fix --all-files. CI (checks.yml) runs these three in this order and fails on any resulting change anywhere in the tree.
  • timeout 600 uv run pytest tests/<eval> --tb=line --color=no, or uv run --group dev tox -e <eval> for an eval under packages/, whose tests are excluded from the root suite. This is the head run that question 6 compares against; run it once.

Reading the results:

  • After the three README commands, the pass criterion is CI's: git status --porcelain shows nothing anywhere in the tree. Running generate_readmes without the two formatters leaves dozens of READMEs reformatted; that is the missing formatting step, not generator drift. A branch behind main can still show changes in other evals' READMEs from listing-derived content; record those as drift from the base and confirm they disappear after merging origin/main before recording the check as green. Restore with git checkout -- . once git status lists only files these commands wrote.
  • A pytest run that collects no tests is recorded as not run. A fired timeout is recorded as a check that did not complete.
  • Fork PRs get no CI until a maintainer approves it, so a local ruff or mypy is often the first anyone has run.
  • gh pr checks reports a cancelled review workflow as fail. Read the run's conclusion before recording a failed check.

Two mypy traps recur:

  • A Metric-typed value is a union and cannot be called. Narrow with cast(MetricProtocol, m).
  • math.isnan(score.value) on the Value union. Narrow with isinstance(score.value, float) first.

Common mistakes

Applying the instrument rule before finding out who writes the artefact.

  • The words "sandbox", "file", and "JSON" describe a mechanism. They do not say who produced the content.
  • If the model under test was instructed to write the file being read, a missing or unparseable file at that path is the model's output failing. That is a verdict (Score(0.0), in the denominator) rather than an instrument failure.
  • The same applies to a judge that is the model under test grading itself.
  • Read the eval's prompts before deciding whose failure it is. Attribution comes before the instrument rule.

Citing a framework behaviour from memory. Sound reasoning on a stale premise produces a confident wrong finding. See the receipts rule.

Missing a version collision between sibling PRs. See question 3.

Trusting the PR body's vocabulary claim. Grep for the constants in the code.

The verdict

## Verdict: MERGE-READY | NEEDS-WORK | UNVERIFIED

One sentence on what the PR does and whether it does it.

**Checks run:** one line per command with its result.

### Findings
1. **[REQUIRED|ADVISORY] <file>:<line>**: what is wrong, the fix, the receipt.

### Gate
| Q | Result | Receipt |
| --- | --- | --- |
| 1 Reproduces on main | yes/no/unverified | ... |

**Outcome-changing or mechanical:** one line.
**Sequencing:** sibling PRs, version collisions, dependencies. Omit if none.

### Learnings
- **skill:** a step in this skill that was missing, wrong, or unclear, and what you did instead.
- **gotcha:** an environment or tooling trap that cost time (a missing extra, a hang, a tool that failed).
- **framework:** an `inspect_ai` behaviour you checked on the pin that belongs in the receipts list, with its `file:line`.
- **eval:** something about this eval a future reviewer needs (who the judge is, where the fallback came from, which extra it needs).

**Reviewer:** model id, PR head SHA, git SHA of this file, inspect_ai version, date.

Rules for the verdict:

  • UNVERIFIED is the verdict whenever question 1 or 2 could not be run.
  • End VERDICT.md with the same content as a fenced JSON block, so a batch runner can collect verdicts and learnings without parsing prose. Keys: verdict, summary, checks_run[], findings[{severity,file,line,summary,fix,receipt}], gate[{question,result,receipt}], outcome_changing, sequencing, learnings[{kind,note,receipt}] with kind one of skill, gotcha, framework and eval, and reviewer{model,pr_head,skill_sha,inspect_ai,date}.
  • Do not post to GitHub. The maintainer decides what goes on the thread. If asked to post, end with *Posted by [Claude Code](https://claude.com/claude-code) on <maintainer>'s behalf.* and ping no one.
  • Do not fix the PR unless asked.

The Learnings section is how the skill improves:

  • Write one bullet per item, each tagged with one of the four kinds, and write none under a kind that had nothing.
  • Include the things that went wrong on the way to the verdict, not only the verdict.
  • A maintainer reads the learnings from every run in a batch and folds the accepted ones into this file. A note here reaches the next reviewer; a note that stays in NOTES.md does not.
  • If a framework behaviour you checked surprised you, or the tree corrected your first reading, it goes here. The maintainer decides what is folded into this file.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.