agentleFS
craigcossairtcraigcossairt/trellis/.claude/skills/pin-guardrail/SKILL.md

pin-guardrail

Skill5 starsChanged 12 days agoMITView on GitHub

Pin a shell guardrail (a hook, a git hook, a checker script) with a behavioral test suite, prove the suite is real by mutating the code, and wire it into CI with its mutation ledger. Use when asked to "add tests for this hook", "pin that guard", "mutation-test this script", "this hook has no test", "wire the suite into CI", or after changing any guardrail's behaviour.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 5 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Read an existing suite in bin/tests/ and match its shape rather than inventing a harness.

    line 41
  2. Distinctive:

    Before mutating, write down for each mutation which named cases should go red.

    line 64
  3. Distinctive:

    Run the suite and record which cases went red.

    line 84
  4. Distinctive:

    Another check backstops the one you are testing: Remove a guard and the exit code stays the same because something downstream catches the …

    line 97
  5. Distinctive:

    Add a named step to the workflow:

    line 119
  6. Very specific:

    Check in the same change: git hooks are executable (git skips a non-executable hook silently, and on a machine with core.fileMode=false a …

    line 143
  7. Distinctive:

    When you report, say what the suite asserts and what it does not.

    line 157

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Pin a guardrail with a suite you can believe

A guardrail with no behavioral test asserts nothing. CI checking that a script parses says nothing about whether it blocks, and a guard that silently stops working is worse than no guard, because somebody is relying on it.

The suite is the deliverable. The mutation run is what makes the suite believable - and most of this file is about the ways a green suite lies to you, because every one of them has actually happened in this repo.

Scope: this is written for shell guardrails - the concrete fixture rules, directory and CI wiring below assume them. The mutation discipline in §3 to §5 is not shell-specific and carries over to anything you can break on purpose.

0. Is it gated? If not, stop

Open the CI workflow and confirm it runs the directory you are about to write into. Do this by reading it, not by assuming - a suite CI never executes is decoration that reads as coverage, which is worse than no suite at all.

At the time of writing that means bin/tests/, with .github/workflows/hooks-ci.yml running each suite as its own named step. Treat that as the answer to check against, not as a reason to skip checking: the layout is exactly the kind of thing that moves. A named step matters either way - read step conclusions rather than the job conclusion, because a job can be green while a step inside it was skipped.

1. Name the behaviours before you write a single case

List what the script promises: each block-or-allow decision, each exit code, each could-not-run path, each normalisation it performs. One case per behaviour, and write the list down first. That list is what you compare the mutation results against in step 4. Without it you will be comparing against a fixture count, which is the wrong number in both directions.

2. Write the suite

Read an existing suite in bin/tests/ and match its shape rather than inventing a harness.

Make it hermetic: temp trees, no network, no installed toolchain. Inject external tools as stubs and build git fixtures directly. Anything a suite needs from the outside world is a reason it will be skipped later, and a skipped suite is an unpinned guardrail.

End with a plain passed: N failed: N line and exit non-zero on any failure.

Fixture rules, each of which has burned this repo:

  • Never assert on an exit code alone when the script distinguishes 1 from 2. A helper that maps any non-zero onto the expected one makes every case vacuous.
  • Assert the message text wherever the message states a rule. An error message is an instruction: if it tells the reader to do something the code no longer supports, no exit-code case can see that. This is also, repeatedly, the only thing that survives a mutation where another check backstops the one you are testing - see §5.
  • A skip is not a pass. Print it out loud, say why, and never count it.
  • If the script has a two-layout dimension - sibling versus nested, relative versus absolute, one worktree versus two - build a fixture for both. A single-layout fixture makes two different resolutions name the same directory and passes whatever the code does.

3. Predict, by name

Before mutating, write down for each mutation which named cases should go red. Not how many. A mutation only has to fail the cases exercising the behaviour it broke, so the other cases staying green is not suspicious - but a case you predicted that stays green is passing for a reason you have not accounted for.

4. Mutate

Commit first. A harness that restores with git checkout -- destroys any uncommitted work, including the fix you are testing.

For every mutation the harness must:

  1. Diff the mutant against the original, and refuse to report a result if they are identical. A pattern that no longer matches leaves the file unchanged, every case passes, and that all-green reads as "my suite does not cover this" - sending you to add a redundant case or rip out a guard that was working. It was never a mutation run at all.
  2. Print the diff, and read it. "It applied" is not "it applied correctly". A replacement can change a different thing than you meant - an unescaped $ in a perl replacement interpolates to empty and blanks an argument instead of changing a flag - and the file changed, so the identical-check passes it.
  3. Run the suite and record which cases went red.

Never delete or weaken a case to make a run green.

5. Read the result properly - five ways a case lies

This is the part that matters, and none of it is visible from reading the suite.

The fixture makes the attack impossible anyway. A case asserting that a path-traversal guard refuses ../outside proved nothing here, because the fetch for that path failed regardless - removing the guard still exited 2. Build the fixture so the bad thing genuinely works without the guard, then assert it did not happen. If you cannot make it work, the case is documentation.

Another check backstops the one you are testing. Remove a guard and the exit code stays the same because something downstream catches the same input a few lines later. The guard's real contribution is then its message, and only a message assertion can go red. Record that in the ledger rather than contriving a red - and read that ledger line before anyone "simplifies" the guard away.

The case went red for the previous case's reason. Cases sharing a fixture are coupled: a mutation that lets bad content through corrupts the fixture, and the next case fails for that rather than for its own subject. An unexplained extra red is as much a signal as a missing one. Chase it; give the case its own fixture.

The capability probe tested the wrong direction. A case gated on chmod +x sticking ran on a filesystem where +x sticks and -x does not - so it could never go red. Probe the exact property the case needs, in the direction it needs.

A skipped case counted as a pass. Say it out loud in the output, and say in the ledger which platform actually exercises it. If a probe can hang rather than fail - ln -s on MSYS without developer mode does exactly this - gate on the platform instead of attempting it, or one unavailable capability wedges the whole suite.

6. Wire it into CI, with the ledger

Add a named step to the workflow:

      - name: Suite - <thing>
        run: |
          # <what this pins, and why it exists at all>
          #
          # Mutation ledger, measured <date> over N cases:
          #
          #   <mutation>                          N predicted, N red
          #   <mutation>                          N predicted, N red
          #     <why the numbers differ, if they do>
          #
          bash bin/tests/test-<name>.sh

The ledger is not optional. It is the only place the figures survive, and a later reader cannot tell a considered suite from a careless one without it.

Record the misses, not just the wins. The most useful line in the block is the one saying a case proved nothing and why it is kept anyway. §5 is a list of what to watch for, not a list of entries you owe - record the ones that actually happened, and if none did, the ledger is just the figures.

Check in the same change: git hooks are executable (git skips a non-executable hook silently, and on a machine with core.fileMode=false a chmod never reaches the index - use git update-index --chmod=+x); no || true on a grep, and no bare if grep -q where the file list came from anywhere but a glob over the working tree. Split the three codes:

rc=0; grep -q … || rc=$?
case $rc in 0) hit;; 1) clean;; *) fail loudly;; esac

Unknown must never resolve to green.

7. Say what it does not pin

When you report, say what the suite asserts and what it does not. A suite that covers wiring but not behaviour, or one whose interesting cases only run on one platform, is worth having and worth being honest about.

And remember what mutation testing cannot do: it bounds the gap between your code and your tests, never between your code and reality. It cannot surface an input you never considered or a rule you stated and then contradicted. For anything whose failure you would care about, follow docs/methodology/adversarial-review.md as well.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.