agentleFS
allxsmithallxsmith/bestax/.github/CLAUDE.md

bestax

CLAUDE.md14 starsChanged 8 days agoMITView on GitHub

Everything in .github/workflows/ is human-authored by design. Two enforcement layers apply here and they are not the same rule — do not collapse them: - Automated sessions (the AI fix loop, ai-triage, ai-scan) are denied write access mechanically: Edit(.github/**) and friends sit in every write agent's --disallowedTools, and the fix loop refuses any PR that touches .github/**. This is not advisory, and editing this file does not relax it. Those sessions ingest untrusted issue and PR text, so for them…

  • Reads credentials
  • Installs packages

The file

Show its rules and text

We split this CLAUDE.md into the rules it gives an agent

20 rules. About 0 are common, 10 are distinctive, and 10 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈10

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 10

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    An interactive session working alongside a maintainer: should not hide behind that.

    line 11
  2. Distinctive:

    When unsure which row a change belongs in, grep the workflows for the thing you are changing.

    line 47
  3. Distinctive:

    Every rule below serves one threat model, stated here once so a review has a place to stop and an author has something to cite instead of …

    line 62
  4. Distinctive:

    Every AI workflow here is built to preserve these.

    line 115
  5. Distinctive:

    When bumping, bump every occurrence in a single commit and verify there is exactly one:

    line 186
  6. Very specific:

    harden-runner has an assertion tied to its pin: Every block-mode job asserts the effective policy out of /home/agent/agent.json (rule 10).

    line 220
  7. Very specific:

    The pin does not fully determine which agent binary runs: Before installing, the pre-step calls isTLSEnabled(owner), a 3-second probe of a …

    line 225
  8. Distinctive:

    Read the third column for which credential — since #455 and #457 neither session can reach repository write any more, and "the session …

    line 245
  9. Very specific:

    Never add Bash(gh api:*) to a session that ingests untrusted issue/PR text — it is a general-purpose write primitive wearing a read-shaped …

    line 258
  10. Very specific:

    Never add Edit, Write, MultiEdit, or Task to ai-scan.yml.

    line 260
  11. Very specific:

    Prefer removing the need for the boundary over hardening it.

    line 263
  12. Distinctive:

    Every repository variable that steers this automation is tabulated in the ai-development docs guide, including its unset default.

    line 294
  13. Very specific:

    Use plain pull_request, never pull_request_target, plus an explicit head-repo guard (#312).

    line 332
  14. Distinctive:

    Do not close this by checking out the PR head — running PR-authored code in a job holding a credential is the thing that pin prevents, and …

    line 368
  15. Distinctive:

    Do not maintain any of this by hand.

    line 591
  16. Distinctive:

    Keep this inventory correct when you add a job, and qualify per job rather than per workflow.

    line 619
  17. Very specific:

    Verify rather than assume, on any run: the assertion step passes, and harden-runner's post-step prints the effective EgressPolicy:.

    line 635
  18. Very specific:

    When the pre-step selects the closed-source agent-ebpf instead (rule 1), none of it can be checked from here.

    line 677
  19. Very specific:

    Run 33221210633's log shows auto-close-duplicates at block connecting to api.github.com and github.com and nothing else.

    line 781
  20. Very specific:

    Never cite a line number in prose.

    line 892

≈ “Common” is an estimate: at least 1 in 5 of the 27,641 CLAUDE.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
CLAUDE.md

.github — CI and AI automation

Everything in .github/workflows/ is human-authored by design. Two enforcement layers apply here and they are not the same rule — do not collapse them:

  • Automated sessions (the AI fix loop, ai-triage, ai-scan) are denied write access mechanically: Edit(.github/**) and friends sit in every write agent's --disallowedTools, and the fix loop refuses any PR that touches .github/**. This is not advisory, and editing this file does not relax it. Those sessions ingest untrusted issue and PR text, so for them the prohibition is absolute.
  • An interactive session working alongside a maintainer should not hide behind that. State the rule, state the security reason below that it exists for, make sure the maintainer has actually weighed that reason — then act on their answer. Refusing to apply a change the maintainer has decided on does not make the repository safer; it just turns one decision into several round-trips. The maintainer was always the one deciding.

The reason to raise, so nobody has to rediscover it: a change in this directory can grant capability with no permissions diff for a reviewer to notice. The permissions: block looks identical before and after an allowlist widens. That is why these changes get argued rather than waved through — not because agents are untrusted, but because the usual review signal is absent.

Scale the pushback to the blast radius:

ChangeWhat to do
--allowedTools / --disallowedTools on a session holding a credentialArgue it properly, every time. Name the credential in that job, explain why each entry cannot write, and get an explicit decision. Never as a side effect of another task. See rule 2.
Workflow logic, triggers, permissions:, action SHAs, verdict/budget paths, anything a workflow reads as a gate or triggerName the invariant at stake (I1, I2, or the rule below), confirm it holds, then act.
Genuinely inert config — a cosmetic label, a docs-build cadence, a semver-major dependabot.yml ignoreJust make the change and say what it does. Do not gate it behind a debate.

The qualifiers in that last row are load-bearing. The dividing line is cosmetic vs. load-bearing, not the file the change lives in:

  • Labels are not uniformly inert. needs-security-review is a refusal gate — claude-repro, claude-fix, @claude and @bestaxbot all decline a flagged item until a maintainer clears it. ai-loop, ai-loop-paused and deep-review steer automation the same way. Deleting or renaming one of those disables a control with no workflow diff at all, which is the exact hazard this section exists to name. A new documentation label is inert; a label a workflow reads is middle row. review-converged is written by review-converged.yml for the owner to read. That workflow reads it back only to decide whether to add or remove it, and no automation gates on it.
  • Schedules are not uniformly inert. A schedule: on ai-scan or the stale sweep decides when a security control runs. A docs-build cadence does not.
  • dependabot.yml ignores are not uniformly inert. A semver-major ignore only declines a breaking upgrade, but one covering patch or minor suppresses CVE fixes for that dependency. Scope every ignore you add, and never widen an existing one's update-types as a drive-by.

When unsure which row a change belongs in, grep the workflows for the thing you are changing. If anything reads it, it is middle row.

Origin: an interactive session hit the old blanket "propose a diff, do not apply one" line while adding a dependabot.yml ignore entry — a change with no security surface at all — and stalled on it across three separate asks. The blanket rule was cheap to state but it spent the maintainer's attention on the one case that did not need it, which is attention not available for the allowlist case that does.

This file is the security contract for these workflows. The rules below are not style preferences — each one is load-bearing, and most were written after a review round or a red-team found the failure it prevents. Where a rule has a documented origin, it is cited.

The trust boundary

Every rule below serves one threat model, stated here once so a review has a place to stop and an author has something to cite instead of shipping another defence for the same step.

What these jobs are exposed to. The deep review and the fix loop check out branch code from an open PR and run its package manager and its tests; @claude and @bestaxbot join that class when a review event triggers them, and check out the default branch when a comment or an issue does. Everything under a PR checkout is attacker-influenced: the sources, the lockfile, package-manager hooks such as .pnpmfile.cjs, .gitattributes, and any file a later step reads back from the working tree. The scan, triage and repro sessions check out the default branch and install nothing, so their exposure is the issue and PR text they ingest, which is attacker-authored on every one of these jobs. A model token shares the job with whatever it reads. Where the PAT sits differs and is the thing to check: triage hands a finished payload to a publish job that holds the PAT and runs no model; repro's publisher holds no PAT at all and posts with GITHUB_TOKEN (I2); the fix loop and @bestaxbot hand the PAT to the session itself. That exposure is the job: reviewing and fixing code means running it, and no rule here pretends otherwise.

The accepted defences. These are the answer, and a review does not re-litigate whether they are sufficient: actions pinned to one SHA repo-wide (rule 1); tool allowlists held as a confinement boundary (rule 2); explicit opt-in for anything that spends usage (rule 3); a deterministic sanitizer or renderer on the publication paths that have one — repro's draft and triage's comment (rule 5, I2); plain pull_request with a head-repo guard, never pull_request_target (rule 7); sender exclusions on comment triggers (rule 8); trust decisions re-verified against the live API rather than read from the event payload (the trusted-labeler gates); an enforced egress policy (rule 10, and read I1 for exactly what that leg adds); and a human merge — the loop never merges. A finding that one of these could be stronger is advisory unless it shows the control absent or false.

Where it stops. A step boundary inside one job is not a trust boundary. A file a step writes for a later step of the same job is trusted to the degree the job is: the checkout already influences every step that runs after it, so hardening that hand-off against the checkout is worth doing when it is cheap, and is not a blocking defect when it is not. Origin: #647, where the step that writes the review delta for a later step drew round after round of correct, ever-narrower findings while every rule in this file held, because nothing said when enough was enough.

What is blocking on that axis, however the change is dressed:

  • anything that widens who can trigger a job — a new trigger, a looser if:, a sender exclusion dropped, a label gate bypassed;
  • anything that widens what a credentialed job can reach — a tool, an endpoint, a scope, a PAT where a GITHUB_TOKEN was, a checkout of PR code where the default branch was;
  • anything that widens what leaves the job — a new comment or artifact path, model or issue text reaching a publisher without the sanitizer, a posting identity that re-triggers.

That list ranks exposure, and only exposure. An exposure finding naming none of those goes on the record as advisory and does not hold the merge. A functional defect — broken shell or jq, a swallowed error, a silent no-op, a dead condition, a race — is blocking on its own merits under the normal review criteria, and nothing in this section lowers it.

The two invariants

Every AI workflow here is built to preserve these. If a change breaks one, the change is wrong, regardless of how convenient it is.

  • I1 — in the publishing pipeline, the model-auth token never shares a job with code execution. CLAUDE_CODE_OAUTH_TOKEN pays for a session; anything that can run code in the same job can read it out of the environment. claude-repro.yml splits drafting from publishing for exactly this reason: the drafting job has no Bash, no Task, no network tool, so it cannot fetch or execute. Note the direction of the guarantee — it raises the cost of exfiltration, it is not a proof. The credential-shape check in Collect draft (literal, base64, and hex forms) is a backstop for the same reason, and is documented as one.

    Read the scope precisely, because the unqualified version is false. This is a property claude-repro is built to preserve, not a repo-wide fact. claude-implement, claude-pr-loop (fix/verify), claude-review, claude and bestaxbot-reply all check out a branch and run repository code in the same job as that token, and they must — fixing and reviewing code is the job. What holds them is a different, weaker set: the tool allowlist, the protected-path deny rules, the trusted-labeler gates, and — as of #578, for all five of them — an enforced egress policy on a measured allowlist. Read that last leg for exactly what rule 10 says it delivers: it narrows where a session that does not escalate can send data, and api.github.com is necessarily on every one of those allowlists, so it does not bound what the session can do through a tool. Nor does it hold an injected session that runs code: that code can reach past the list ("What the agent allows beyond the list" under rule 10). Saying I1 covers those jobs would be the exact overstatement the checklist at the bottom of this file ends on. It covers the pipeline that was designed around it.

    harden-runner's egress block enforces again as of #487 — block used to degrade silently to audit, and now enforces and is asserted (rule 10). Do not promote it to a third leg of I1, which an earlier draft of this file did while claude-repro.yml said the opposite: on the exfil path I1 exists to close — Write plus a later publishing job — the drafting job's egress policy bounds nothing, because the publishing job has no policy of its own. Be exact about what it does add: it narrows where a session that does not escalate can send data, not what it can do with the hosts it is allowed. api.github.com is necessarily allow-listed in every one of these jobs, so egress-block cannot stop a write issued through a tool. Never trade away a tool restriction on the grounds that "egress is blocked anyway" — the two controls cover different things, which is the whole reason both exist.

  • I2 — no untrusted or model-authored free text reaches a re-trigger-capable identity. Comments posted with GITHUB_TOKEN do not emit workflow events. Comments posted with a PAT do. Two shapes address this, and only one of them is airtight — be precise about which you are relying on.

    claude-repro.yml changes the IDENTITY: the drafted reproduction is sanitized deterministically and posted via GITHUB_TOKEN, whose comments emit no events at all. That is structural — no text can re-trigger anything, whatever it says.

    ai-triage.yml keeps the PAT identity (#361 wanted it) and narrows the TEXT instead. Since #457 the session posts nothing; a deterministic renderer owns every structural string (the markers, Duplicate of #N, Fixes #N, the auto-close notice) and every issue reference is a validated integer. But the published body still carries two bounded model-authored fields, a candidate's title and a one-line reason, escaped and defanged against the KNOWN re-trigger vectors — mentions, markers, autolinks, machine sentinels. It cannot defang a trigger token nobody has invented yet: a literal /retest in a title survives into the comment today. So for that residual class rule 8's sender exclusions are still the layer, not a second one, and a future comment-triggered workflow that forgets them reopens it.

    Publishing the two fields is a deliberate, revisitable trade — a prose-free comment (bare #N references, or a renderer-owned enum in place of the reason) would make this structural too, at the cost of the signal a human reads the comment for. Do not describe the triage path as closing I2 by construction; describe it as narrowing the surface and keeping rule 8.

Hard requirements

1. Pin every third-party action to a full commit SHA — the same SHA everywhere

SECURITY.md advertises SHA pinning as an active control, so this is a promise to users, not housekeeping. One SHA per action across all workflow files, and the trailing comment must name the version that SHA actually is.

Current pin: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 (this is v7.0.1).

When bumping, bump every occurrence in a single commit and verify there is exactly one:

grep -rho "actions/checkout@[a-f0-9]\{40\}" .github/ | sort | uniq -c   # expect one line

That one is scoped to the action you are bumping. Nothing scopes the rule that way, so also run the repo-wide form, which names any action carrying more than one SHA:

grep -rhoE "[A-Za-z0-9_.-]+/[A-Za-z0-9_./-]+@[a-f0-9]{40}" .github/ \
  | sort -u | cut -d@ -f1 | uniq -d          # expect NO output

The per-action grep only ever sees the action you thought to type, so a second action left on a stale SHA by an earlier bump stays invisible to it however many times you run it.

Note both greps cover .github/, not .github/workflows/: .github/actions/verified-commit/ pins actions/github-script too, and the narrower path cannot see it — it would print exactly the "one line" this rule tells you to expect while the composite action sat on a stale SHA.

Those greps are the whole verification. No CI gate checks workflow content — format:check does lint .github markdown and .mjs, and ci.yml runs .github/scripts/pin-react.mjs, so the directory is not untouched by CI, but nothing verifies a pin, a policy or a permission. Three consequences worth holding on to:

  • Do not move a security-relevant step into a local composite action. Not for the pin's sake — both greps recurse over .github/, so a pin under .github/actions/ still matches, and an earlier draft of this bullet claimed the opposite. The reason is that reading the job is the whole review signal here, and harden-runner is the case that matters: its egress-policy and allowed-endpoints are argued per job against the credential in that job (rule 2), and rule 10's inventory is checked by reading each one. Behind a wrapper, a job's egress posture is no longer visible where the job is, and a shared wrapper collapses those separate allowlists into one — a widening with no permissions: diff, which is the hazard at the top of this file.
  • harden-runner has an assertion tied to its pin. Every block-mode job asserts the effective policy out of /home/agent/agent.json (rule 10). That path is an internal detail of the action with no compatibility guarantee, so a bump that moves or renames it fails every block job at once. That is the intended direction, but check it when bumping rather than being surprised by it.
  • The pin does not fully determine which agent binary runs. Before installing, the pre-step calls isTLSEnabled(owner), a 3-second probe of a StepSecurity endpoint that fails open — on any error or timeout it logs Unable to check TLS_STATUS. Defaulting to TLS enabled. and returns true, selecting agent-ebpf instead of agent. Only the latter is source-available, so the paths and status strings rule 10 asserts can only be verified against that one. Today this org's endpoint answers 403 (TLS_NOT_ENABLED), so the verifiable binary is what runs; a StepSecurity outage would swap it with no diff and no pin change. If every block job goes red at once with no workflow change, check this before anything else.

Origin: #361 shipped two new workflows pinned to 9c091bb… (v7.0.0, twelve commits behind) while nineteen other usages were on v7.0.1 — and both were commented # v7, so the drift was invisible to a reader. Copilot caught it; nothing in CI would have.

2. Never widen --allowedTools on a session that holds a credential

Treat every allowlist in this directory as security-critical. It is a confinement boundary, not a convenience list, and widening it grants capability with no permissions diff for a reviewer to notice — the permissions: block looks identical before and after.

Two sessions where the allowlist is the only thing between untrusted text and a credential. Read the third column for which credential — since #455 and #457 neither session can reach repository write any more, and "the session can't write to the repo" is not the same claim as "the allowlist stopped mattering":

WorkflowCredential in the session's jobWhat the allowlist is holding back
ai-triage.ymlCLAUDE_CODE_OAUTH_TOKEN; job GITHUB_TOKEN is read-scoped (contents/issues/pull-requests) since #457Repository write: nothing any more. This session held AI_LOOP_PAT — full repo write, unscoped by the job's permissions: block — until #457 split publishing out: the session emits a structured payload, scripts/render-triage-comment.mjs renders the comment, and the PAT lives only in a publish job that runs no model. The model credential: everything. The session still runs Bash (GET-only gh) and Task beside CLAUDE_CODE_OAUTH_TOKEN, so by I1 the allowlist is what stands between an injected session and reading that token out of the environment. #457 closed the concrete instance of that — Read is not workspace-confined, and the prefix match accepted gh issue comment N --body "$CLAUDE_CODE_OAUTH_TOKEN". Narrow, not retired.
ai-scan.ymlCLAUDE_CODE_OAUTH_TOKEN; job GITHUB_TOKEN is read-scoped (contents/issues/pull-requests) since #455Repository write: nothing any more. The budget marker and the needs-security-review label moved to separate gate and label jobs that run no repository code, and only the coarse verdict enum crosses between them, so widening the allowlist can no longer grant issue/PR write. The model credential: everything. This job still runs Bash beside CLAUDE_CODE_OAUTH_TOKEN, so by I1 the allowlist is what stands between an injected session and reading that token out of the environment. Egress-block IS enforced here as of #487 (rule 10), and the two are complementary rather than redundant: while the allowlist holds, it narrows where data can go, though api.github.com is necessarily allow-listed, so it cannot stop a write issued through a tool. It is no backstop once the allowlist is defeated: a shell on the runner can reach past the list ("What the agent allows beyond the list"). Narrow, not retired.

Concrete rules:

  • Adding any entry to those two allowlists is a security change. Say so in the PR description and explain why the entry cannot write.
  • Never add Bash(gh api:*) to a session that ingests untrusted issue/PR text — it is a general-purpose write primitive wearing a read-shaped name.
  • Never add Edit, Write, MultiEdit, or Task to ai-scan.yml.
  • --disallowedTools is defense in depth, and its deny rules do take precedence over the allows — but do not lean on it as the primary control. Narrow the allowlist.
  • Prefer removing the need for the boundary over hardening it. #455 is the worked example: ai-scan.yml was one job whose write scopes the model session merely happened not to use, and it became three (gate / scan / label) so the session's own job grants are read-only. The allowlist did not change; what changed is that it is no longer the only thing standing behind it. When a session's job holds a write scope for the benefit of some other step, that is the shape to look for.
  • #457 is the same move against the harder case, and worth reading second: ai-triage's session did not merely sit near a write credential, it held one — the Claude action installs its github_token as the session's GH_TOKEN, so passing AI_LOOP_PAT handed an untrusted-text-ingesting session full repo write. Splitting the job was the only way to get it a genuinely read-only credential without minting a new secret, because GITHUB_TOKEN is job-scoped and the budget marker needs issues: write somewhere. The lesson to carry: when a session's credential is passed to the action rather than merely present in the job, narrowing the allowlist cannot reach it at all — only moving the work can.

3. Opt in explicitly for anything that spends model usage

Gate on == 'on', never on != 'off'. Unset, empty, a typo, and every spelling of "stop" must all mean off — no misspelling should be able to start a control that spends usage on every incoming item.

if: vars.AI_LOOP_ENABLED == 'true' &&
  (vars.AI_SCAN_MODE == 'on' || vars.AI_SCAN_MODE == 'y')
  • GitHub Actions == on strings is case-insensitive, so ON and Y work too. Do not add case variants to the expression.
  • The one deliberate exception is AI_TRIAGE_MODE, whose label path is != 'off' so that human-applied labels still work with the variable unset. It is an exception because a label requires a trusted human first.
  • Every repository variable that steers this automation is tabulated in the ai-development docs guide, including its unset default. Add new ones there in the same PR. Origin: that table documented AI_LOOP_ENABLED backwards until Copilot caught it — the gates are == 'true', so unset means disabled.

4. Fail closed on verdicts, fail open on budgets

Both directions are deliberate and the split is the point:

  • A verdict (is this item malicious?) fails closed — a crashed, missing, unparsable, or forged sentinel flags rather than passes.
  • A budget or counter fails open — if the daily-limit read errors, the item is simply not scanned. A counter problem must not be able to wedge every incoming issue.

The cost of the second is real and should stay written down: downstream, an unscanned item is indistinguishable from a clean one.

5. Secret masking scrubs logs, not comment bodies

Anything a session writes into a comment is published verbatim. Never build a comment body out of model output without a deterministic sanitizer between them, and never assume masking will save you — it applies to the job log only.

6. Scope comment edits by marker, never --edit-last

gh ... --edit-last picks the most recent comment by that author, which is not necessarily yours. It could overwrite a historical <!-- ai-triage:dedupe --> marker that auto-close-duplicates.mjs still reads — silently destroying an auto-close candidate. Select by the marker your workflow owns.

More generally: when probing for a machine comment, match on marker + (bestaxbot OR a Bot-type author). Never probe one specific login. bestaxbot is a machine User account, not a Bot-type app, so a type == 'Bot' test alone misses it and a login test alone breaks the next time the identity changes. A probe that grants trust rather than finding a comment pins the identity instead, and isDeepReviewAuthor in scripts/review-converged.mjs says why.

7. Fork PRs never run with secrets

Use plain pull_request, never pull_request_target, plus an explicit head-repo guard (#312).

8. Sender exclusions belong on every comment-triggered workflow

contains() matches a raw substring, so an @mention reproduced anywhere in a body — including inside a quoted block or a code fence — is enough to re-trigger a write-capable session. Require sender.type == 'User' and sender.login != 'bestaxbot', and forbid the session from writing the trigger string at all.

9. Logic worth testing does not belong in YAML

Shell embedded in a workflow step cannot be unit-tested without extracting it first. Put non-trivial parsing in scripts/*.mjs with a test sibling — root pnpm test runs every scripts/*.test.mjs. Three extractions exist. The two from #454: the scan-verdict parser (scripts/parse-scan-verdict.mjs, called by ai-scan.yml) and the publish sanitizer (scripts/sanitize-repro-draft.mjs, called by claude-repro.yml) — their test siblings pin the fail-closed matrix and the byte behavior of the shell they replaced, so edit script and tests together. Then #457's triage renderer/publisher (scripts/render-triage-comment.mjs, called twice by ai-triage.yml — once per mode), which is the largest and the one to read first: it validates a model-authored payload, renders the comment from renderer-owned constants, and upserts it by marker. Its test sibling runs the real auto-close-duplicates.mjs consumer over rendered output, so the two cannot drift. Smaller instances of the same shape remain inline (the exec-file sentinel checks in ai-triage.yml, claude-review.yml, and claude-repro.yml's author job); when one of those next needs an edit, extract it and reuse parse-scan-verdict.mjs's exported helpers rather than growing the YAML.

Extraction has a bootstrap gap: a new script flags its own introducing PR. A workflow runs the YAML from the PR merge ref, but the jobs here deliberately check out the default branch (never PR head code). A script the PR adds therefore does not exist in the workspace its own run reads: node exits MODULE_NOT_FOUND, the caller takes its fallback, and a fail-closed verdict path labels the item on infrastructure grounds without the verdict ever being read. Origin: #570 applied needs-security-review to itself this way, which then gated claude-repro, claude-fix, @claude and @bestaxbot on that PR.

Do not close this by checking out the PR head — running PR-authored code in a job holding a credential is the thing that pin prevents, and trading it away for a cosmetic label is a far worse deal than the flag. Treat the gap as expected cost instead: say in the PR description that the flag is self-inflicted, and clear the label by hand once the script is on the default branch. The same applies to moving or renaming a script a workflow already calls — the rename lands in the YAML a run before it lands in the tree that run checks out.

10. harden-runner on new jobs starts at block, and every block job asserts it

New jobs ship with egress-policy: block. An existing live job may be introduced at audit first, because flipping straight to block risks breaking it if the action's runtime egress needs an un-allow-listed host — but that is a temporary state that owes a follow-up issue, not a resting place.

Block jobs also set disable-sudo-and-containers: true, unless a step needs sudo or Docker. A job that leaves it off says which step needs it, in the comment above its harden-runner. What the setting and its check do and do not prove is under "What disable-sudo-and-containers and its check prove", below.

Every block-mode job must carry the assertion step (placement rules below the snippet):

- name: Assert egress policy is enforced
  run: |
    set -uo pipefail
    [ -e /home/agent/agent.json ] || { echo "::error::harden-runner installed no agent"; exit 1; }
    jq -e '.egress_policy == "block"' /home/agent/agent.json || exit 1
    # In a job that sets disable-sudo-and-containers, this line and the two in the loop:
    jq -e '.disable_sudo_and_containers == true' /home/agent/agent.json || exit 1
    for _ in $(seq 1 30); do
      if grep -q '^Initialized' /home/agent/agent.status 2>/dev/null; then
        if sudo -n true 2>/dev/null; then echo "::error::sudo still works"; exit 1; fi
        if [ -w /var/run/docker.sock ]; then echo "::error::docker.sock is writable"; exit 1; fi
        exit 0
      fi
      sleep 1
    done
    exit 1

Copy the full version from a block job that matches the new one on disable-sudo-and-containers, which scripts/harden-runner-sudo.test.mjs holds together. The shapes above are each load-bearing:

  • The -e check comes first because harden-runner has deliberate paths that install nothing and still exit 0 (a StepSecurity outage, the skip-harden-runner repo property, a container or slim runner). Without it a vendor outage fails every block job at once, with a bare jq: could not open file. It still fails, because a job holding a credential must not run unprotected, but it says why. Do not write down how many block jobs that is: an earlier version of this line did, and it was wrong before the ai-scan/ai-triage job splits. Derive it, and grep for the key rather than the string — grep -rn "^ *egress-policy: block$". A plain grep egress-policy: block counts one more: deploy-worker.yml names the policy in a comment, which is the same way a node -e in a comment misclassifies a job below.
  • The status check polls rather than testing once. The pre-step waits only for the file to exist and gives up after ~9s, while the agent resolves every allow-listed host before writing its status, so a cold resolver or a long list can leave it absent or empty at this point.
  • ^Initialized, not -qx. writeStatus appends without a trailing newline, so a second status would concatenate onto the same line and an exact-line match would stop matching.
  • The disable-sudo-and-containers lines are explained under "What disable-sudo-and-containers and its check prove", below.

Placement. The default is immediately after harden-runner, and most jobs use it. Two things override it, both derived from one rule:

A fail-closed check must not be able to consume, skip, or precede the thing that keeps the fail-closed path reachable.

  • After any gate an always()-guarded fail-closed step depends on, and before any step that spends a metered budget. ai-scan's gate is the live example, and it needs both: putting the assertion first would abort before run is set and skip the labeler; letting the same step read and charge the budget would spend a slot on every run that failed to prove enforcement, and AI_SCAN_DAILY_LIMIT such failures in a UTC day (a StepSecurity outage reaches every incoming item at once) set run=false, which skips that same labeler. So: decide → assert → charge. That job also records enforcement=failed before exiting, because label deliberately fails OPEN on ordinary gate trouble and must fail CLOSED on this one — see its comment.
  • After the write, when the job is the fail-closed path. ai-scan's label and ai-triage's cleanup exist to guarantee a label is applied and removed respectively; a check that can fail ahead of that call defeats the guarantee outright, which is also why their harden-runner steps are continue-on-error. Assert afterwards under always(): the write has happened, and a job that ran unprotected still goes red. These two are the only assertions in the repo placed after their job's work — if you add a third, say why here.

Both exceptions were previously "resolved" by simply leaving the assertion off those jobs. Do not go back to that: an unasserted block job is indistinguishable from an enforcing one, which is the whole failure #487 was.

A related trap, one step earlier: harden-runner is step 0, so if it fails the job can die before emitting anything. In ai-scan that skipped the fail-closed labeler, because its guard reads needs.gate.outputs.run. The fix is continue-on-error: true on that harden-runner so the gate step still emits its outputs, with the assertion immediately after to fail the job and set enforcement=failed. Non-blocking there is not "unprotected and ignored": the job still goes red, it just gets to record why first. Whenever a downstream always() job keys off an upstream job's outputs rather than only its result, check what happens if step 0 dies.

That fix covers RUNTIME failure only, and the distinction is not pedantic. Action resolution and download happen during job initialization, before any step is added to the job. A failure there fails the job with no step ever running, so continue-on-error — a step property — never applies and the assertion never executes. Do not describe the flag as covering "download" or "bootstrap" failures; an earlier revision of this section did, and it was wrong.

Residual, stated rather than papered over. If harden-runner cannot be prepared at all, the item goes unscanned and unflagged, and no guard can fix it: every job that would carry the flag pins the same action, so label fails initialization too. This is vendor-availability shaped rather than attacker-reachable — the pin is a SHA served from GitHub's own CDN — but it is a genuine hole in the fail-closed path and it should be named when the posture is described.

Worth tracing the whole matrix when touching any of this, because these paths look alike and only some should flag:

What fails in ai-scanlabel runs?Correct outcome
harden-runner at runtime (gate)yes, via enforcement=failedflag — nothing was enforcing
harden-runner preparationno — label cannot initialize eitherresidual above; nothing reaches it
the assertion (gate or scan)yesflag — enforcement was not proven
the scan jobyes (result is failure, not skipped)flag — empty verdict hits the enum default
the gate's budget readnono flag — counter trouble fails open (rule 4)
budget spent, or bot authornono flag — nothing was scanned and nothing is wrong

Both lines are required; either alone is a false pass. agent.json holds the policy the pre-step decided, serialized after every policy decision, so it catches a downgrade — which is what #487 was. agent.status is written only once the firewall rules are actually installed, and it is the line that catches the other failure: agent.json is written before the agent starts, so if the agent fails to come up, the pre-step logs timed out, prints the agent log and still exits 0, leaving block in the config and no firewall.

Note the actor there, because it is easy to state wrongly: the pre-step never reverts anything — it contains no revert call at all (grep -c RevertChanges src/setup.ts → 0). Every revert is the agent's; the pre-step only makes one visible by printing agent.log. When debugging a red assertion, Reverted changes is in the agent's output, not harden-runner's.

What the pair does NOT prove is that the policy stays armed. In step-security/agent, the sequence is writeStatus("Initialized") in agent.go and then a serve loop whose error branch calls RevertChanges. So any runtime error after initialization tears the firewall down while agent.status still reads Initialized and agent.json still reads block — both assertion lines pass, nothing is enforcing. Separately, refreshDNSEntries re-resolves every 30s and on failure only logs failed to insert new ipaddress in firewall, so an allow-listed host whose IPs rotate can quietly stop being reachable.

Two further false-pass modes are known and not covered, both latent for this repo but worth knowing before anyone leans harder on the check:

  • A private repo or fork without an active StepSecurity subscription. The agent re-decides the policy in-process (if config.Private && !isActive → audit) and writes Initialized outside every policy branch, while agent.json — written by the pre-step — still reads block. Both lines pass, audit rules are installed. bestax is public, so this does not bite here; a private mirror would inherit a green check asserting the opposite.
  • denied-endpoints, new in v2.21.0. Set beside allowed-endpoints it is dropped with a core.info; set alone on a non-enterprise org the tier gate drops it too, leaving block with an empty allow list and an empty deny list behind a core.warning. Both files still read as armed. We do not use the input; do not adopt it without re-reading this rule.

Take the assertion for exactly what it is: proof that enforcement was armed at that step, not a guarantee for the life of a 30-60 minute session. There is no step-level check for the latter — if a session behaves oddly, read the StepSecurity run report for Reverted changes rather than trusting the green step.

Do not "fix" this by adding a liveness check, and this note exists because review has now proposed it twice. Asserting systemctl is-active agent would only cover the fraction of a second between the agent initializing and this step running; the revert that matters happens minutes later, mid-session, where no step-level check reaches. In exchange it would hard-code an internal unit name and bind every block job to the TLS-path binary selection described in rule 1 — so a StepSecurity outage would turn all of them red at once. The gap is real and is stated here deliberately rather than papered over with a check that does not close it.

That second failure is not theoretical. The PR that introduced this assertion shipped with only the jq line and passed green on a job where the firewall had reverted. Which leads to the sharpest operational rule here:

An unresolvable host in allowed-endpoints disables the entire policy. harden-runner's agent aborts and reverts when it cannot resolve an allow-listed domain. A dead entry is not inert — statsig.anthropic.com had gone NXDOMAIN, and its presence silently turned the firewall off. Before adding a host, resolve it; when a run reports Reverted changes or timed out, suspect the allowlist before anything else.

Where the state actually stands, since "which jobs enforce" was mis-stated repeatedly during issue #487 and the distinction is load-bearing. Scope: this covers the AI/automation workflows and the jobs adjacent to them, per job, not every job in the directory — the ordinary build jobs (ci.yml, deploy.yml, test-deploy.yml, visual-regression.yml, story-screenshots.yml, scorecard.yml, dependency-review.yml) carry no harden-runner and are out of scope here.

  • Enforcing and asserted — every ai-scan job (gate, scan, label), every ai-triage job (gate, triage, publish, cleanup), claude-repro (author only), deploy-worker (deploy), supply-chain (consumer-sbom and sign-sbom), security-txt-expiry (check), create-vite-drift (check), auto-close-duplicates (auto-close), claude (claude), claude-implement (implement), bestaxbot-reply (respond), claude-review (review), claude-pr-loop (fix and verify), mcp-registry (publish), codemod-registry (validate and publish), review-converged (sync), skills-publish (relay, generate and publish), templates-publish (generate and publish). The command below lists them, and grep -rn "disable-sudo-and-containers: true" .github/workflows/ lists the jobs that set disable-sudo-and-containers.
  • Audit, deliberately, pending a measured allowlist — none, as of #578. That issue closed the group by measuring every member instead of guessing for them. They all now use the same allowlist, and the gap between that and what the measurement produced is the part worth keeping: the runs surfaced a strict subset of the application hosts the Claude-session jobs above already used, and nodejs.org appeared in no run at all. It is on the list anyway, and that unmeasured host is why "measured" is not the same as "complete". None of ai-scan/scan, ai-triage/triage or claude-repro/author runs actions/setup-node, so their list never needed it; all six of #578's jobs do run it, and setup-node falls back to nodejs.org/dist when the toolcache misses and the actions/node-versions lookup fails. The audit runs could not surface that, because the toolcache hit every time and an unexercised path leaves no endpoint in any log. Review caught it, not measurement. Keep the group here rather than deleting it: audit remains the correct starting point for an existing live job whose egress has never been observed, and the recipe under this rule is what keeps it a short state rather than an open-ended one. What it is not is a resting place, which is the whole reason #578 existed.
  • No harden-runner at all, and these are two different groups — do not merge them into one "API-only" line, which understates the second:
    • Genuinely API-only — no checkout, no repository code: on-slop, auto-label-claude-prs, close-stale-bestaxbot-prs, stale; claude-repro's prepare and cleanup; claude-pr-loop's sweep, gate and halt; supply-chain's attach-sbom (it moves release artifacts and runs nothing).
    • They check out and EXECUTE repository code — claude-repro's publish (scripts/sanitize-repro-draft.mjs, over attacker-influenced text while holding issues: write), claude-pr-loop's handoff (scripts/handoff-message.mjs), and supply-chain's sbom (installs the monorepo, runs SBOM generators) and verify-provenance (installs published packages, runs verification scripts). Calling any of these API-only — as this list did twice — understates the unmonitored execution and egress surface in the one inventory meant to state it precisely. auto-close-duplicates was the fifth here until it was given an enforced policy; it is now in the first group, and it is the only one that has moved.

Do not maintain any of this by hand; it has now been wrong three times, and each time the error moved a code-executing job into the harmless-looking group. Derive it:

# every job with a harden-runner, and its policy
grep -rl "harden-runner@" .github/workflows/ | sort

# candidate code-executing jobs: a checkout is the tell, since nothing here
# runs repo code without one
grep -rln "actions/checkout@" .github/workflows/ | sort

Then read the jobs — and read them, rather than grepping for node, because a node -e inside a comment is enough to misclassify a job (claude-pr-loop's gate has exactly that and runs no code at all).

The authoritative check is that no block job lacks the assertion; parse the YAML rather than eyeballing it, because the two are in different jobs and sometimes several steps apart.

on-slop and auto-label-claude-prs are the two to look at first if this group is ever worked: both run on pull_request_target — the highest-privilege trigger class here — with write scopes. They hold no model token, which is why they are not in the group above, but they are not "API-only" in the harmless sense either.

claude-repro deserves the emphasis: only its author job is hardened. publish — the job that runs the sanitizer over attacker-influenced text and holds issues: write — has no egress policy at all. Anyone citing "claude-repro enforces egress" is overstating it by every job but author.

Keep this inventory correct when you add a job, and qualify per job rather than per workflow. Its first version omitted bestaxbot-reply and named claude-repro unqualified — a table that quietly misses a model-token job, or implies coverage a workflow only partly has, is worse than no table, for the same reason a comment that overstates its mechanism is.

Citing egress-block as a control is now legitimate for the first group only, and only for what it actually does: it narrows where a step that does not escalate can send data, without bounding it ("What the agent allows beyond the list" says what reaches past a job's list, root included), and it does nothing about what a session does with an allow-listed host (see I1). Widening an allowlist remains a security change under rule 2, and sign-sbom's list is still assembled by reading the actions rather than from a measured run. So are mcp-registry's publish list, codemod-registry's validate and publish lists, skills-publish's relay, generate and publish lists and templates-publish's generate and publish lists, which no run had exercised when they were written. The codemod-registry, skills-publish and templates-publish comments give the reason for each host.

Verify rather than assume, on any run: the assertion step passes, and harden-runner's post-step prints the effective EgressPolicy:.

What disable-sudo-and-containers and its check prove

Each job's comment points here. The setting closes the ways a step gets root past the list ("Root, and containers" under "What the agent allows beyond the list", below), and none of the paths that need no root. Citations are to harden-runner at the pin and to step-security/agent at the version the pin installs.

  • The agent.json line shows that harden-runner took the input: the pre-step writes the field from it (src/setup.ts), and the agent reads it from that file (config.go). Without the line, a dropped input would surface only as "sudo still works". Like the egress_policy line, it shows what the pre-step decided, not what the agent did.
  • The sudo and docker.sock tests wait for Initialized, because the agent closes both before it writes that status (disableSudoAndContainers in sudo.go). They show both were closed when the step ran, not that they stay closed: a later agent revert, which the flag's own Docker purge might set off, puts sudo back along with the firewall (RevertChanges in agent.go), and as with the policy above, no step-level check sees it.
  • They cover sudo and docker.sock, not every way to reach a container. containerd.sock is not tested: on the hosted runners checked so far, the runner user cannot write it whether or not the agent closed it, so a test there would tell us nothing.
  • They depend on the runner image as well as the pin: on how the image grants the runner sudo, and on the modes it gives the sockets. So a runner image change, not only a pin bump, can turn every job that sets the flag red at once. In ai-scan, a failure in gate or scan flags the item, as their other assertion failures do. In supply-chain's sign-sbom it leaves the SBOMs unsigned: attach-sbom still attaches them with a warning, and the release still ships.
  • The agent also purges Docker in the background (uninstallDocker in sudo.go), alongside the job's first steps. It runs whenever the agent starts with the flag, including on runs that stop early and run no repository code, and it holds the dpkg lock while it runs. A step in such a job has no sudo for apt in any case.
  • The post step skips the agent.service journal. harden-runner prints it with sudo journalctl only when neither sudo setting is on (src/cleanup.ts), so a job that sets the flag does not print it. agent.log still prints.

What the agent allows beyond the list

A block job's allowed-endpoints names hosts the job adds to the firewall. Others are added beside them: the agent's, below, and the Actions cache host, which the pre-step may append. The list is not a bound on the job's egress. Everything in this section is about the open-source agent, step-security/agent at the version the pin installs, read from its source. When the pre-step selects the closed-source agent-ebpf instead (rule 1), none of it can be checked from here.

The agent adds these to every block job's list, each on port 443 (addImplicitEndpoints):

  • The meta list. GitHub's actions domains as StepSecurity's /github/meta serves them at startup, minus any ending in githubusercontent.com, which is why a job that needs objects.githubusercontent.com still lists it. It has included github.com, *.github.com (which covers api.github.com), ghcr.io, *.githubapp.com, and the exact entries productionresultssa0.blob.core.windows.net up to productionresultssa<N>.blob.core.windows.net, N being wherever the list stops at the time. The agent resolves each of those at startup, as it does any exact entry (below), and admits the address it gets on 443 when the firewall goes up, before the job looks anything up. They are shared storage front-end addresses, so other storage accounts served from them get through too ("Addresses, not names", below). And each one is a further entry that reverts the firewall if it fails to resolve at startup. The list is StepSecurity's, not ours, and can change with no diff here: a run's Post Harden runner log shows what that run fetched (fetched GitHub meta domains), and curl -s https://agent.api.stepsecurity.io/v1/github/meta what it would fetch now.
  • The built-in entries, present even when that fetch fails: codeload.github.com, *.actions.githubusercontent.com, actions-results-receiver-production.githubapp.com and productionresultssa*.blob.core.windows.net. They carry the Actions cache and the runner control-plane hosts "Measuring an allowlist" names, except hosted-compute-*.githubapp.com, which no built-in entry matches: the meta list's *.githubapp.com admits it when the fetch succeeds.
  • StepSecurity's telemetry hosts, agent.api.stepsecurity.io and prod.app-api.stepsecurity.io, unless the job sets disable-telemetry: true (grep -rn disable-telemetry .github/workflows/ lists any that do).

A wildcard entry admits a host when the job looks it up. Every other entry, the job's own included, is resolved at startup, and one that does not resolve reverts the firewall (rule 10), so the meta list is the other place to look when every block job fails its assertion at once. A failed meta fetch narrows the policy rather than opening it, which is why a job still lists a GitHub host it needs. StepSecurity's global blocklist is applied to every entry, the job's own included, so it can drop a host a job lists, also with no diff here (removing globally blocklisted in the agent log).

These are the known paths past a job's list. They are what reading the source has turned up so far, not a complete account, so add to them rather than treating them as one:

  • GitHub, under any account. The meta list's github.com, *.github.com and ghcr.io take writes from any account whose token a step holds, the job's own or one the step brought with it: a gist, a push to a repository, a package. It needs no root, so neither sudo setting below closes it. For a job whose own list names none of those hosts, the meta list is what opens it.
  • Any storage account whose name starts productionresultssa. The built-in wildcard matches on prefix and suffix alone (matchWildcardDomain), and Azure account names run to 24 characters, so an account anyone can create matches it and takes uploads. That is a way out of every block job here, and it is older than the meta list.
  • Root, and containers. In a job that leaves disable-sudo-and-containers off, any step can sudo, and root can change the firewall rules themselves. disable-sudo takes away less than its name says: it empties the runner's sudoers file (sudo.go) and leaves the Docker socket open to the runner user, which is in the docker group, so a step can still get root in a container (docker run --privileged --net=host) and rewrite the rules from there. A container on a user-defined Docker network needs no sudo to get out: its traffic arrives from that network's bridge, and the agent's DOCKER-USER rules match only docker0 (firewall.go). disable-sudo-and-containers also closes the Docker and containerd sockets, and it is the only setting that shuts both. grep -rn disable-sudo .github/workflows/ lists any job that sets either.
  • Public DNS over HTTPS. The firewall (addBlockRules in the agent's firewall.go) accepts TCP 443 to 8.8.8.8, 8.8.4.4 and 1.1.1.1 from the agent's own user, which is root, so a step that can get root, through sudo or a host-network container, can carry data out in DNS queries and leave the rules as they are.
  • Addresses, not names. The rules admit IP addresses, so a host served from the same address as an admitted one, such as another storage account behind the same front end, is reachable too.
  • Connections already open. ESTABLISHED,RELATED traffic is accepted, so a connection open before the firewall went up stays usable.
  • Azure platform and private addresses. 168.63.129.16, 169.254.169.254 and the private ranges are accepted with no lookup (addBlockRules).
  • The Actions cache host. harden-runner's pre-step appends the cache's storage host to the job's list when it can look the host up (Adding cacheHost in its log).

A comment says a host is not listed; it never says a job cannot reach a host. Where a step can get root, no host is out of reach, and the paths that need no root still carry data out. A comment may give the reason a job leaves a host off its list and point here. It does not restate what the agent adds.

Measuring an allowlist

An audit-mode run's endpoints are in the run log, not only the StepSecurity web UI — the Post Harden runner step dumps the agent log, which records every DNS resolution and connection. So this is a command rather than a transcription exercise, and it is the answer to "where does the list come from" wherever this rule says a list must be measured:

gh run view <run-id> --log \
  | grep -oE 'domain: [a-z0-9.-]+\., pid' \
  | sed 's/domain: //; s/\., pid//' | sort -u

Two things about reading its output, both learned assembling the #578 lists:

  • Runner infrastructure appears in that log and need not be allow-listed. results-receiver.actions.githubusercontent.com, productionresultssa<N>.blob.core.windows.net, run-actions-<N>-azure-*.actions.githubusercontent.com and hosted-compute-*.githubapp.com are the runner talking to its own control plane. The agent's built-in entries admit the first three without a listing ("What the agent allows beyond the list"). The meta list's *.githubapp.com admits the last when the meta fetch succeeds, and one run passed with it not admitted at all (run 33286967625, below), which suggests the runner can do without it but does not prove it ("One run is not a measurement"). Leave them off a list. The blob host has no one name a list could carry, since N varies between runs. An exact entry for one of the others is not free, though: the agent resolves every exact entry at startup, and one that does not resolve then reverts the firewall (rule 10), so an entry the job never needed can still turn its policy off. Run 33221210633's log shows auto-close-duplicates at block connecting to api.github.com and github.com and nothing else. That shows the job did not use these hosts, not that leaving them off works for a job that does. Run 33286967625, below, is the evidence for leaving them out.

    What leaving them out costs was read from a run, not assumed. Run 33286967625 ran claude-review at block on harden-runner v2.21.0, whose agent had the built-in entries and no meta list. Its setup-node step restored the pnpm cache through results-receiver and productionresultssa6 before pnpm install --frozen-lockfile completed. Neither was on the job's list, and the pre-step had added a different blob host. So the built-in entries carry the cache on their own. The same run's agent log reads domain not allowed for a hosted-compute-*.githubapp.com host, and the job passed, which suggests the runner can do without that host and does not prove it. After a pin bump, check the built-in entries are still in addImplicitEndpoints before leaning on that run. Worth stating plainly because the first revision of this bullet predicted a cache miss and called it an accepted cost. That was a guess, it was wrong, and the run is what showed it. If you catch yourself writing down what a policy will cost, go and read a run instead.

  • One run is not a measurement. Vendor telemetry samples, so a host can be absent from a run and present in the next — telemetry.vercel.com (turbo) showed up exactly once across the six #578 runs. Prefer denying telemetry at the source (CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC, TURBO_TELEMETRY_DISABLED) over allow-listing a host you would rather not talk to; that is the standing call here, and it also keeps the next audit report clean for whoever reads it.

Then resolve every host before it goes in the list, per the rule above.

When the agent drops a job's first connections

On some hosted runners the agent drops a step's connection to a host the job allows. The agent admitted one address for the host and the step's own DNS lookup returned another, so the agent log reads ip address dropped and the step gets connection refused. It shows at the start of a job, and a later re-resolution (refreshDNSEntries, above) can admit the address. The upstream report is step-security/harden-runner#708, and #1258 has this repo's runs.

The cost here was a misread. The checks that re-verify the triggering user's live role read a failed call as role none, so a dropped run logged that the user lacked access and skipped its work as if they were untrusted. Each check now calls lookup_role. Its waits add up to 95 seconds and each call is capped at 10 seconds, so at worst it takes 155 seconds before it fails the step with an ::error:: that says the API call failed. scripts/role-lookup.test.mjs computes that worst case from the function, and a step that calls it needs a timeout longer than that.

  • An answer from the API, read and none included, is handled as before.
  • A 404, which the endpoint returns for a renamed or deleted account, reads as none.
  • A rate limit (403 or 429) is retried like a dropped connection.
  • Any other 4xx fails the step at once, since a retry won't change it.

scripts/role-lookup.test.mjs runs each check against a stub gh, holds the copies of lookup_role equal, and checks each step's timeout against the worst case. The copies stay inline: each one runs before any checkout, and rule 1 keeps security steps in the workflow.

In the deep review the steps after the role check read the API the same way. The dedupe step's read of the reviews is the job's first call on an ai-loop opened event, since the role check calls nothing there. The focus step reads the comments and looks up each steer author's role. A failure there stops the run rather than picking a review mode. A missing steer, or an author misread as untrusted, can pick the wrong one, such as a verify pass where a full review was asked for.

What it does not cover:

  • A later step's host can be dropped the same way: a checkout of github.com, the npm registry, the model API. The retries cover only the GitHub API reads above.
  • On those runners a job can also sit in the runner's post-job cleanup, every step finished, until its timeout-minutes ends it. No step timeout reaches that. Where the job's own timeout is long, the role lookup, the deep review's dedupe and focus steps, and the checkout carry short step timeouts, so that a step which hangs fails on its own. The job timeouts are unchanged.

Review checklist for a workflow change

  • Does any allowlist grow? If yes, name the credential in that job and justify each entry.
  • Does a model session gain the ability to execute, fetch, or reach the network?
  • Does model or issue text reach a comment body? Through what sanitizer?
  • Is the posting identity GITHUB_TOKEN (inert) or a PAT (re-triggering)?
  • New repository variable? Gate is == 'on'-shaped, and the docs table lists its unset default.
  • Action SHAs match the repo-wide pin, and the version comment is truthful — run both greps in rule 1, including the repo-wide one that covers every action rather than the one you thought to type.
  • New block job carries the effective-policy assertion (rule 10), placed per that rule's placement section; security-relevant steps stay in the workflow rather than moving behind a local composite action (rule 1 — pins under .github/actions/ are fine and one exists, so this is about per-job review, not pin location).
  • New block job sets disable-sudo-and-containers and checks it in that assertion, or says above its harden-runner which step needs sudo or Docker (rule 10; scripts/harden-runner-sudo.test.mjs checks both).
  • New verdict path fails closed; new counter path fails open.
  • Security comments claim exactly what the mechanism delivers — no more.

That last item is not padding. Three separate review rounds on #361 flagged comments that overstated their mechanism: a flag described as blocking "every AI entry point" when third-party reviewers never see it, a session called "read-only" when its token was write-scoped, and an encoding check described as a proof when it is a backstop. A comment that overstates its control is worse than no comment, because the next reader stops checking.

How to be exact: claim less

Exact means claiming less, not explaining more. The way to make a security comment true is to delete the clause that overstates it, not to add the qualifiers that would make it true.

  • A security comment names the control and, when the control is a third party's, cites where it lives, in one line: # matched against the run actor by check-human-actor in anthropics/claude-code-action at the pinned SHA. Do not restate the third party's internals; the restatement rots the moment the pin moves, and nobody re-reads it then.
  • A count lives in a command, never in a sentence. Write the grep that produces it (rule 10's egress-policy: block grep is the model). A hand-maintained number is wrong within months, and this file has been caught doing exactly that.
  • Evidence (dates, tallies, measurements) goes on the issue and is linked. A comment says what the mechanism is; the issue says how we know. A run id may stand in a workflow comment as the receipt for the flip it justified, and nowhere in a guide.
  • One home per fact. A mechanism is explained once, where it is configured; every other mention is a pointer (# same list as fix; rationale there). Two copies are two things to keep equal, and a reviewer will find the day they differ.
  • Claim the case in front of you. "Every", "only", "always" and "exactly one" are one counterexample from false; "the review arm" survives, "every arm" does not.
  • Never cite a line number in prose. Lines move on the next edit; a step id, a job name, a rule number, or a heading does not.
  • A PR body says what changed and why. State an invariant as the command that checks it (grep -c the pin), not as today's count of what it found.

The fragile-prose conformance check enforces the count, run-id, and line-reference rules; mark a deliberate exception with bestax:count-ok on the same line and say why.

Where the rest is documented

  • Design and operation of the AI loop, and the repository-variable table — the ai-development guide in docs/docs/guides/getting-started/.
  • User-facing security posture — SECURITY.md.
  • Release and publish pipeline — ci.yml, plus VERSIONING.md for the semantic-release contract. Note that main is protected by a repository ruleset whose only automation bypass is the release GitHub App, and its token is minted after install and build so repo-owned build code can never reach it. bestax-mcp's MCP Registry listing is published from each release by mcp-registry.yml, which is kept out of ci.yml on purpose: npm's trusted publisher names ci.yml, and the header of mcp-registry.yml says why the third-party publisher binary must not run under it.
  • Codemod Registry. codemod-registry.yml publishes bestax-migrate's wrapper package by manual dispatch. It is kept out of ci.yml for the reason its header gives: npm's trusted publisher names ci.yml, and the third-party codemod CLI must not run under it.
  • The bestax plugin. skills-publish.yml pushes the tree scripts/gen-skills-repo.mjs generates to allxsmith/bestax-skills, with a deploy key held in the skills-publish environment, which admits main only. The generator runs in one job and the key sits in another, which runs no repository code. The workflow's header gives the triggers, that split, why a release reaches it through a dispatch, and why the environment has no required reviewer.
  • The templates mirror. templates-publish.yml pushes the tree scripts/gen-templates-repo.mjs generates to allxsmith/bestax-templates, which StackBlitz opens the ported templates from, in the same two-job shape as skills-publish.yml: a deploy key held in the templates-publish environment, which admits main only, and a key job that runs no repository code. It has no release trigger and no relay. Its header points to skills-publish.yml wherever the reasoning is the same.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.