agentleFS
lohi-ailohi-ai/babysit/.claude/skills/harness-audit/SKILL.md

harness-audit

Skill0 starsChanged 3 days agoMITView on GitHub

Audit AGENTS.md, CLAUDE.md and linked project instructions against repository code, scripts, CI and agent configuration. Use to find stale commands, broken references, conflicting rules or missing verification paths; setup-project owns initial 2build configuration.

  • Reads credentials

26 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use harness-design, project-init and 3 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • VictoriakaeySkill

    harness-design

    Use when designing or improving the action space, observation format, tool boundaries, and evaluation signals of an LLM agent. Covers the five layers of harness design: what tools the agent has, what it sees after acting, how each tool is scoped, how behavior is evaluated, and how to iterate when the agent misbehaves. Prevents the most common harness failures — agents that pick wrong tools, ignore critical information, or loop without progress.

    Also in 3 projects that use this

  • u9401066Skill

    project-init

    Initialize new projects using this template. Triggers: init, new, 新專案, 初始化, create project, 建立專案, bootstrap, scaffold project.

    Also in 3 projects that use this

  • nicolasestremSkill

    brainstorm

    Structured idea generation and solution exploration with systematic analysis.

    Also in 3 projects that use this

  • MicrosoftSkill

    handoff

    Compact the current conversation into a handoff document for another agent to pick up.

    Also in 3 projects that use this

  • GoogleSkill

    tdd

    Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.

    Also in 3 projects that use this

The file

We split this skill into the rules it gives an agent

10 rules. 0 are common, 4 are distinctive, and 6 are very specific to this project.

  • 0

    Common

    Most of the 22 edited versions we read keep these.

  • 4

    Distinctive

    This author's own choices. Worth a look.

  • 6

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  • could be automatic: a setting could enforce it
  1. Distinctive:

    Check whether an agent can discover the right constraints, run the project and verify a change from the instructions actually available to …

    line 7
  2. Distinctive:

    Follow preamble for bootstrap and telemetry, and Auto-Decision Framework for decisions.

    line 11
  3. Distinctive:

    Do not create branches, commit, push, deploy, install tools, rotate credentials or change global settings as part of an audit.

    line 21
  4. Very specific:

    Start at the requested repo/service or file.

    line 24
  5. Very specific:

    Follow links needed to establish an operational claim.

    line 30
  6. Very specific:

    Read git flow only for 2build git checks.

    line 78
  7. Very specific:

    When --local release support is claimed, trace its checks, builds, packaging, publication and deploy commands statically against CI and …

    line 84
  8. Distinctive:

    Check file existence, anchors and script/target definitions first.

    line 102
  9. Very specific:

    Do not eval resolver output, source .env, or run commands copied from docs just to audit them.

    line 110
  10. Very specific:

    If fixes are authorized, patch the owning file, preserve unrelated user edits and existing policy, then recheck affected …

    line 134

“Common” is measured: kept by at least 6 in 10 of the 22 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

harness-audit

Check whether an agent can discover the right constraints, run the project and verify a change from the instructions actually available to it. Findings need repository evidence, not a preferred documentation template.

Follow preamble for bootstrap and telemetry, and Auto-Decision Framework for decisions. Shared refs are filesystem paths beside this skill's directory, so read them by path, not as skill://.

Scope and authority

Audit is read-only by default, apart from normal skill telemetry and a report when requested. An explicit request to fix or optimize the harness authorizes scoped edits; do not turn “check AGENTS.md” into setup, policy changes or a rewrite of every linked doc. No ticket or 2build configuration is required. Do not create branches, commit, push, deploy, install tools, rotate credentials or change global settings as part of an audit.

Start at the requested repo/service or file. Discover root and relevant nested AGENTS.md / CLAUDE.md, imports/symlinks and referenced architecture, design, run/test/deploy/release docs. Also search each service for an unlinked architecture document (ARCHITECT*.md, ARCHITECTURE*.md, docs/architecture*.md); an existing but unlinked one is a reachability finding. Discover the linked release profile, otherwise .babysit/release.md, RELEASE.md or an existing inline release section. Follow links needed to establish an operational claim; exclude dependency/vendor/generated trees unless referenced. Do not traverse every sibling repo or entire monorepo for a service-scoped request.

Record which instructions apply to each checked service, including inherited and local rules. Use the actual harness's loading/override rules when known; do not assume Codex, Claude Code and other hosts load the same files. If that is unknown, report the assumption. Missing one filename is not itself a defect when another entrypoint correctly serves the intended harness.

Evidence checks

First check the project context links contract: AGENTS.md must lead to the project's architecture guidance, and to design/deployment/release guidance when those apply. Verify service coverage and when each document should be read. Distinguish a missing link to existing guidance from missing documentation; report either gap even if there is no file to inspect. Do not require a particular filename or design/deployment docs for projects that do not need them. Apply the owner audit in project docs: per-service architecture coverage, a system document limited to cross-service scope, and every remaining TODO(2build): placeholder listed as an open gap.

For a releasable app, CLI, library or pack, use the release skill's profile contract to audit release coverage. An equivalent linked inline section/runbook is valid; missing RELEASE.md alone is not a finding. Missing release guidance for an actual release surface, an unreachable existing profile or conflicting duplicate owners is a finding. A non-releasable repo can mark this N/A. The profile must say where each release unit deploys, how, and what is verified and observed after deploy; a field the repo contradicts is drift that the next release also corrects, so name the field and its evidence.

Then inspect the following where relevant or present:

SurfaceCompare againstReport when
Instruction entrypoints and linked docsActual paths, link anchors, imports/symlink targets and service ownershipBroken links, circular imports, unreachable required guidance or instructions applying to the wrong service
Run/build/test commandsPackage scripts, lockfiles, Makefile targets, compose, runner config and CI; resolve working directory and prerequisitesCommand missing, wrong runtime/port/cwd, suites excluded from documented checks, required infrastructure omitted
Architecture and operational rulesRepresentative implementation and current configuration at the named pathGuidance contradicts reality, duplicates have drifted, ownership or exceptions are ambiguous
Architecture ownersService directories, nested entrypoints and the system/service split in project docsA service without an owner; system doc not linking a service doc; system doc holding service internals, schema detail, deploy env/commands, content conventions, roadmaps, resolved history or account IDs that another owner holds (name the target owner)
Placeholdersrg -n 'TODO\(2build\):' across entrypoints and linked ownersEach remaining placeholder (file:line, missing fact) as an open gap; a placeholder the repository already answers is drift
Base branchgit flow resolution against actual refsResolves to a branch the repo does not release from; a leftover .babysit/git-flow.yaml is ignored (note, not defect)
.babysit/qa.yaml and local overridesEffective local environment, service commands, credential variable names and meaningful flowsWrong/missing target, hosted-only QA, no failure/empty/validation case, override changes target or credential source unexpectedly
Release profile (RELEASE.md, .babysit/release.md or linked inline owner)Version/mirror/lockfile sources, tags/changelog, CI/script stage ownership, package registries, current deploy/rollback runbook, observation rules and CLI installation pathsWrong repo/component/version identity, stale commands, missing required stage or unjustified N/A, ambiguous artifact/SHA, unverifiable hypercare bounds, missing local update verification or profile unreachable from entrypoints
Ignore rules and examplesTracked filenames, .gitignore, .env.example, configuration referencesSecret-bearing local files tracked or not ignored, committed credential literals, names pointing at the wrong environment
Browser automation hygieneBrowser/QA bullets in the entrypoint against /bbs:browse § Browser hygiene; scripts, CI or docs that launch ChromeEntrypoint tells agents to use the installed Chrome, pkill/kill/close --all as cleanup, or an ad-hoc session name; a browser-using project with no close-the-browser instruction
Related repo/workspace pointersOnly relevant entries in ~/.babysit/config.yaml, named local paths and RELATED_*_REPO fallbackReferenced repo unavailable, roles disagree, machine paths committed, project-local .babysit/config.yaml mistaken for the registry

Read git flow only for 2build git checks. Missing .babysit files or workspace registration in a repo that does not use those features is not a finding. For CLI/library repos, validate their actual check path instead of requiring a browser URL. An unavailable submodule or machine-local repo is an evidence gap, not proof its documentation is wrong.

When --local release support is claimed, trace its checks, builds, packaging, publication and deploy commands statically against CI and helpers. Report missing CI-equivalent checks, remote build/Actions dependencies disguised as local execution, absent guard/restoration for relevant automatic workflows, or instructions that bypass landing/authentication policy. Check that original workflow states survive failures/resumes and already published artifacts are verified before retry. Distinguish an intentional unsupported local path from a false claim of support; missing live permissions are unverified facts.

For instruction quality, keep project-specific constraints and failure lessons that change decisions. Flag generic filler, repeated policy, stale examples or large always-loaded detail only when they obscure a real task or drift from an owner. Recommend one owning file plus precise pointers; do not delete rules merely to hit a line/token budget. If implementation violates a deliberate invariant, report that conflict rather than rewriting the invariant to match broken code. Separate factual drift from policy decisions that need an owner.

Validate without executing the instructions blindly

Check file existence, anchors and script/target definitions first. Parse YAML with an available parser; CLI output is additional evidence, not a complete schema validator. Inspect these read-only resolvers when 2build is configured:

bbs secrets qa default-env
bbs secrets qa probe --env <local-name>

Choose the actual local name from config; a simple top-level target is local. Do not eval resolver output, source .env, or run commands copied from docs just to audit them. Inspect tracked/ignored status without dumping secrets. Capture and redact command output before reporting if URLs or legacy configs contain credentials. Record file/key and risk, never secret values.

Resolve target/cwd/command consistency statically by default. A safe read-only probe of an already running local service can strengthen evidence; report it separately from configured intent. Execute application checks or start services only when runtime verification is requested, after inspecting side effects and prerequisites. Missing tools/services become NOT RUN with a reason. Never run migration, rollback, seed, deployment or destructive prepare hooks to “verify” documentation. Auditing release guidance also does not authorize version bumps, tags, publication, workflow suspension/dispatch or local CLI upgrades, including npm pack --dry-run whose lifecycle hooks can mutate/build. A health response proves reachability, not auth or user flows; use browse / qa for requested UI execution.

Findings, optional fixes and output

For each actionable finding give severity, file:line, the conflicting evidence, impact on a concrete agent task, and the smallest correction. Prioritize unsafe commands/secret exposure, instructions that prevent execution or verification, then drift and maintainability. Separate confirmed defects, suggestions and unverified claims. Do not invent a numerical harness score.

If fixes are authorized, patch the owning file, preserve unrelated user edits and existing policy, then recheck affected links/config/commands. Reuse setup-project when the requested fix is missing 2build configuration or a missing architecture/release owner (skeleton with TODO(2build): placeholders). Moving over-scoped content to its owner is a fix only when requested; otherwise report it. Do not silently switch release model, credential names or command semantics as a documentation cleanup.

Return the report directly unless the caller requests a file or supplies a handoff destination. Include inspected scope, prioritized findings, checks performed, changes made and gaps. Findings do not prevent completing an audit; inability to inspect its primary target does.

STATUS: DONE | DONE_WITH_CONCERNS | NEEDS_CONTEXT | BLOCKED
VERDICT: CLEAN | FINDINGS | INCOMPLETE
SCOPE: <entrypoints/services and linked files actually inspected>
VERIFY: <static checks, optional runtime evidence, NOT RUN and why>
GAPS: <TODO(2build) count per file, or none>
NEXT: <smallest correction, missing evidence, or none>

CLEAN covers only the inspected scope and requires no unresolved material verification gaps. INCOMPLETE means required evidence is missing; it is never an application QA PASS. Report known findings even when the audit is incomplete.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.