agentleFS
on-paron-par/software-factory/.claude/skills/verify/SKILL.md

verify

Skill6 starsChanged 2 days agoMITView on GitHub

Run and fix the software-factory verification gate (scripts/verify.sh) from the repo root — format, build, config-JSON and oxlint-plugin guards, typecheck, lint, knip, tests with coverage thresholds and the coverage ratchet, the stub eval, and the shell-script tests. Use before every commit or push, when asked to "run verify", "run the checks", "make CI green", when a CI `ci` job fails, when the coverage ratchet complains, or when running the pipeline integration tests.

189 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use genie-hacks, workfly and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • automagik-devSkill

    genie-hacks

    Browse, search, and contribute community hacks — real-world patterns for provider switching, teams, skills, hooks, cost optimization, and more.

    Also in 3 projects that use this

  • automagik-devSkill

    workfly

    Discover a procedure and build its saved workflow — dynamic discovery, drafted script, adversarial verification, landed in the catalog.

    Also in 3 projects that use this

  • TheKREATORSSkill

    instinct-export

    Export instincts from project/global scope to a file

    Also in 3 projects that use this

  • TheKREATORSSkill

    instinct-import

    Import instincts from file or URL into project/global scope

    Also in 3 projects that use this

  • AviadPSkill

    risk-assess

    Assess the risk and blast radius of a proposed code change before implementing it. Evaluates CI impact, framework-wide vs localized scope, rollback path, and identifies blindspots. Use when user says "what's the risk", "is this safe", "assess the impact", "risk assess", "blast radius", or before applying framework-wide or CI-affecting changes. Do NOT use for code quality review (use code-quality) or plan validation (use design-validator).

    Also in 3 projects that use this

  • diego-tobalinaSkill

    build-fix

    Fix build and compilation errors systematically. Run when build fails.

    Also in 6 projects that use this

  • +2 more from automagik-dev
  • +1 more from TheKREATORS

The file

We split this skill into the rules it gives an agent

2 rules. 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • 0

    Common

    Most of the 168 edited versions we read keep these.

  • 1

    Distinctive

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Run every command from the repo root.

    line 8
  2. Very specific:

    No lintable file may exceed 500 lines.

    line 72

“Common” is measured: kept by at least 6 in 10 of the 168 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Verify

Run every command from the repo root. Tests only resolve from there.

Which path to run

WhenCommand
Before every commit (fast loop)bash scripts/verify.sh --no-e2e
Before you push (adds the coverage ratchet)bash scripts/verify.sh
Pipeline integration suites (on demand)npm run test:integration

The factory's own CHECK phase runs bash scripts/verify.sh --no-e2e in the build worktree (packages/core/src/checkers/index.ts).

What scripts/verify.sh runs, in order

  1. npm ci
  2. npm run format:check (Prettier, includes Markdown)
  3. npm run build (tsc -b)
    • then bash scripts/cli-help-baseline.sh — factory --help must match scripts/fixtures/factory-help.txt
  4. npm run adr:index -- --check — the ADR index table in docs/adr/README.md must match the ADR files
  5. bash scripts/check-config-json.sh — no JSON may exist under packages/config/src or dist (#716)
  6. bash scripts/check-oxlint-plugin-version.sh — @oxlint/plugins must match oxlint exactly (#795)
  7. npm run typecheck
  8. npm run lint — Oxlint, type-aware, --deny-warnings
  9. npm run knip — dead code and unused dependencies (default and --production runs) against the shrink-only knip-baseline.json; npm run knip:raw shows the human-readable report
  10. npm run duplication — jscpd clones against duplication-baseline.json; fails only when clones rise, a drop prints suggested lower numbers and passes, percentage is informational
  11. npm run loc-budget — per-package line budget against loc-budget.json
  12. Tests:
    • full path: npm run test (Vitest with coverage thresholds), then npm run coverage-ratchet
    • --no-e2e: npm run test (Vitest with coverage thresholds), no ratchet
    • after the tests, npm run slowest-tests prints the 5 slowest files and npm run test-budget compares the test-step wall-clock time against TEST_BUDGET_SECONDS in scripts/test-budget.ts (450s; override for one run with FACTORY_TEST_BUDGET_SECONDS). It prints test-budget: PASS within budget, a warning plus the slowest files when over locally, and fails verify in CI (CI set, e.g. CI=true) with test-budget: FAIL plus the slowest files
  13. npm run eval -- --stub
  14. Shell-script tests: auto-merge-sweep, filter-green-prs, repo-merge-settings, repo-about, repo-issue-optics, ruleset-copilot-review, launchd/install-sweep-plist, publish-workspaces (guards the publish list scripts/publish-workspaces.txt against the workspace graph)

CI (.github/workflows/ci.yml) runs the same checks as the full path, split into parallel jobs: static (format, build, guards, typecheck, lint, knip, Duplication) and test (1/3)…test (3/3) (Vitest shards that write blob reports with raw coverage and no thresholds, via FACTORY_COVERAGE_SHARD=1). The required ci job fails unless all of them succeeded, then merges the shard reports (vitest --merge-reports) and enforces the coverage thresholds, the ratchet, and the stub eval on the merged result. A red ci names the failing job; read that job's log. CI also runs scripts/quickstart-smoke.sh in a separate job. It does not run the install-sweep-plist test.

The pre-commit hook (lefthook.yml) auto-fixes staged files with oxlint --fix and prettier --write. A direct mkdtemp(tmpdir()) in a test fails npm run lint via anti-slop/no-direct-tmpdir-mkdtemp; the fix is createTempDirs (per-file allowances live in tmpdir-mkdtemp-baseline.json and can only shrink).

toHaveBeenCalled*, toBeCalled*, toHaveReturned* and toHaveResolved* matchers, and reads of fn.mock.calls|results|lastCall|instances|contexts|invocationCallOrder, fail npm run lint via anti-slop/no-call-assertions; the fix is to assert observable behaviour (return value, resulting state, output, emitted event). To shrink call-assertions-baseline.json, remove assertions from a file, then lower its entry to the new count, or delete the entry at 0; lint reports a stale allowance until you do. Never add or raise an entry; the CI guard fails it. The rule cannot be disabled inline.

Comments baseline

anti-slop/no-comments fails npm run lint on any comment except justified lint overrides, rule notes in oxlint.config.ts and toolchain directives; to lower an entry in comments-baseline.json, delete comments then set the entry to the new count (or delete it at 0), never add or raise one (the CI guard fails it), and the rule cannot be disabled inline.

anti-slop/package-boundaries fails npm run lint on a forbidden package edge: config importing a workspace package or declaring runtime dependencies, core importing cli or tui, a relative import leaving its package, a deep import of another package's src/, another package's test-support/, or @on-par/factory-core/internal and /testing outside the places ADR-0004 allows. The fix is the one each message names (import the published name, move the code, or use the public root). The rule is never disabled inline.

LOC budget

npm run loc-budget compares each package's TypeScript line counts with loc-budget.json. Every .ts, .tsx, .mts and .cts file under packages/<pkg>/ counts, *.test.* files as tests and the rest as source; blank lines and comments count.

  • Over budget: delete or reuse existing code before adding more.
  • Under budget: lower the number to the printed count in the same PR so the trimmed code stays gone.
  • Deliberate growth: raise the number in loc-budget.json; a human applies the loc-budget:override label, then re-runs the static job.
  • Labels applied by a bot or a FACTORY_BOT_LOGINS identity do not count.
  • Agents never apply the label and never raise a number themselves; they ask a human in the PR body or park comment.

Coverage gate and ratchet

  • Thresholds live in vitest.config.ts: one global set plus one set per package. Read the file for the current numbers.
  • A drop below any threshold fails npm run test. Add tests. Never lower a threshold.
  • npm run coverage-ratchet fails when measured coverage beats a threshold by more than 2 points (DEFAULT_RATCHET_SLACK). Raise that threshold to the suggested value it prints, in the same PR.
  • packages/core/src/types/** and packages/core/src/test-support/** are excluded from coverage.

Max-lines guard

No lintable file may exceed 500 lines. Tests count, and so do blank lines and comments. The limit is set in oxlint.config.ts and enforced by npm run lint (step 8). scripts/max-lines-config.test.ts enforces it again in the test step.

max-lines-baseline.json grandfathers files that were already over the limit. It is a flat JSON object that maps a repo-relative path to the line count recorded when the file was grandfathered. That count is the file's own ceiling, so the file may shrink but never grow. Never add a new entry and never raise a count. CI enforces this; see Human override.

New file over 500 lines

Lint fails with File has too many lines, and the test step fails because the file is not baselined. Split the file by responsibility into colocated modules of 500 lines or fewer. Keep .js import extensions, and split the colocated tests the same way. Do not add the file to the baseline.

Baseline file grew

Lint fails for the file, and the test step reports it grew past its recorded ceiling. Move code out, or split the file, until it is at or under its recorded count. Never raise the number.

Baseline file shrank or was deleted

The test step names the file in a remove it from max-lines-baseline.json message. Delete its line from max-lines-baseline.json so the default 500 ceiling applies again.

Human override

The CI static job's "Max-lines baseline guard" step diffs max-lines-baseline.json against the PR base. It fails on a raised count or a new entry.

Only a human may approve one, by applying the max-lines:override label to the PR. Then re-run the failed static job (or push), because a label change does not re-trigger CI. A label applied by a bot or app identity, or by a login in the FACTORY_BOT_LOGINS repo variable, does not count. Lowering or removing entries never needs the label.

Agents must never apply the label, never raise a count themselves, and never compact unrelated code to fit. They split the file, or ask a human for the label in the PR body or park comment.

Local verify.sh does not enforce this.

Integration tests

*.integration.test.ts files (real git worktrees, whole plan → build → check → ship cycles) are excluded from both verify paths. They run nightly in .github/workflows/nightly-integration.yml (07:00 UTC). Run them with npm run test:integration when you touch packages/core/src/phases/ or worktree handling.

npm run mutation (Stryker) runs on demand only. It is not part of scripts/verify.sh and CI does not run it (#805).

Fixing common failures

Failing stepFix
cli-help-baselinenpm run build && bash scripts/cli-help-baseline.sh --update, commit the fixture.
format:checknpm run format, then re-run.
adr:index --checknpm run adr:index, commit the README change. If it names an unparseable ADR, fix that file's # ADR-NNNN: heading or - Status: line.
check-config-jsonMove the data into packages/config/src/defaults.ts as typed TS. Delete the JSON file.
check-oxlint-plugin…Bump oxlint and @oxlint/plugins to the same version in package.json, then npm install.
lintFix the code. Warnings fail the build too.
lint / test: max-linesSee Max-lines guard.
knipFix the finding. If you fixed a baselined one, delete its key from knip-baseline.json. Never add keys for new findings.
duplicationClones rose: deduplicate via existing test or harness helpers. A drop passes and prints suggested lower numbers; percentage is informational.
loc-budgetSee LOC budget.
test thresholdAdd tests for the uncovered lines the report names.
coverage-ratchetRaise the named threshold in vitest.config.ts to the suggested value.
eval --stubSee the run-evals skill.
Tests pass in root, fail inside a packageExpected. Run from the repo root. Do not change packages/core/package.json's test script.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.