agentleFS
guimatheus92guimatheus92/pr-review/AGENTS.md

pr-review

AGENTS.md3 starsChanged 31 days agoMITView on GitHub

Generic, plugin-based PR review tool for GitHub, Azure DevOps, and GitLab, packaged as a plugin for Copilot CLI or Claude Code. Orchestrates parallel review passes — each one a skill applied by a generic agent — in a single agent session via the task tool (Copilot) / Task tool (Claude Code), retains every deduplicated finding locally, and posts every publication-eligible finding as a resolvable inline review thread (all findings by default). The bundle at dist/cli.cjs is the single-file distribution artifact.…

  • Reads credentials
  • Commits and pushes
  • Where files are headingNew AGENTS.md files have more rules about keeping secrets out of the code, and fewer about code style.

    Of the AGENTS.md files we read, the 33,315 first seen in the last six months, against the older ones: the kind of rule that gained most share, and the kind that lost most. About files like this in general, not this one, and an estimate, since a kind is a pattern in the words.

  • In this repository4 other files it gives its agents.

The file

We split this AGENTS.md into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Every reconciliation test asserts assertNoDuplicateComments: no path may leave two comments at the same location with the same text.

    line 70

≈ “Common” is an estimate: at least 1 in 5 of the 42,283 AGENTS.md files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
AGENTS.md

pr-review

Generic, plugin-based PR review tool for GitHub, Azure DevOps, and GitLab, packaged as a plugin for Copilot CLI or Claude Code. Orchestrates parallel review passes — each one a skill applied by a generic agent — in a single agent session via the task tool (Copilot) / Task tool (Claude Code), retains every deduplicated finding locally, and posts every publication-eligible finding as a resolvable inline review thread (all findings by default).

Build & test

npm run build          # tsc + esbuild → dist/cli.cjs
npm run test           # node scripts/test.mjs → node --test over tests/**/*.test.ts (990 tests, ~15s — on Windows run it from Git Bash: PowerShell resolves `bash` to WSL and fails `tests/command-finder.test.ts`)
npm run test:mod       # claude plugin test mods/live → the live mod's 35 kit tests, offline, ~8 s (needs Claude Code >= 2.1.287 on PATH)
npm run validate:mod   # claude plugin validate → the module's hooks:/calls: lines; the calls line is its read-only contract
npm run build:watch    # tsc watch (re-run `npm run bundle` for esbuild)

The bundle at dist/cli.cjs is the single-file distribution artifact. The slash command (commands/pr-review.md) finds it via $CLAUDE_PLUGIN_ROOT/dist/cli.cjs under Claude Code (falling back to ~/.copilot/installed-plugins/), locates the checkout whose git origin matches the PR URL (current dir, its subdirectories, then its siblings; a primary worktree beats a linked one), prints a project-skill count that approximates the loader's rule, and runs node "$CLI" review $ARGUMENTS.

Architecture

Two-layer model: slash command → Node CLI → single agent session (Copilot CLI or Claude Code) dispatching review passes as generic agents (subagent_type/agent_type general-purpose) via task() / Task().

  • src/cli.ts — commander entry, subcommand routing
  • src/commands/review.ts — gather → selection → planned delivery → complete-only dedupe/post. Schema-v1 --resume validates HMAC authority, PR identity, execution-mode transition, and hashes before targeted recovery (a live PR must still be open only to publish); legacy consolidated output keeps replay support, while legacy Phase 1 is diagnostic-only.
  • src/commands/status.ts / src/commands/detach.ts — authenticated live/recoverable/terminal status plus detached execution. Exit 21 means targeted recovery is possible; exit 22 is terminal.
  • src/dispatch/single-session.ts — materializes context/pass/companion files and an immutable plan, emits deterministic named dispatch-only task calls, adopts uniquely identified missing Copilot sidecars, promotes attempt outputs, performs one automatic selective recovery, assembles Phase 1/final output in Node, and runs a direct conditional verifier.
  • src/dispatch/delivery.ts / src/dispatch/runtime-events.ts / src/util/atomic-json.ts / src/util/control-auth.ts / src/util/finalization-lease.ts — strict Finding[] delivery, bounded top-level Copilot JSONL parsing, missing-only create-once adoption, atomic create-only canonicals, authenticated started/completed/spawn-rejected attempt lifecycles, requested/resolved model and composed runtime-error evidence, attempt ceilings, digests, non-mutating control reads, Windows transaction-safe state, HMAC authority, and one recoverer/finalizer/poster per run.
  • src/dispatch/runtime.ts — runtime selection (resolveRuntime, runtimeSpawnArgs, taskCall, normalizeModel); --runtime copilot|claude|auto (default auto: probes PATH, copilot first). Planned sessions add the materialized run dir while denying posting paths. Copilot removes shell aliases from the visible tool surface, requests non-streaming JSON output, and runs with COPILOT_PLUGIN_DIR_ONLY=true (1 is ignored by Copilot CLI 1.0.84-3); Claude keeps its explicit tool allowlist. MCP is denied at the PROCESS level, not only the tool level — denying the mcp__* tools alone still lets every server boot (a cmd.exe + conhost + npx + node each on win32, every console window leaking) only to be unreachable. MCP_PROCESS_DENIAL: Record<Runtime, string> holds the per-runtime switch, so a new runtime fails to compile until it declares one. The two are NOT symmetric: claude's --strict-mcp-config is categorical (every config source ignored, and no --mcp-config is passed, so the run-dir .mcp.json is inventory/provenance only — no runtime loads it); copilot's --disable-builtin-mcps covers built-ins and is completed one name at a time by --disable-mcp-server, so its reach is bounded by discoverMcpCapabilities.
  • src/util/progress.ts / src/util/posted-marker.ts — the phase/heartbeat feed and posting idempotency guard. Schema-v1 decisions use authenticated authority; posted.marker in the run dir is diagnostic only.
  • mods/live/hooks/register.js — the Claude Code mod (hooks module), declared by the hooks field of .claude-plugin/plugin.json and deliberately NOT a root hooks/ dir: claude plugin test <dir> takes the mod's folder and runs every *.test.ts beneath it, so the repo root would drag the node:test suite in (observed: 60+ the file did not load). Copilot's root plugin.json never names it, and Claude Code older than 2.1.287 ignores it. It is a READER of the run dir — progress.ndjson, reviewer-progress.ndjson (the per-reviewer timeline; the mod is its first reader in production), dispatch-plan.json, delivery-state.json, passes.json, error.txt — ticked once a second (the band redraws every tick) and re-read every second tick, each file only on a size/stamp change, never status (which reaps). A run that died without a terminal phase (error.txt, newer than the last progress line) settles as failed with the pr-review status <id> hint; a file that exists but cannot be read is named on the band, never mistaken for a --context-only preview. It attaches only to a validated id (one path segment: no separator, .., leading ./- or control character — an Azure DevOps id may hold a space) that exists under the runs root: the banner a --detach launch prints at the start of stdout (past the key: value lines the launch block prints first — the model composes that block from commands/pr-review.md, so their names, order and count vary: a live session added cli: …, and a preamble pinned to two names missed the launch), a --resume <id> typed into the Bash tool, a status <id> poll when nothing live is attached, or /pr-review-live — never a dir: line or anything inside a foreground summary, because finding bodies are model output about PR content (a forged dir: would have pointed $.fs.stat at a UNC path). The slash command sends node "$CLI" review … --detach with $CLI unexpanded, so the launch is keyed on the banner, not on the command text; mods/live/tests/live.test.ts runs that literal block, and tests/mod-surface.test.ts pins it to commands/pr-review.md and the banner to src/cli.ts. The run dir is written by the CLI under the user's own account and is writable by the reviewer session itself, so the mod shows what it finds there and decides nothing from it — status and the summary stay the authoritative readings. It draws the boxed band above the prompt (title and status, a bar with one stretch of cells per reviewer coloured by state, its legend, the counts, the sources — the PR label shortens below 96 and 64 columns, and under 7 rows of room the legend and the sources go), the pr-review-live pane (reviewers grouped under their source, one row each: state, time, short name, match kind) and the spinner suffix, registers /pr-review-live [run-id | off], strips control and Unicode format characters from every name and detail it shows, logs its own failures to the debug file only ($.ui.log … { to: 'debug' }), and stays inert when session.start says isInteractive: false, because pr-review's own claude -p reviewer sessions load the user's plugins (only Copilot is isolated, INV-CTX-06). Its surface is pinned read-only by tests/mod-surface.test.ts, which also refuses a rewritten or mutated Bash event in the observer, any $ alias, and a forbidden call on any receiver.
  • src/dispatch/codex.ts — optional read-only Codex sibling; strict attempt-scoped Finding[] output and crash-safe attempt accounting (opt out: --no-codex)
  • src/dispatch/line-snap.ts — snaps finding line numbers to the nearest valid diff line before posting
  • src/dispatch/lockfile-digest.ts — the only code that knows the package-lock.json format: reduces the two sides gather read (via PrProvider.readFileAt) to a LockfileDigest and renders ## Lockfile Digest; pure, never fetches (INV-FETCH-04's exception)
  • src/dispatch/call-sites.ts — the ## Call sites section: names the declarations a diff touches and finds their references with one read-only git grep at the PR head (else local HEAD, named as such), bounded, never fetching (INV-CTX-07)
  • src/providers/github.ts / azuredevops.ts / gitlab.ts — PR data fetchers + comment posters (GitHub inline comments go out as one review batch — ONE attempt, runPost owns retry because only it can reconcile — with per-comment fallback; GitLab posts per-discussion via plain fetch). Each one delivers a COMPLETE file list or says so: GitHub carries the PR's changed_files as PrMetadata.changedFileCount (its pulls/:n/files stops at 3000 entries silently) and fetches no whole-PR text diff (pulls.get with the diff media type 406s above 300 files, so it is unusable exactly on the PRs that would need it); Azure DevOps pages iteration changes to completion ($top=2000, nextSkip, a cursor that does not advance throws), drops folder entries before they can count toward the file guard, and synthesizes each patch from two whole-file getItem calls — which is why it, alone of the three, honours the content-narrowing opts of fetchChangedFiles; GitLab maps changes_count ("N+" = changedFileListTruncated, because /diffs serves exactly the capped set)
  • src/dispatch/parsers.ts — JSON / bracketed-markdown / section-header output parsers
  • src/dedupe.ts — Jaccard token similarity, strict/loose/off modes
  • src/config.ts — 5-level config merge (flags > env > repo yaml > global yaml > defaults)
  • src/util/retry.ts — retry/backoff helper for transient posting errors
  • src/packs/sync.ts / load.ts — skill packs: clone/pull configured git repos into ~/.pr-review/packs/<name>/ (fail-soft; >30 days without packs sync warns on every review) and load their skill files per include/exclude globs
  • src/stack/linguist.ts / manifests.ts / detect.ts — stack detection: GitHub Linguist languages.yml (auto-downloaded to ~/.pr-review/cache/, refreshed on packs sync) tags changed files by language; dependency names parsed from manifests only when cwd's git origin is the PR's repo (identity via src/providers/identity.ts, so legacy *.visualstudio.com remotes match canonical dev.azure.com); beyond the shallow root scan, changed manifests and manifests beside/above changed files are read, so a deeply nested project contributes its own dependencies; evidence stays categorized (canonical Linguist languages, ecosystems, full dependency names, dependency tokens, dependency groups) rather than collapsing into one tag bag
  • src/dispatch/pass-select.ts — pass selection: project skills (repo/configured/forced) become shared CONTEXT in every pass (skills-project.md, never pass slots) — every match injects, no numeric cap, bodies inlined whole (no byte truncation); passes are ranked by EVIDENCE TIER — specific path glob > manifest-backed dependency/framework token > language-consistent type/manifest glob > exact stack tag (cap MAX_STACK_PASSES=6) — plus EVERY baseline, which is contractual and rides on top of the total pass ceiling. A product guide cannot qualify from a bare language or a generic manifest alone, and product identity tokens must co-occur in ONE dependency group. Up to two installed-plugin passes may be admitted from repository/path/topic evidence. With no pack passes, project skills become the passes; overflow + unmatched + index-mode packs get materialized bodies referenced by digest-bound skills-index.md shards.
  • src/commands/packs.ts — packs list / packs sync / packs add / packs suggest subcommands
  • src/plugins/loader.ts — discovers skills from standard paths (.claude/skills, .claude/rules, .copilot/skills, .github/skills, .github/instructions, .agents/skills, plus the ~/ mirrors) and from configured dirs. Follows directory links (symlinks and NTFS junctions) ONE hop, in every discovery and configured dir: a link the PR added or changed (its path, or any parent of it, is in the diff) is refused BEFORE anything behind it is read and named as degraded coverage; a link met inside a linked directory is not followed. Under a skills root a subdirectory is a skill only through <dir>/SKILL.md — a SKILL.md-less subdirectory is skipped (stderr warning when it holds .md files), flat .md files at the root are still skills, a README.md directory entry is never a skill; rules/ and instructions/ roots recurse as before. Applies rule trust before same-name dedupe, and folds semantically identical mirrors silently while still warning on divergent same-name rules. It is the only walker for repo, personal and configured dirs (packs and installed plugins have their own): walkSkillDirs/loadFromDir in src/plugins/builtin.ts were removed because a second walker meant a second link policy
  • src/plugins/trust.ts — a rule file added or modified by the PR under review is UNTRUSTED input: it is dropped from both authoritative context and the on-demand index, and named as degraded coverage. Checks lexical AND real paths, so a changed later mirror cannot evict its unchanged counterpart; comparisons fold letter case and Unicode (NFC) on EVERY platform — win32-only folding let a PR committing .Agents/skills bypass a macOS reviewer. Trust is by authorship, not location: content whose real path is outside the checkout is trusted when the PR did not author the link that reaches it AND the file is committed and clean in its home git repository (git ls-files + git status; a SKILL.md also needs its whole directory clean) — the same gate applies to every rule outside the checkout (linked, configured or personal), and a repository git cannot read is skipped, never trusted — on Windows git checkout of a PR branch writes THROUGH an NTFS junction into the shared directory, so a planted file would otherwise become a trusted rule for every sibling repo's review. Untracked or modified files there are skipped and named; a directory under no git repository at all is trusted as the reviewer's local configuration (one stderr note per directory reached through a link). Nothing depends on one company's layout: any link, any discovery root, any OS, git or not. In-repo --skill files stay scoped and must live inside the checkout; --force-skill is the explicit override
  • src/plugins/installed.ts — host-agnostic discovery of installed plugins from their manifests, in BOTH runtimes: Copilot CLI (~/.copilot/installed-plugins/<marketplace>/<plugin>/) and Claude Code (~/.claude/plugins/cache/<marketplace>/<plugin>/<version>/, whose generic discovery path is read from installed_plugins.json because several versions can sit side by side). It validates active roots selected by the runtime (Copilot's config.json is JSONC; Claude's plugin list --json supplies the enabled installPath, scoped to the reviewed directory by governsCwd in companions.ts, whose manifest identity must match — the version is recorded, never adjudicated). Skills are namespaced <plugin>/<skill>; declared paths and symlinks are constrained to the plugin root. Also builds the MCP capability inventory (capabilities.json) from trusted repo config, user config, and plugin manifests
  • src/providers/identity.ts — canonical checkout/PR identity: canonicalPrAuthority normalizes legacy <org>.visualstudio.com, encoded HTTPS paths, and ssh.dev.azure.com:v3/<org>/<project>/<repo> onto one authority, including the ADO project so a same-name repo in another project can't supply manifests or rules
  • src/util/git.ts — gitTopLevel(); the pipeline resolves the checkout root once so a run from a subdirectory still finds repo config and rules. gitOut() / gitZ() run read-only git and THROW on failure (a failed read is never an empty result); rule provenance and the gather's truncated-list completion both go through them
  • src/plugins/companions.ts — detects installed companion plugins (pr-review-toolkit, code-review); copilot via copilot plugin list, claude via claude plugin list --json, which supplies each entry's enabled status, exact installPath, and scope. That list is not cwd-filtered — run from one checkout it still reports every project-scoped install on the machine — so governsCwd drops entries whose projectPath is not the directory under review (containment, foldPath-folded: pr-review may run from a subdirectory, and the same project appeared twice differing only by drive-letter case). Reading the list as already-scoped turned an unrelated project's installs into conflicting active installations and silently dropped all 7 dispatches — and so did two records of the SAME project on different versions (Claude Code writes one per spelling, C:\… and c:\…, and updates only one of them), for 36 reviews in a week (#54). There is no conflict gate any more: every enabled record governing the directory under review becomes a root, the version string is never compared, and agreedDefinition is the integrity check — it compares the whole definition FILE (frontmatter and CRLF normalized, before any criteria extraction), so identical files across roots dispatch and divergent files drop THAT companion only, naming every root with a digest. A record whose installPath has no manifest naming the plugin is named in the Degraded block and ignored while a sibling root dispatches. Each runtime has its own registry — installing under one leaves the other empty, and --runtime auto probes copilot first, so the machine that has them under Claude Code only still dispatches zero companions. Detection failure is unknown, never "not installed"; companionReviewerNames() is the single source of up to 7 planned identities. Supplied active roots are constrained to each plugin, every root must provide agreeing definitions, and an unterminated header or shell/network/acquisition/fan-out/posting directive fails closed. Frontmatter is stripped lexically because its metadata is not execution authority — and so is a ## When to invoke section, which addresses the HOST that dispatches the agent, not the agent: the official code-reviewer.md says "Spawn this agent on the freshly written files" there, and scanning it as criteria refused the companion over prose describing its own call sites. Removed, not exempted, so nothing inside it reaches the dispatched agent either. A companion that cannot be materialized is degraded coverage, never fatal — materializeCompanionBriefs returns failures instead of throwing, and runReview drops that id from installedCompanions BEFORE the planned roster is fixed (planning it and not dispatching it is what INV-DEL-01 reads as missing output, which would turn the degradation back into an exit 2). One bad plugin .md used to abort a run holding nine healthy passes and five working companions. One companion-brief-*.md per identity is hash-bound into the plan. Every companion runs as general-purpose; code-review keeps only its materialized-diff criteria, never its native gh/posting workflow — and those criteria have TWO marketplace wordings (CODE_REVIEW_SHAPES): the claude-code-plugins HIGH SIGNAL block, and the claude-plugins-official 0–100 rubric with its 80 cutoff, which is what Claude Code's default marketplace and a team enabledPlugins list install (all 14 cached official versions). Matching only the first refused every official install for a month; tests/fixtures/code-review-command/ holds both real files byte for byte, and a body matching neither is refused naming both. The Degraded line states the cost (0 of 7 companion reviewers dispatched, N reviewer(s) lost). formatWarning takes the runtime because the install command is not portable: slash commands under claude, copilot plugin … shell commands under copilot (it has no /plugin), from marketplace anthropics/claude-code — the larger anthropics/claude-plugins-official carries the same plugins but Copilot's marketplace.json schema rejects it

Key conventions

The guarantees themselves — what always holds, why, and what verifies it — live in INVARIANTS.md, one addressable block per ID. This section carries the implementation detail a contributor needs while editing these files, and cites the IDs rather than restating them. Changing what the product guarantees means changing INVARIANTS.md first, in its own commit. IDs are append-only: retire, never rename or reuse. pr-review verify <run-id> grades a finished run against the whole list.

  • No repo pollution. (INV-FETCH-03) All run artifacts go to ~/.pr-review/runs/<id>/. Never write files to the user's working directory — and that includes the checkout's .git: pr-review never runs git fetch or git checkout and never writes a ref in the reviewer's repository (git credentials are not the API token, a fetch inside a detached child can take minutes, and it would import branch-authored objects). The truncated-list completion in src/commands/gather.ts only reads objects already present and otherwise fails with the exact fetch command for the user to run.
  • Clean output. (INV-POST-03) Posted comments contain only the finding body — no severity prefix, no bot chrome, no footer. (The --- separators live only in the local pr-review-summary.md that renderSummary writes; nothing separates posted comments, because each is its own thread.)
  • Full retention, eligible-only inline posting. (INV-POST-01, INV-POST-02, INV-POST-07) On a publish run every eligible finding lands as a resolvable review thread (GitHub review comments, ADO threads, GitLab discussions) — never a top-level issue comment. Lines outside the diff are snapped to the nearest valid diff line. On GitHub and GitLab, findings that can't anchor where they point (file outside the diff, or no location) are re-anchored to the first valid diff line with the original file:line kept in the body (reanchor in src/commands/post.ts); on Azure DevOps threads are posted at the reported file:line as-is (ADO threads are not limited to diff lines) and a finding with no location lands as a resolvable PR-level thread — a thread ADO rejects is reported as an error, never dropped. skipped exists only for --dry-run; below-threshold findings are suppressed, not skipped. Never reintroduce an issues.createComment fallback.
  • Publication policy is Node-owned. (INV-POST-01, INV-POST-07, INV-OUT-01, INV-OUT-02) --publish-min-severity is CLI-only on review and post. src/util/severity.ts owns parsing, ordering and partitioning; --fail-on uses the same ordering but always evaluates all finalFindings. Complete delivery and full dedupe precede durable artifact persistence and publication filtering. Never inject the threshold into prompts, change verifier gating, or hide suppressed findings from the summary. New dispatch plans are schema v2 and authenticate execution.publishMinSeverity (NIT when omitted); v1 reads remain publish-all without rewriting their signed bytes. Resume rejects an explicitly conflicting threshold before recovery, including under --force-post. Retention is reconstructed with dedupeFindings against the original gather, while eligible pending publication is separate. Authenticated markers accumulate exact confirmed file:line:body keys; effective finalization mode records dry-run promotion without mutating the original plan. verify recomputes retention and eligibility instead of trusting artifact counts. Standalone post never rewrites its input or treats its metadata as policy authority.
  • A failed write is not proof that nothing was written. (INV-POST-04, INV-POST-05) POSTing a comment is not idempotent, so a 5xx/timeout means unknown: the server can commit and lose the response. The rules that follow from that, all in src/commands/post.ts:
    • Providers make ONE attempt and throw — postLineComment and postBatchComments, on all three providers. withRetry is for reads and other idempotent calls only. Retry lives in runPost, because only runPost can reconcile first.
    • Read the PR back before deciding anything — before a retry, before the per-comment fallback, and before reporting a count (readLanded/claim).
    • Unknown ≠ empty. readLanded returns null when the read fails, and a null must never be treated as "nothing landed": the outage that 504s a write is the one that fails the read. On null the run reports and stops rather than re-issuing — re-issuing is how 3 findings became 15 live comments in one reviewed scenario.
    • Identity is file:line:body, not body. Two findings can legitimately carry the same body (dedupeWithinBatch only folds same-file, ±3 lines), and a body-only match promotes a finding that was never posted — which then fills posted.marker and locks --resume out of recovering it. Comments with no file are never ours (inline-only invariant).
    • Reconciliation is one-way — errors may be promoted to posted, never the reverse. Demoting on a stale read would mark a live comment un-posted and make the next --resume write it twice.
    • Every publish attempt writes authenticated posting state plus the posted.marker mirror, carrying verified. An unverified run fails closed on resume like a corrupt marker. Gating this on posted > 0 is what left the incident run with no guard at all.
    • --resume re-reads the PR before publication (on --dry-run too), unions confirmations rather than overwriting them, and adopts only comments matching an eligible finding it would post. It never removes retained findings or adopts a suppressed finding. A failed re-read aborts a publishing resume unless the operator explicitly uses the existing --force-post escape hatch.
    • Every reconciliation test asserts assertNoDuplicateComments: no path may leave two comments at the same location with the same text. This is not theoretical: a 504-after-commit turned 56 findings into 112 comments while the run reported posted 0 / errors 56.
  • An incomplete file list is unknown, never empty. (INV-FETCH-01, INV-FETCH-02) gather.changedFiles feeds the rule-trust gate, .pr-review.yaml trust, the MCP-config gate and manifest reading, and a path missing from it reads as "unchanged" — i.e. trusted. So a provider list is never taken on faith: runGather compares it with the provider's own count (PrMetadata.changedFileCount) and honours changedFileListTruncated; on any mismatch it completes the list from the reviewer's checkout (git diff-tree -r -M -z from the single merge base, plumbing so reviewer diff config cannot reshape it, one hunks-only patch per missing file) when cwdMatchesPr says that checkout is the PR's repository and both SHAs are present, and otherwise throws BEFORE anything is cached (exit 2, error.txt in detached mode) naming the counts and the git fetch to run. A criss-cross history (two merge bases) or a shallow clone is refused too — git and the provider could diff against different ancestors. Cache entries carry changedFilesComplete; an entry without it predates the gate and is refetched once. Providers paginate fetchChangedFiles to completion and set the count whenever the API offers one. This is not theoretical: Azure DevOps was reviewing every >100-file PR on its first 100 files ($top default) from 0.6 through 0.10.
  • The list of paths is complete; the CONTENT is only fetched where it can be read. (INV-FETCH-04) The 500-file guard (MAX_FILES_GUARD, now in src/dispatch/diff-filter.ts beside applyDiffExclusions — gather.ts cannot import review.ts) runs after gather, so everything fetched for a too-large PR is paid for and thrown away: one getItem per added file and two per modified one, each a whole file body — measured live, 501 requests and 5.1 s on a 501-file ADO PR, then "PR is too large". Now 0. patchPolicy answers "can this file's content still reach a pass?" and the two cost sites ask it — AzureDevOpsProvider.fetchChangedFiles before spending two whole-file getItem calls, and completeFromGit before spawning a git diff-tree -p per missing file. Paths are NEVER narrowed, only content; patch-less rows were already normal (binary, pure rename, mode-only, ADO deletions). The guard applies to every runGather caller, so pr-review gather on a >500-file PR returns paths without content — deliberate — while pr-review post passes patchesRequired because it gathers ONLY to build the valid-line map, and an empty map is the 422-on-the-batch failure that call exists to prevent. Two things follow, and they are keyed on different conditions on purpose: a list assembled while withholding content is never cached (restored under wider excludes it would look like a whole diff), and GatherOutput.patchesOmitted — set only when nothing carries a patch — is read by earlyExitGate before the exclusions, because with every patch gone the byte clause sums to zero and waves the run through. The checkout's own diff_excludes reach gather as repoExcludes so a 600-file PR its config trims to 480 stays reviewable; they are branch-authored, so they may shrink the COUNT but never suppress a file — otherwise diff_excludes: ['**/*'] in a PR would deliver a review of its own code with no diff. The one bounded exception: an excluded package-lock.json is read through the provider API at the two commits the provider's diff compares (two reads per lockfile, ten lockfiles per PR, never past the guard) and only its digest — never the content — lands in GatherOutput.lockfileDigests; a cache entry holding a lockfile but no digest is refetched once.
  • A synthesized patch is hunks, not the whole file. (INV-FETCH-04) Azure DevOps has no diff endpoint, so synthesizePatch builds one from two whole file bodies; toHunks frames it with git's 3 lines of context. A hunkless whole-file patch is not wrong — validLinesFromPatch starts its cursor at 0, so the numbers work out — which is exactly why it survived from the first release: it fails silently, by making the cache, the ## Diff block every pass reads, and the 2 MB budget scale with the size of the FILES. It also made every line "in the diff", so the ADO posting behaviour AGENTS.md documents was true only by accident. postingPolicy states it instead: ADO neither snaps nor re-anchors; GitHub and GitLab do both. Use postingShape — runPost applies it and resumeReview and verify both RECOMPUTE it to recognize the run's own comments, so a fourth copy of the rule is a double-post (INV-POST-05) waiting to happen.
  • The CLI is the ONLY writer — dispatched agents never post. (INV-POST-06) Every dispatch prompt (review passes, materialized companions, verifier, the PR-description check, and the orchestrator itself) carries NO_POSTING_DIRECTIVE from src/dispatch/single-session.ts, and a session-context test fails if any dispatch line loses it. This is not theoretical: the official code-review companion's command allows gh pr comment and instructs posting a top-level "### Code review" verdict — a live run posted one (Preco-Pratico/PrecoPratico-Backend#586) before the directive existed. When adding ANY new dispatch path, thread the directive through it.
  • Passes, not reviewers. (INV-CTX-04) Every review pass is ONE skill applied by a generic agent — no reviewer .md is ever dispatched. The only briefs this repo ships are the conditional verifier and internal/pr-description (DESCRIPTION_BRIEF, planned on every dispatched review, removable only with --skip pr-description); neither is a skill or counts toward the pass cap. Pass names are <pack>/<skill> (repo skills keep their plain name). Knowledge comes from skill packs (git repos cloned to ~/.pr-review/packs/<name>/, configured via skill_packs — the one yaml key that REPLACES rather than pushes, so [] disables packs and a repo list overrides the global one entirely) plus the standard tool skill dirs (.claude/skills, .claude/rules, .copilot/skills, .github/skills, .github/instructions, .agents/skills) and any configured dir (--skills-dir, extra_skills_dirs, PR_REVIEW_SKILLS_DIR), selected the same way. src/dispatch/pass-select.ts separates lenses from context: matched project skills inject into EVERY pass as authoritative rules, and the passes are pack stack hits ranked by evidence tier (specific glob > dependency > language-consistent weak glob > tag, cap 6) plus every baseline pointer, which rides on top of the total cap; everything else lands in digest-bound skills-index.md shards for on-demand reading. applies_to / applyTo / paths frontmatter all route the same way; inject_into is deprecated (stderr warning, then ignored).
  • Untrusted input is anything the branch under review authored. (INV-TRUST-01, INV-TRUST-02, INV-TRUST-03) A rule file (.claude/skills, .claude/rules, .github/instructions, …) added or modified by the PR — or reached through a link it added or changed — cannot instruct that PR's own review: src/plugins/trust.ts drops it from BOTH authoritative context and the on-demand index, and the summary names it as degraded coverage. Same rule for .pr-review.yaml (config is re-loaded without the checkout's copy) and for repository MCP config. --skills-dir / extra_skills_dirs / PR_REVIEW_SKILLS_DIR are NOT a bypass: they are selected like repo skill dirs (targeted → scoped rule, untargeted → name+description relevance heuristic, unmatched → on-demand index; origin configured, still applied when cwd is not the PR's repo) and trust-checked — a PR-changed file inside a configured dir is skipped even when the dir comes from an unchanged .pr-review.yaml. 0.10 injected them whole into every pass; observed live: 1.4 MB × 8 passes on a 5-file PR. The ONLY bypass is --force-skill <file|dir> — per run, CLI only. There is deliberately NO yaml or env key for forcing, so a committed .pr-review.yaml can never pre-authorize branch-authored content.
  • A parseable review is not a completed review. (INV-DEL-01, INV-DEL-03, INV-OUT-01) finalizeReview collects operationalFailures and exits 2 for them even when findings parsed fine: a planned companion that delivered no output (or two), a missing/invalid raw-<reviewer>.json sidecar, a post that failed or could not be verified, a failed review prerequisite. Exit 0 means the pipeline completed — without --fail-on it says nothing about the finding count, and the CLI prints how many were retained. Never let a partial delivery render as a clean run. The mirror error is just as real: a resume re-dispatches unresolved reviewers, companions included, but writeCompanionArtifact lives in runReview and never runs there — so companions.json kept the interrupted attempt's verdict and a COMPLETE recovery exited 2 over seven companions whose outputs were in the same run dir. resumedCompanionFailures now reconciles the record against the outputs the resume holds, one-way (INV-POST-04's rule: a recorded failure may be cleared by evidence, never invented from its absence), and writes the cleared verdict back — pr-review verify grades INV-DEL-01 from those same fields, so a stale artifact also fails the audit of a run that behaved correctly.
  • A runtime result can recover only its own missing attempt. (INV-DEL-04) Copilot JSONL is a fallback transport, not a second parser ladder. runtime-events.ts trusts top-level events only, pairs deterministic task names through toolCallId, and accepts only exact valid Finding[]. adoptRuntimeReviewerAttempt is create-only and runs before the existing promotion path. Never replace an invalid attempt file or write a canonical sidecar directly from runtime output.
  • Single Phase-1 dispatch session, Node-owned completion. (INV-DEL-02) All passes dispatch in one runtime process via task() / Task(). Node owns accounting, bounded selective recovery, aggregation, verifier gating, and the complete-only posting gate. The direct verifier and optional Codex sibling are separate sessions by design.
  • Stack-agnostic prompts. (INV-HYG-01) The prompt text this repo still ships (PASS_RULES, VERIFIER_BRIEF, DESCRIPTION_BRIEF, the codex prompt) must never reference specific frameworks — stack knowledge enters a review only through skills. Enforced by a denylist grep in tests/zero-passes.test.ts.
  • Stack, capability, runtime and model discovery are recorded. (INV-CTX-01, INV-CTX-02, INV-CTX-05, INV-CTX-06) Every run writes stack.json, passes.json and capabilities.json — including the resolved host runtime and requested model after normalization. auto is a delegation: when Copilot emits one valid pre-dispatch session.auto_mode_resolved, authenticated delivery-state.json records that root route separately per attempt. It is not per-reviewer served-model evidence. Selective reviewer redispatch reuses the initial route; a schema-v1 Copilot Auto run without it cannot complete or recover rather than inventing an alias. Use --default-model to request a specific model. Finding nothing is a valid result; not recording it is not. Empty stack evidence must carry a note saying why.
  • No window, no orphan. (INV-HYG-04) Every node:child_process call in src/ passes windowsHide: true, and none passes detached (it voids windowsHide on win32) except the --detach launcher. A detached run has no console, so on Windows a console child started without it gets a fresh console that Windows 11 hands to Windows Terminal as an empty window (measured: ~10 per review, one per synchronous git/gh call, each blocked ~3 s by the handoff; eight concurrent reviews froze the desktop). tests/windows-hide.test.ts is the whole guard — INV-HYG-04 is tests-only in verify — so it is strict: the call-site count is pinned (a new site changes it on purpose), every import shape of child_process other than a named import { … } from 'node:child_process' is refused, a spread options object counts only as what its declaration in the same file says, .kill( on any receiver but process is refused, and every spawnCli child must be registered with killOnExit. Each rule has a control that proves it can fail. Children are killed with everything they started (killTree in src/util/spawn.ts): taskkill /T /F on win32, where they run behind cmd.exe; on POSIX SIGKILL to the process and each descendant the process table lists (an unreadable table kills the root only and is reported as not complete). A kill is reported only once the pid is gone (a zombie is gone); a failure always names image, pid and reason — on timeout on stderr plus an error line in progress.ndjson, at exit on fd 2. killOnExit covers exit and SIGINT/SIGTERM/SIGHUP, which Node's default disposition would end without it. A hard kill runs no handler, so status (a dead run.pid, a dispatch plan, and no summary newer than run.pid) and --resume (before it claims run.pid, refusing while a session survives) run reapAndRecord: only inside the runs root, it kills processes whose argv carries the run dir with a flag only this CLI's argv has (--add-dir <dir> plus the MCP-process denial, or Codex's -C <dir> plus --skip-git-repo-check), re-reading each target and run.pid just before the kill, and records killed / survived / scanError under the reap progress phase. By argv, not a recorded pid — measured live, the cmd.exe in front of the runtime died with the CLI and left the runtime unreachable through it. taskkill and powershell run from their absolute System32 path (a bare name is searched in the cwd first, and the cwd is often the PR's checkout), and the process table is read as UTF-8.
  • esbuild bundle. (INV-HYG-02) dist/cli.cjs is a single-file zero-dependency bundle. No npm install needed at the plugin install site.

Testing

Tests use node:test + node:assert. Run with npm run test. Tests are in tests/ mirroring src/ structure. Every test is hermetic: provider tests inject a stub Octokit client, GitApi, or fetch, so none needs a credential or the network. That is also the limit of the suite — nothing in it proves a real Azure DevOps or GitLab call works, which is what npm run acceptance exists for.

The live mod's tests are the one exception to the layout: they live in mods/live/tests/ and run with npm run test:mod (claude plugin test mods/live — offline, no sign-in, but it needs the Claude Code CLI, which CI installs on ubuntu). claude-code/testing imports do not load under node:test, and a kit test placed under tests/ would fail the node suite. The kit has no file access, so the fixtures are a TypeScript module, mods/live/tests/fixtures/run-798.ts: a pruned copy of a real run dir in two snapshots, running and done, regenerated with node scripts/mod-fixtures.mjs <run-dir> <out-file>.

A green test is not proof the behaviour is right — it can be proof the defect was written down twice. Three ways this suite has been green over a live bug, none of them findable by reading the test, all of them found by running the product:

  • The assertion transcribed the current output. tests/runtime.test.ts asserted normalizeModel('copilot', DEFAULT_MODEL) === DEFAULT_MODEL — that the shipped default reaches the CLI verbatim. It did, and Copilot CLI answers Model "claude-opus-4.8" … is not available and exits 1 before dispatching anything, so every reviewer read as failed to deliver. When the assertion is about a value crossing a boundary you do not own, assert what the far side accepts, never what this side happens to send today.
  • The fixture made the comparison impossible to fail. An early INV-TRUST-01 check compared passes.json's source (absolute) against the changed-file list (repo-relative), so no PR-authored skill could ever match and the check PASSed on every input, including the ones it exists to catch. Its test supplied a relative source — a shape the product never writes. A check that compares values from two different producers has to be fed the real shape of both, copied from an actual run dir.
  • The fixture answered its own question. See evals/acceptance/README.md — the control handler in the acceptance fixture carried a doc comment that contradicted its own code, the reviewer flagged it (correctly), and the false-positive assertion failed. A control has to be genuinely correct, not merely intended to be.

The common shape: each one was written by looking at the code instead of at what the code talks to. npm run test cannot tell you a real CLI rejects your model string, a real run dir writes absolute paths, or a real reviewer is right — npm run acceptance and npm run dogfood can.

Run artifacts under ~/.pr-review/runs/<id>/ are part of the contract, not debug spill: passes.json (routing), stack.json (detected stack), companions.json (installed / recognized / planned / completed dispatches), hash-bound companion-brief-*.md, capabilities.json + capability-<pass>.json (MCP inventory and per-pass usage evidence — both runtimes deny MCP at the process level, so the brief instructs passes to record all three arrays empty. A sidecar that reports otherwise raises a named degraded warning listing the fields and servers it claimed — the run still completes, because Node has no view of the session tool surface and so cannot tell a real denial leak from a fabricated one. The brief deliberately keeps that escape hatch so a denial that stopped working can still surface here, and pr-review verify raises the same claim as an INV-CTX-05 FAIL — an audit read after the fact is exactly where an unexplained claim should stop the reader, even when the run itself does not fail on it), attempt state with requested/resolved model and structured-result adoption evidence, raw-<reviewer>.json (one canonical sidecar per pass, companion and the description check), reviewer-progress.ndjson (the per-reviewer timeline — attempt starts, first-seen outputs, promotions, invalidations, recoveries, verifier steps — which the live mod reads and INV-OUT-02 lists), error.txt on any failure, and posted.marker on any publish attempt. --resume, the operational-failure checks and the live mod read them, so a change that stops writing one is a behaviour change.

Common tasks

  • Audit a finished run against the guarantees: pr-review verify [run-id] (or --pr <url>). Read-only; one PASS/FAIL/SKIP row per invariant, always the full list, exit 2 on any FAIL. --offline skips the live PR read, --json for CI.
  • Prove it works on every provider and runtime: npm run acceptance — 3 providers × 2 runtimes against real PRs, posting for real. See evals/acceptance/README.md; the estate is seeded once by npm run acceptance:seed.
  • Add/curate a skill pack: Edit DEFAULT_PACKS in src/config.ts (entry: git source + include/exclude globs, optional ref/mode/baseline), or per-user via skill_packs: in yaml (remember: it replaces the whole list) / pr-review packs add <owner/repo|url> (materializes the defaults first, then appends and clones).
  • Sync packs: pr-review packs sync — clones/pulls every configured pack and refreshes the Linguist cache. Needed once per machine before the eval harness; >30 days without it warns on every review.
  • Run the eval harness: node scripts/eval.mjs [case] — needs npm run build, a real runtime on PATH, and synced packs; fixtures in evals/fixtures/ (diff.patch + expected.yaml, optionally pr_title / pr_description in expected.yaml, a lockfile-digests.json that becomes GatherOutput.lockfileDigests, and a repo/ that is committed with the diff applied as the PR head in a temp git repo the CLI runs inside, so ## Call sites has callers to find). Assertions live in scripts/eval-assertions.mjs: must_find, must_not_find, distinct_findings (one generic finding can't satisfy two expectations), must_dispatch / must_not_dispatch, and stack / dependencies include/exclude. A case may be negative-only — that is how a defective/control pair proves the finding is causal rather than vacuous.
  • Dogfood the local branch (no PR needed): npm run dogfood -- --base origin/main [--include-untracked] — turns the branch/working-tree diff into a synthetic gather under ~/.pr-review/runs/ and drives the real bundled CLI with --from-gather --dry-run. Refuses a stale bundle (npm run build first) and refuses secret-bearing names/content before writing artifacts. Companion coverage is intentionally off for a synthetic gather; exercise it on a real-PR dry-run.
  • Add a provider: Implement PrProvider from src/providers/types.ts, wire in src/providers/index.ts, and teach src/providers/identity.ts its remote forms (checkout identity + cache scoping depend on it).
  • Change config defaults: Edit src/config.ts DEFAULTS object.
  • Change auto-discovery paths: Edit autodiscoveryPaths() in src/config.ts (repo rule dirs and their ~/ mirrors).
  • Force a skill past its declared scope: --force-skill <file|dir> — a file, or every .md under a directory, injected whole into every pass with no scope and no trust check. --skill <file> KEEPS the file's applies_to/applyTo/paths scope, so the bypass has to be asked for — and asked for on the CLI: there is no yaml/env equivalent on purpose.
  • Change dedupe behavior: Edit src/dedupe.ts. Threshold constants are at the top.
  • Change diff exclusions, the file guard or the patch budget: Edit src/dispatch/diff-filter.ts — DEFAULT_EXCLUDES, MAX_FILES_GUARD, MAX_PATCH_BYTES, and patchPolicy, which decides what is worth fetching, all live there together.
  • Change runtime spawn args or model mapping: Edit src/dispatch/runtime.ts.
  • Change Codex reviewer behavior: Edit src/dispatch/codex.ts.
  • Preview pass selection: pr-review review <url> --context-only — writes pr-context.md + per-pass files, prints the ## Stack block (languages, dependencies, notes) and the ## Passes table (Pass | Matched by | Matched on | Source) + index count, exits without spawning the runtime (exit 2 when zero passes on a code PR).
  • Resume a killed run: schema-v1 runs validate authenticated state and redispatch only incomplete coverage; complete runs replay directly, and a complete dry run may be resumed without --dry-run to post what it previewed (an incomplete one may not). Legacy Phase 1 cannot publish. Authenticated posting state prevents duplicate repeat resumes (--force-post overrides posting only, never delivery trust gates).
  • Run in the background / check a run: pr-review review <url> --detach returns a run-id immediately; pr-review status <run-id> shows the live progress feed, or the summary once done. This is how the slash command avoids the host's ~10-min Bash timeout.
  • Check the environment: pr-review doctor — runtimes on PATH, resolved runtime/model, codex + companions, provider auth, skill packs (git on PATH, per-pack state/freshness, Linguist cache), config sources.
  • Change the live mod: edit mods/live/hooks/register.js, keep npm run validate:mod listing only read calls ($.fs.read/stat/exists/list, $.clock.*, $.env.get, $.ui.*, $.command.register), and run npm run test:mod. Develop against a session started in a terminal with claude --plugin-dir <checkout> --debug-file <log>: the module hot-reloads on save, and the log's hooks module pr-review@inline loaded … events: … line proves it loaded from the manifest path. Write each $ call in full and each on('event') as a literal — the validator reads them from the source — and never call status, write a file or start a process (tests/mod-surface.test.ts refuses it). The VS Code chat panel draws no mod: prove a change in a terminal.
  • Cut a release: node scripts/release.mjs <patch|minor|major|x.y.z> — bumps every manifest, verifies no stale version string, rolls CHANGELOG, rebuilds, commits and tags. It stops there: pushing (git push --follow-tags) and creating the GitHub Release are manual, and a pushed tag alone never shows on the /releases page — see CONTRIBUTING.md.
  • Before opening a PR: review the diff, don't just validate it. Build/tests/smokes prove the change works; only a review of the diff catches a doc that promises more than the code delivers, a constant deleted out from under its comment, or a behavior validated on one corpus and shipped product-wide. Dogfood it: push the branch, open the PR, run pr-review review <pr-url> --dry-run against it, and triage before asking for human review (PR #16 and #17 both shipped real defects that only this step caught). When any guarantee wording changes (docs/CHANGELOG), grep every mode the code path has — a guarantee is universal unless the doc scopes it.
  • Editing files from a script: replace anchors, never slice ranges, and never through a shell heredoc. Two ways this repo has lost working code to a "safe" scripted edit. A patch script that rebuilt a file from the text between two markers silently dropped credentialEnv from scripts/acceptance-reset.mjs — everything between the markers that the script did not know about went with it; a one-cell smoke run is the only reason it was caught. And a Git Bash heredoc collapses backslashes, so "\\n" in the payload reaches disk as "\n" — it broke expected.yaml and a fixture twice before the pattern was believed. Write the file with the editor tool, or s.replace(anchor, () => replacement) in a Node script where the replacement is a function (a string replacement re-interprets $&); assert every anchor matched before writing, and re-read what landed.

Maintaining review quality

The review knowledge lives in skill packs, not in this repo. Keeping quality up over time:

  • Packs own the knowledge. The defaults (DEFAULT_PACKS in src/config.ts) point at upstream repos; pr-review packs sync pulls them, and a pack unsynced for >30 days warns on every review. Baseline pointers (each pack's baseline: skill list) break loudly, not silently: a pointer that no longer resolves surfaces as a missingBaseline warning — fix the pointer when upstream renames or removes a file, don't let it rot.
  • A real production miss becomes an eval fixture — as a causal pair. Distill the miss into evals/fixtures/<case>/ (diff.patch + expected.yaml) rather than expanding prompt text, and add a <case>-safe control whose diff is the corrected version and whose expectations are the same regexes under must_not_find. The control moves exactly ONE variable: when the defect lives outside the diff — a false description (pr_description), a lockfile the diff excludes (lockfile-digests.json), a caller that never reads a new field (repo/) — the diff stays byte-identical and that input moves instead (tests/gather-from-patch.test.ts pins this for the shipped pairs). A positive-only fixture passes on a fixture so broken that anything would be flagged; the pair is what proves the finding tracks the defect. Use distinct_findings: true when two separate issues must each be found. Then run node scripts/eval.mjs <case> (needs npm run build, a real runtime, and synced packs).
  • The remaining prompt text stays stack-agnostic. PASS_RULES, VERIFIER_BRIEF, DESCRIPTION_BRIEF, and the codex prompt must never name a framework — tests/zero-passes.test.ts greps a denylist over them, so a hardcoded framework name fails the suite. Stack-specific rules belong in a pack or a repo skill.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.