agentleFS
glrodaszglrodasz/dotfiles/skills/harvest-reviews/SKILL.md

harvest-reviews

Skill20 starsChanged 18 days agoMITView on GitHub

Harvest human code-review comments from recently merged PRs and distill them into the team-reviewer rules — fetch threads with the code they targeted and the fix that followed, classify them, and merge them into a per-computer, gitignored store. Use when the user runs /harvest-reviews, optionally with a window ("90d", "since 2026-10-01") and PR filters (repo, org, label, author, path, reviewer).

1 other project uses a version of this.

Adopted by

Most recently taken up by

The file

We split this skill into the rules it gives an agent

10 rules. About 0 are common, 2 are distinctive, and 8 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 8

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Run from the target repo's root: node ${CLAUDE_SKILL_DIR}/scripts/fetch-reviews.mjs <since> <flags>.

    line 53
  2. Very specific:

    If newItems is 0, report the query and stop — nothing to learn.

    line 61
  3. Very specific:

    Read the pending items in batches: node ${CLAUDE_SKILL_DIR}/scripts/digest.mjs --offset 0 --limit 40, then follow the Next batch line until …

    line 65
  4. Very specific:

    Read rules.json (it may not exist yet) so you map items to existing rule ids before inventing new ones.

    line 68
  5. Very specific:

    Write every item, including noise, so the run can be marked processed.

    line 86
  6. Distinctive:

    Never delete a rule because this batch lacks evidence for it — staleness (unseen for 6 months) and contradictions are computed and shown by …

    line 111
  7. Very specific:

    Run: node ${CLAUDE_SKILL_DIR}/scripts/render.mjs --commit.

    line 116
  8. Very specific:

    If it says NOT committed, classify the missing ids and rerun.

    line 121
  9. Very specific:

    Squash/rebase workflows: follow-up commits are matched by commit date after the first comment and by file path.

    line 143
  10. Distinctive:

    Reset: delete the store folder to start over.

    line 150

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Harvest Reviews

Learn how this team reviews code from what human reviewers actually wrote on merged PRs, and keep the team-reviewer skill's rules current. Rules are paraphrased and carry no reviewer names; the per-example files keep @reviewer and project because they never leave this computer. Nothing is posted to GitHub, and nothing outside the store is written.

Inputs

  • Arguments: $ARGUMENTS
  • Requires: gh (authenticated) and Node 18+; no npm packages
  • Store: ${CLAUDE_SKILL_DIR}/../team-reviewer/.review-harvest/ — gitignored in the dotfiles repo, per computer. Override with REVIEW_HARVEST_DIR. Layout:
    • state.json — watermark per filter set, seen item ids, run log, pending fetch
    • raw/<date>.jsonl — fetched PRs, one per line (script output, never edit)
    • cache/ — fetched PRs (kept a day) and commit patches (kept 7 days), so a crashed or repeated run resumes instead of refetching
    • classified/<date>.jsonl — your classifications, one item per line
    • rules.json — source of truth for rule text (you edit this)
    • index.md, rules.md, false-positives.md, top-rules.md, examples/<category>.md — generated by render.mjs; never edit by hand
  • Default scope: merged PRs of the current repo, since the last run with the same filters, else 30 days.

Translate the arguments into fetch-reviews.mjs flags. Ask only when a filter is genuinely ambiguous:

Asked forFlag
a window: 14d, 2w, 3m, 2026-10-01positional since
another repo / a whole org--repo owner/name / --repo org:acme (repeatable)
labels, PR author, base branch, title words, team--search "label:frontend -author:me base:main"
an area of the code--path src/payments/ or --path "**/*.tsx" (repeatable)
one reviewer's style--reviewer <login> (repeatable)
ignore someone (another bot, an AI reviewer)--exclude-user <login>
re-read threads already processed--rescan

Phase 1 — Fetch

  1. Run from the target repo's root: node ${CLAUDE_SKILL_DIR}/scripts/fetch-reviews.mjs <since> <flags>.
  2. It prints a JSON summary (query used, PR and item counts, raw file). Bots, empty approvals/LGTM, the PR author's own top-level comments and threads the author opened on their own code are already excluded, as are items processed in earlier runs.
  3. Network errors and GitHub 5xx/rate limits are retried. If the script still dies or reports failedPRs > 0, rerun the same command: cached PRs are skipped, and the watermark never moves past a PR that failed.
  4. If newItems is 0, report the query and stop — nothing to learn.

Phase 2 — Classify

  1. Read the pending items in batches: node ${CLAUDE_SKILL_DIR}/scripts/digest.mjs --offset 0 --limit 40, then follow the Next batch line until done.
  2. Read rules.json (it may not exist yet) so you map items to existing rule ids before inventing new ones.
  3. For every item write one line to classified/<today>.jsonl in the store (append if it exists):
    {"id":"PRRT_…","category":"tests","outcome":"changed","scope":"rule","ruleId":"tests-cover-error-path","stance":"supports"}
    • category: correctness security perf architecture naming tests a11y types dx style
    • outcome: changed (a follow-up commit or suggestion addressed it), pushed-back (the author argued and the reviewer conceded or the thread ended unresolved in their favour), ignored (no reply, no change)
    • scope: rule (would apply to any PR in this codebase) or one-off (only makes sense for this change: a typo, a product question, "can you rebase")
    • stance: supports the rule in ruleId; contradicts it (a reviewer or the team explicitly accepted the opposite); false-positive (a reviewer said not to flag something — ruleId points to a kind: "false-positive" entry); none for one-offs and noise, with ruleId: null
  4. Write every item, including noise, so the run can be marked processed. Judge outcome from the follow-up patch and the replies, not from isResolved alone.

Phase 3 — Merge into rules.json

rules.json is {"rules": [ … ]}; each entry:

{"id":"tests-cover-error-path","kind":"rule","category":"tests",
 "rule":"Test the failure path of every new branch, not only the happy path.",
 "why":"Regressions here shipped twice because only success was asserted.",
 "before":"it('saves', …)","after":"it('saves', …)\nit('surfaces a 500 as a toast', …)"}
  1. Strengthen existing rules: point matching items at their ruleId. Rewrite rule or why only when the new threads state it more precisely; replace before/after when a newer, clearer example exists.
  2. Add a rule for a recurring point. Ids are <category>-<kebab-slug>, stable once created. render.mjs admits it only with ≥2 supporting items or 1 that caused a change; below that it stays a silent candidate — add it anyway so evidence accumulates.
  3. Paraphrase the reviewers into an imperative rule plus the reason. No names, no quotes, no repo-specific jargon the next reader would not understand.
  4. before/after is one short hunk (≤6 lines each) taken from the newest example, trimmed to the essence.
  5. Things reviewers said not to flag become "kind": "false-positive" entries.
  6. Never delete a rule because this batch lacks evidence for it — staleness (unseen for 6 months) and contradictions are computed and shown by the renderer.

Phase 4 — Render and verify

  1. Run: node ${CLAUDE_SKILL_DIR}/scripts/render.mjs --commit. It recomputes evidence count, last seen, contradicted and stale status, regenerates the markdown and the top-15 list that team-reviewer injects, marks the run processed (--commit advances the watermark in state.json; it is not a git commit), and prints added / changed / removed rules plus the rules.md diff.
  2. If it says NOT committed, classify the missing ids and rerun. Use --force only when the user agrees to drop them.
  3. Fix every warning it prints (unknown category, dangling ruleId) and rerun.

Report

## Review harvest — <since> → <today>

Query: `<query>` · <N> PRs · <N> items (<N> led to a change) · <N> one-offs skipped

### Added
### Changed
### Removed
### Top 15 now
### rules.md diff
<collapsed diff from render.mjs>

Notes & edge cases

  • Not in a repo: the fetch needs --repo; ask which repo or org rather than guess.
  • Squash/rebase workflows: follow-up commits are matched by commit date after the first comment and by file path; a force-push that rewrites dates can hide the fix. Fall back to isOutdated plus the author's reply when judging outcome.
  • AI reviewers: anything GitHub marks as a bot is excluded. Human accounts that relay an AI tool's output are not — add them with --exclude-user.
  • Huge runs: the search caps at 300 PRs (--limit). Narrow the window or filters rather than raising it far; classification quality drops with volume.
  • Reset: delete the store folder to start over; nothing else depends on it.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.