pr-flow-audit
Skill331 starsChanged 4 months agoApache-2.0View on GitHub
Use for the weekly measurement of PR review flow in phase.rs: record the week's metrics, rate a sample of review findings, check rater drift, judge due change-ledger entries, and write a one-page report. Wraps scripts/pr_flow_metrics.py.
In this repository33 other files it gives its agents.
- CLAUDE.mdCLAUDE.md
- add-ai-feature-policySkill
- add-card-data-pipelineSkill
- add-engine-effectSkill
- add-engine-variantSkill
- add-frontend-componentSkill
- add-interactive-effectSkill
- add-keywordSkill
- add-replacement-effectSkill
- address-reviewSkill
- add-static-abilitySkill
- add-triggerSkill
- ai-duelSkill
- audit-card-parsingSkill
- batch-mechanicsSkill
- bug-coverage-classifierSkill
- bug-triageSkill
- card-testSkill
- casting-stack-conditionsSkill
- changelogSkill
- engine-implementerSkill
- engine-plannerSkill
- oracle-parserSkill
- parser-velocitySkill
- pr-contribution-handlerSkill
- project-referenceSkill
- pr-review-loopSkill
- retrain-ai-weightsSkill
- review-engine-implSkill
- review-engine-planSkill
- ship-commitsSkill
- unlock-setSkill
- validate-cr-annotationsSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 27
Never edit history.jsonl, changes.jsonl or a snapshot by hand.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
PR Flow Audit
Run on Monday from the canonical checkout (the data lives untracked under .agents/pr-flow-audit/weekly/). M below is python3 scripts/pr_flow_metrics.py. Every command exits 0 on success or on an already-recorded no-op, and 2 on a refusal that wrote nothing. A refusal is not a result: read the message, fix the cause, rerun.
- Record metrics.
M append --last-week, thenM append --settled-due. Appending is idempotent per week, age class and definitions digest. - Sample.
M sample --week <latest settled week>prints the sample PRs and their round heads (3 with the most rounds, 5 random). The latest settled week is the newest week whose metrics record hasage_class: settled. - Rate. Brief two Sonnet-class agents with
rubric.mdverbatim and half of the PRs each. They write a ratings TSV with every column exceptseverity(they writeUNSURE). The lead (Opus-class) then setsseverityon every row. Save asweekly/<week>/ratings.tsv. - Drift.
M sample-drift --week <week> --ratings <tsv>prints the seeded rows as a table (pr round fid anchor finding, identification only). Save it asweekly/<week>/drift-items.tsv. A fresh Opus-class agent re-rates exactly those rows blind, given onlydrift-items.tsv, the sample listing andrubric.md, never the ratings TSV, and writesweekly/<week>/drift.tsvwith columnspr round fid location origin late severity. The drift rater ratesseverityitself on every row; writingUNSUREthere is for step 3's raters only. - Record the sample.
M sample-record --week <week> --ratings <tsv> --drift <tsv>. It validates both files, computes every number and flags axes whose drift check failed. A failed axis (including one with no comparable drift rows) isunreliable: the ledger excludes that week's sample values for every metric that reads the axis, in baselines and verdicts alike, and two consecutive failures on one axis make a rubric revision one of the proposals. - Due entries.
M ledger show --duelists each landed change whosereview_afterhas passed, with weekly baseline and post values, the baseline range, the post median and the mechanicalrule_verdict. - Judge. One Opus-class pass reads that output, the last 6 records of each kind in
history.jsonland the sample record. For each due entry it writes a verdict JSON with exactlyid,week,verdictandnote, and runsM ledger verdict <file>. The command computes the per-target evidence itself, through the same eligibility rules asledger show --due, and stores it with therule_verdict; supplyingevidenceis refused. The verdict follows the rule: medians of weekly values against the baseline range, nothing pooled. A verdict that differs fromrule_verdictneeds a non-emptynoteexplaining why. - Report. Write
weekly/<week>/report.md, about one page: early and settled key metrics against their baseline ranges, verdicts, sample and drift results, and at most 3 proposals. A proposal is a draft change entry (hypothesis, primary target with version and age class, expected movement) with its evidence. Proposals are not entered into the ledger; the maintainer decides.
Change ledger
M ledger add <file.json>: the primary target needs a registered metric, a direction and a numericexpected.M ledger baseline <id>fills null baselines from weekly values; add--refreshto replace them.M ledger landed <id> --commit <sha> --date <YYYY-MM-DD>refuses while the primary baseline is null.
Rules
- Never edit
history.jsonl,changes.jsonlor a snapshot by hand. After a definition change, recompute withM append --week <week> --input <snapshot>. - The module reads GitHub and the event log only. It writes to neither.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

