agentleFS
patricksavallepatricksavalle/investigate-journalism-skills/.agents/skills/peer-review/SKILL.md

peer-review

Skill8 starsChanged 3 months agoApache-2.0View on GitHub

A standalone orchestrator for rigorous, citation-verified peer review of scientific papers, manuscripts, preprints, studies, systematic reviews, methods papers, and research reports. Use when the task is to evaluate whether a paper's methods, statistics, causal claims, citations, reproducibility, and conclusions support its stated contribution; verify that cited sources say what the paper claims; assess pre-registration, transparency, and conflicts of interest; or produce a referee-style report with Fatal / Major / Minor findings and an explicit recommendation. Route article-level reporting about a paper to journalistic-article-review, individual claim status to scientific-fact-classification, source/network checks to osint-research or investigative-reasoning, rhetoric to fallacy-bias-and-manipulation-analysis, and later corrections to belief-revision.

51 other projects use a version of this.

Adopted by

Most recently taken up by

Used with

Projects with it also use usfiscaldata, research-grants and 10 more.

Other files people use alongside this one.

Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
  • yf8578Skill

    usfiscaldata

    Query the U.S. Treasury Fiscal Data API for federal financial data including national debt, government spending, revenue, interest rates, exchange rates, and savings bonds. Access 54 datasets and 182 data tables with no API key required. Use when working with U.S. federal fiscal data, national debt tracking (Debt to the Penny), Daily Treasury Statements, Monthly Treasury Statements, Treasury securities auctions, interest rates on Treasury securities, foreign exchange rates, savings bonds, or any U.S. government financial statistics.

    Also in 7 projects that use this

  • garri333Skill

    research-grants

    Write competitive research proposals for NSF, NIH, DOE, DARPA, and Taiwan NSTC. Agency-specific formatting, review criteria, budget preparation, broader impacts, significance statements, innovation narratives, and compliance with submission requirements.

    Also in 11 projects that use this

  • garri333Skill

    research-lookup

    Look up current research information using Perplexity's Sonar Pro Search or Sonar Reasoning Pro models through OpenRouter. Automatically selects the best model based on query complexity. Search academic papers, recent studies, technical documentation, and general research information with citations.

    Also in 13 projects that use this

  • SoljournerSkill

    get-available-resources

    Enumerate installed packages, databases, and tools for engineering workflows

    Also in 10 projects that use this

  • +8 more from garri333

The file

Show its rules and text

We split this skill into the rules it gives an agent

15 rules. 0 are common, 11 are distinctive, and 4 are very specific to this project.

  • 0

    Common

    Most of the 42 edited versions we read keep these.

  • 11

    Distinctive

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Do not use this as the primary tool for:

    line 16
  2. Distinctive:

    Keep verdicts separate: a weak paper can be reported accurately.

    line 29
  3. Very specific:

    When a media article rests on a paper, run this skill on the paper, then return the result to journalistic-article-review to test headline, …

    line 30
  4. Very specific:

    If a specialist skill is unavailable, use the fallback checks in this skill and label any memory-based facts (memory — unverified).

    line 57
  5. Distinctive:

    Every load-bearing factual claim made by the review carries a warrant:

    line 82
  6. Distinctive:

    Commit to position neutrality: do not make "conclusion disagreeable" do the work of "methods flawed".

    line 111
  7. Distinctive:

    Use specialist routing first:

    line 193
  8. Distinctive:

    Verify the citations most able to change the recommendation:

    line 227
  9. Distinctive:

    Follow citation chains to the primary at least once for load-bearing claims.

    line 245
  10. Distinctive:

    Check retraction, expression of concern, PubPeer or equivalent where relevant.

    line 247
  11. Distinctive:

    If the paper is being used downstream as causal, clinical, regulatory, or policy authority while its own design cannot bear that load, flag …

    line 287
  12. Very specific:

    When the literature itself is contested and source-counting may be laundered through one originator network, route to …

    line 289
  13. Very specific:

    Use osint-research for author/funder/retraction/affiliation checks when those facts become load-bearing.

    line 328
  14. Distinctive:

    If the paper is already published, the recommendation vocabulary is retrospective: read *Major* as "would have required major revision", …

    line 344
  15. Distinctive:

    Do not resolve a mismatch between the findings and the recommendation by regrading a finding downward.

    line 347

“Common” is measured: kept by at least 6 in 10 of the 42 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Peer Review

Audit a scientific paper as a fair, sceptical reviewer. Keep the review paper-shaped: methodology, statistical adequacy, causal licence, citation integrity, reproducibility, literature context, deployment gap, and recommendation.

Activation

Trigger on requests such as: "review this paper", "is this study sound?", "audit this manuscript", "do the results support the conclusion?", "verify the citations", "is this reproducible?", "produce a referee report", "is the methodology adequate?".

Do not use this as the primary tool for:

  • A news story, press article, newsletter, or explainer about a paper — use journalistic-article-review for the journalism layer, then invoke this skill only for the underlying paper.
  • A single empirical claim detached from a paper — use scientific-fact-classification.
  • A source, author, funder, account, image, domain, or organisation background check — use osint-research.
  • A contested event narrative — use investigative-reasoning.

Relationship To Journalistic Article Review

peer-review and journalistic-article-review are sibling orchestrators:

  • This skill evaluates whether the paper itself is sound.
  • journalistic-article-review evaluates whether an article accurately and fairly reports what the paper can bear.
  • Keep verdicts separate: a weak paper can be reported accurately; a strong paper can be reported misleadingly.
  • When a media article rests on a paper, run this skill on the paper, then return the result to journalistic-article-review to test headline, framing, omissions, and proportionality.

Orchestration Contract

This skill owns:

  • Genre and field calibration.
  • Methodology audit against the paper's own standards.
  • Statistical and causal sufficiency as far as needed to grade the paper.
  • Citation verification and misrepresentation grading.
  • Reproducibility, transparency, COI, and funder-role review.
  • Literature context and deployment-gap audit: what later papers, policy, media, or institutions use this paper to support.
  • Severity grading and publication recommendation.

Offload specialist work where possible:

NeedRoute toReturn value needed
Pre-review hunch, gut feeling, or anomaly signalintuitive-thinkingHunch register and search leads
Claim type, evidential status, causal licence, GRADE-style strengthscientific-fact-classificationClaim classification and confidence
Logical fallacy, rhetorical overclaim, theoretical argument structurefallacy-bias-and-manipulation-analysisSurviving fallacy/manipulation findings
Author/funder/affiliation/retraction/replication/public footprintosint-researchVerified source-network facts
Coordinated campaign, contested literature, narrative launderinginvestigative-reasoningIndependent-node and hypothesis analysis
Named framework decomposition, hidden assumptionsfirst-principles-thinkingRequirement list and assumption map
Correction, retraction, replication after an earlier reviewbelief-revisionCalibrated update from prior verdict
Journalism about the paperjournalistic-article-reviewArticle-level accuracy/framing verdict

If a specialist skill is unavailable, use the fallback checks in this skill and label any memory-based facts (memory — unverified).

When This Skill Is Silent Or Ambiguous

First check whether another project skill owns the missing layer: hunch / gut feeling / anomaly signal -> intuitive-thinking; scientific claim status -> scientific-fact-classification; paper methods/statistics/citations/reproducibility -> peer-review; article framing/reporting accuracy -> journalistic-article-review; source identity/funding/public records -> osint-research; contested events or competing narratives -> investigative-reasoning; definitions, hidden assumptions, or argument bedrock -> first-principles-thinking; fallacies/rhetoric/statistical framing tricks -> fallacy-bias-and-manipulation-analysis; new evidence changing a prior verdict -> belief-revision.

If no skill clearly owns the gap, reason from first principles and explicit warrants. Built-in knowledge may suggest hypotheses, search terms, possible failure modes, or questions to verify, but any empirical premise remains (memory — unverified) until traced. Reasoning may connect warranted premises; it may not manufacture premises.

Research Discipline (CLAUDE.md/AGENTS.md)

This skill is standalone. Apply these rules even if CLAUDE.md / AGENTS.md are not loaded:

  • Rule 1 (pre-search hypothesis registration) — Before searching, register the reviewer's prior expectation about the field consensus, likely paper strength, and possible failure modes.
  • Rule 2 (steelman from primary literature) — Restate the paper's central claim and strongest methodological rationale before criticising.
  • Rule 3 (primary before secondary) — Fetch the cited paper, protocol, dataset, guideline, registry, or primary document before relying on a secondary characterisation.
  • Rule 4 (map institutional networks) — Treat studies, authors, funders, labs, policy bodies, and commentators as one node when they share material alignment until independent corroboration is shown.
  • Rule 5 (Tier 0 priority for time-sensitive claims) — For citation chains and time-sensitive fields, prefer original/contemporary primary sources over later reviews or summaries.
  • Rule 6 (bias self-audit) — End by answering whether the same verdict would have been reached if the paper's conclusion ran the other way.
  • Rule 7 (minimum search volumes) — Typical paper: verify 5-15 load-bearing citations/sources. Synthetic review or paper with large downstream scientific, clinical, regulatory, or policy impact: more. Beyond 40, recommend a dedicated deep-research session.
  • Rule 8 (hostility check on sources) — For each cited source used by the review, record funding, ownership, mandate, author/funder alignment, and national alignment where relevant.
  • Rule 9 (interactive refinement: user contributions are inputs, not warrants) — User-supplied corrections are (user-supplied — unverified) until fetched and verified. Do not soften severity grades on pressure absent new primary evidence.
  • Rule 10 (objective report voice) — Write the report as a standalone verdict on the paper. Do not refer to the requester in the report prose.

Warrant Labels

Every load-bearing factual claim made by the review carries a warrant:

LabelMeaning
(traced)Followed the evidence chain to a primary source fetched in this session via WebFetch/WebSearch, or an explicit terminal/API fetch where the browser fetch path is unsuitable. State URL + access date.
(deferred to consensus)Relying on a named social/institutional consensus mechanism (literature body, regulatory body, textbook, official record system). Consensus is not scientific warrant; for scientific claims, treat it only as a political/social prior unless traced to reproduced or replicated evidence.
(deferred, fragile)Deferred to consensus, but scientific-fact-classification Phase 6c failure modes apply — funder capture, ideological capture, prestige cascade, replication crisis, publication bias, or similar. State which.
(memory — unverified)Recalled from training data, not verified this session. Permitted only with this label, and never load-bearing without an explicit "this could be wrong" caveat.
(user-supplied — unverified)Provided during interactive refinement and not verified in-session. Never load-bearing on its own; treat as a hypothesis to test or an input to verify.
(intuition — unwarranted)A gut feeling, anomaly signal, or pattern impression. It may generate hypotheses and search leads. It is never evidence, never load-bearing, and cannot revise, refute, or establish a claim.

The consensus mechanisms most often deferred to here are reporting and conduct standards — CONSORT, STROBE, PRISMA, ICH-GCP, ARRIVE — and a regulator, textbook, or literature body. None of them is scientific warrant on its own.

A citation-verification verdict is itself a reviewer claim. Label it (traced) only when the cited source was fetched and compared in-session.

Phase 0 — Pre-Review Setup

Register before search:

ItemEntry
Paper identityTitle, authors, venue, publication/preprint date, DOI/URL
GenreEmpirical primary / theoretical / methods / systematic review / narrative review / position / preprint / replication / registered report
Field and subfieldUse the field's actual standards
Stated contributionOne sentence, quote where possible
Strongest fair readingThe best version of what the paper claims
Prior expectationField consensus as a political/social prior, plus likely paper strength before detailed checking
Failure modes to testMethod mismatch, inferential gap, causal overreach, citation mismatch, p-hacking, missing data/code, COI, deployment gap

Commit to position neutrality: do not make "conclusion disagreeable" do the work of "methods flawed"; do not make prestige do the work of evidence.

Venue/status discipline: peer review, journal reputation, publisher, author prestige, institution, and citation count are provenance facts, not scientific principles. Record them in the paper identity, but do not upgrade the paper's evidential strength or recommendation because of them. Upgrade only for valid design, reproduced analyses, independent replication, transparent data/code/materials, adequate measurement, and conclusions that stay inside the evidence.

Phase 1 — Paper Map

Map the paper's internal load:

ElementCapture
Research questionExact question or problem
HypothesesPre-specified vs exploratory
DesignWhat was actually done
Inputs/materials/dataConcrete physical material, dataset, corpus, survey frame, intervention, or sample
OutcomesPrimary, secondary, exploratory
ResultsEffect sizes, uncertainty, null/negative results, affected/unaffected counts by group
Title / abstract claimExact headline claim; named standards or proof-language are load-bearing, not packaging
ConclusionsExact language; especially modal verbs, causal verbs, named standards, and proof-language
Load-bearing citationsSources that make the contribution possible
Inference gapDistance between what was measured and what is claimed

Route load-bearing claim typing to scientific-fact-classification when classification affects severity.

Phase 2 — Field And Methodology Calibration

Hold the paper to its own genre's bar:

Field / genreMinimum adequacy check
RCT / clinicalCONSORT; randomisation; allocation concealment; blinding; ITT; pre-registration and deviations
Observational epidemiologySTROBE; confounding strategy; temporality; reverse-causation check; representativeness
Lab biology / animalReagent/sample identity; active-agent/material isolation; carrier-matrix controls; biological and technical replicates; animal/group counts; affected/unaffected distribution; contamination checks; ARRIVE for animal work
Psychology / behaviouralPre-registration; effect sizes and CIs; power; replication-crisis awareness; sample scope
SurveysSampling frame; response rate; non-response; validated instrument; question-order effects
Economics / policyIdentification strategy; parallel trends or exclusion restrictions; clustered SEs; robustness checks
ML / NLPData leakage; test contamination; multiple seeds; matched baselines; hyperparameter budget; compute confound
Theoretical / mathematicalFormal claim; assumptions; proof/derivation; novelty relative to prior results
Systematic review / meta-analysisPRISMA; protocol; reproducible search; risk of bias; heterogeneity; publication bias
ReplicationSame operationalisation; pre-registered; powered for original effect; nuanced success/failure
QualitativeSampling logic; reflexivity; analytic procedure; triangulation or member checking where warranted

Framework-To-Operation Traceability

When a paper invokes a named standard, criterion set, checklist, model, diagnostic category, or proof threshold, the invocation is itself a load-bearing claim. Do not pass through phrases such as "fulfils X", "meets Y", "according to Z", "gold-standard", "validated", "isolated", "purified", "causal", "diagnostic", "representative", "pre-registered", or "CONSORT/PRISMA/STROBE/ARRIVE-compliant" without decomposing the operational requirements.

Apply this invariant:

A framework claim is warranted only when each required operation is mapped to a concrete action, material object, measurement, control, citation, and warrant.

Build a criterion table before grading the claim:

RequirementWhat the framework requiresWhat this paper directly didWhat is borrowed from citationsMaterial / dataset / intervention usedControls or exclusionsStatus
[criterion][operation required]Shown / not shown / unclearSource + warrant or noneExact object, preparation, version, population, or constructNegative, mock, placebo, sham, contaminant, carrier, confounder, or sensitivity checkSatisfied / partial / unreported / contradicted

Rules:

  • Claim-standard identity. The title, abstract, methods, and conclusion must name the same standard. If the title says one standard but the body applies a modified, weaker, narrower, or different standard, flag title/abstract overclaim. Grade Major or Fatal when the standard claim is central.
  • Criterion-by-criterion satisfaction. "Framework fulfilled" is false or overstated if any required criterion is unreported, citation-borrowed without verification, only partially satisfied, or satisfied under a different standard. Grade by centrality.
  • Current-paper vs citation-borrowed warrant. Separate "this paper directly showed X" from "this paper cites prior work for X." Citation-borrowed criteria must be citation-verified in Phase 4 before they can carry the framework claim.
  • Local warrant boundary. A strong broader literature can support the scientific conclusion while leaving this paper's own framework/proof claim overstated. Do not let consensus or later corroboration repair the wording of the paper under review.
  • Operational equivalence check. The operation actually performed must match the operation claimed. "Measured a proxy", "used a preparation", "analysed a convenience sample", or "applied a classifier" is not automatically equivalent to the claimed active agent, construct, population, or endpoint.

Active-Agent / Material Isolation Audit

Mandatory material question: what, in concrete terms, was actually used? If "purified X", "isolated Y", "treated W", "administered Z", "the dataset", "the intervention", or a named construct carries the conclusion, check whether the paper characterises it enough to support the active-agent or construct claim.

For any causal, mechanistic, intervention, infection, toxicology, material-science, ML, survey, or dataset claim, identify:

DimensionMinimum question
Active agent / constructWhat exact entity is claimed to do the work?
Carrier matrixWhat else was administered, measured, exposed, cultured, processed, sampled, or bundled with it?
Preparation / processingHow was it produced, filtered, purified, extracted, passaged, cleaned, labelled, transformed, split, or normalised?
Dose / version / routeWhat amount, version, route, time window, population, or environment was used?
Exclusion controlsWere mock, sham, placebo, heat-inactivated, vehicle-only, blank, negative, contamination, carrier-only, confounder, leakage, or sensitivity controls used?
Identity verificationHow was the active agent, sample, construct, dataset, endpoint, or label verified independently of the outcome?

If the claimed active agent is not separated from plausible co-administered causes, carrier effects, processing artefacts, contaminants, batch effects, dataset leakage, measurement artefacts, or construct drift, flag active-agent / construct under-determination. Grade as Major when the conclusion can be repaired by caveat or controls; Fatal when the central causal/proof claim depends on the unresolved identity.

Phase 3 — Statistical And Causal Sufficiency

Use specialist routing first:

  • Route causal licence and evidence strength to scientific-fact-classification when the paper's conclusion depends on causation.
  • Route hidden assumption stacks or named criteria to first-principles-thinking.
  • Route rhetorical or theoretical overreach to fallacy-bias-and-manipulation-analysis.

Fallback minimum checks:

RiskCheck
p-hacking / forking pathsPre-registration, number of tests, selective reporting, p-values clustered below threshold
Low powerA priori power, confidence intervals, Type-M/sign errors
Small n / responder skewPer-group n, affected/unaffected counts, non-responder fraction, whether the central claim describes all subjects or only responders
Multiple comparisonsCorrection or explicit exploratory framing
Wrong model/testDistribution, dependence, clustering, unit of analysis
Practical irrelevanceAbsolute effects, baseline risk, clinical/policy significance
Relative-only effect claimEvent counts, baseline risk, absolute effect, relative effect, timeframe, population
Odds / hazard / rate ratio as riskMetric named exactly; conversion justified before risk language
Subgroup cherry-pickPre-specified subgroup, interaction test, all subgroups reported
PseudoreplicationUnit of analysis equals unit of randomisation/sampling
ML leakageSplit integrity, contamination, repeated test-set tuning
Causal overreachDesign identifies causation, temporality, confounding, reverse causation, language calibration
Reverse causation unresolvedDirectionality failure; downgrade causal language unless design or analysis rules out reverse direction

Quote the abstract/conclusion language and compare it with the design. "Associated with" in results becoming "causes" in the abstract is a Major or Fatal finding depending on centrality.

For any causal claim, state whether reverse causation was ruled out by design, tested directly, made implausible by mechanism/timing, or left unresolved. If reverse causation remains plausible, require association/prediction language. Grade Major or Fatal when a central conclusion, abstract claim, clinical/policy implication, or media-use hook depends on causal direction that the design cannot establish.

For animal, primate, clinical, intervention, challenge, toxicity, infection, or behavioural studies, always write the denominator sentence before grading: n=[total]; groups=[n per group]; affected/responded=[x/y per group]; unaffected/non-responded=[x/y per group]. If a headline, abstract, or conclusion implies a general effect while a substantial fraction of subjects were unaffected, flag responder-generalisation overreach. Grade Major or Fatal when the central claim depends on treating partial response as general effect.

For any comparative effect claim in epidemiology, clinical work, animal work, policy, safety, efficacy, behavioural outcomes, or model evaluation, write the effect decomposition before grading: counts/events by group, baseline/control risk, comparison/treatment risk, absolute risk difference, relative risk or named ratio, timeframe, population, and uncertainty. Keep ARR/ARI, RRR/relative increase, odds ratio, hazard ratio, and rate ratio separate. A relative-only benefit or harm claim is under-contextualised by default; grade Major when it carries the abstract, conclusion, clinical/policy implication, or media-use hook.

Phase 4 — Citation Verification

Verify the citations most able to change the recommendation:

  • Citations supporting the central claim.
  • Citations invoked to satisfy a named framework or criterion.
  • Citations supporting methodological choices.
  • Citations establishing novelty or gap.
  • Citations supplying effect sizes, priors, or power assumptions.
  • Citations to authoritative or contradicting prior literature.

If the paper's central claim relies on a cited empirical study, do not stop at "the citation says what the paper claims." Run a targeted cited-study peer review on that source: design, sample/materials, statistics, causal licence, reproduction/replication status, COI, and whether the cited study can bear the load assigned to it. This is mandatory for any cited study without which the reviewed paper's main conclusion, novelty claim, causal claim, or policy/clinical implication would materially weaken.

Recursion limit: one cited-study layer is mandatory when central. Follow a second layer only when the first cited study's own central warrant depends on another citation and that dependency could change the recommendation. Beyond two layers, stop and recommend a dedicated citation-chain review.

Process:

  1. Fetch DOI, publisher page, preprint, PMC, repository, protocol, registry, or other primary source.
  2. Locate the specific cited claim.
  3. Compare paper's characterisation with source content.
  4. Follow citation chains to the primary at least once for load-bearing claims.
  5. For central empirical dependencies, perform the targeted cited-study peer review above.
  6. Check retraction, expression of concern, PubPeer or equivalent where relevant.

Verdicts:

VerdictMeaning
SupportsCited source says what the paper claims.
PartialSource supports a weaker or narrower claim.
ContradictsSource points against the paper's claim.
UnrelatedSource does not bear on the claim.
UnverifiableSource inaccessible, unpublished, "data not shown", personal communication, or otherwise not checkable.

Discrepancy template:

Cited as:             [paper's claim about the source, verbatim]
Source actually says: [quote or precise summary from fetched source]
Discrepancy type:     Supports-weaker | Partial-overlap | Contradicts | Unrelated | Misquotes | Wrong-direction | Retracted
Severity:             Fatal | Major | Minor
Warrant:              (traced — fetched [date]) | (memory — unverified, source unavailable)

Phase 5 — Literature Context And Deployment Gap

Audit both the paper's lane and its downstream use:

QuestionCheck
Is the paper consistent with recent systematic reviews/meta-analyses?Fetch relevant synthesis if load-bearing
Are strong contradictions or replications engaged?If not, flag selective context
Is novelty real?Check whether similar methods/results already exist
Has the result been reproduced or independently replicated?Prefer reproduction/replication over venue or citation status
Are downstream users citing it correctly?Sample later papers, policy, regulatory, media, or institutional uses

Deployment-gap categories:

  • Inside-lane — downstream use matches what the paper actually shows.
  • Lane-stretched — downstream use is stronger than the paper supports.
  • Outside-lane — downstream use makes the paper bear a claim it is silent on or contradicts.

If the paper is being used downstream as causal, clinical, regulatory, or policy authority while its own design cannot bear that load, flag the deployment gap separately from in-paper faults. If the paper itself encourages the overuse, grade accordingly.

When the literature itself is contested and source-counting may be laundered through one originator network, route to investigative-reasoning.

Phase 6 — Reproducibility, Transparency, And COI

Check:

AreaMinimum
DataPublic, controlled-access with clear process, or unavailable
CodeAvailable, versioned, executable enough to reproduce analyses
MethodsProtocol detail sufficient for reproduction and independent replication
MaterialsReagents, suppliers, catalogue numbers, corpus/data versions, survey instrument
Pre-registrationPresent, followed, deviations declared
Ethics and consentFor human- or animal-subjects work: named approving body, approval or waiver basis, and consent status, stated in the article
COIFinancial, institutional, ideological, author contribution, funder role
Openness limitsPaywalls, inaccessible supplements, unavailable data, undisclosed materials

Ethics and consent are a named-standard check, not an optional courtesy. ICMJE requires that authors "seek approval to conduct research from an independent local, regional or national review body… and be prepared to provide documentation when requested by editors", and that where informed consent was obtained "it should be indicated in the published article" (deferred to consensus — ICMJE, an editor-body coordination mechanism, not scientific warrant). Search the full text explicitly for approval, waiver, consent, IRB, and ethics-committee language before concluding it is absent, and record the search.

Distinguish two failures that look identical in the text:

  • Unstated but obtained — a reporting defect. The remedy is one sentence. Grade Minor.
  • Absent in fact — the work should not have proceeded. Grade Fatal.

A text-only review usually cannot tell which. Default to the reporting defect, and state the escalation condition explicitly: what evidence would move it to Fatal.

Separate two checks:

  • Reproduction: can the reported analyses/results be recovered from the same data, code, materials, and protocol?
  • Replication: has an independent team, sample, lab, population, instrument, or dataset tested the same claim?

A prestigious peer-reviewed paper with unavailable data/code or unreproducible analyses is downgraded. A preprint or low-status venue with open materials and reproduced analyses can outrank a high-status paper on scientific warrant, though it may still need independent replication before strong generalisation.

Use osint-research for author/funder/retraction/affiliation checks when those facts become load-bearing.

Phase 7 — Severity And Recommendation

Severity:

TagDefinitionTypical effect
FatalCentral claim falsified, unsupported, or unevaluableReject or fundamental redesign
MajorCore claim overstated or method insufficient, fixable with substantial revisionMajor revisions
MinorSpecific overclaim, check missing, caveat neededMinor revisions
OptionalUseful improvement but not requiredSuggestion
PraiseGenuine strength worth namingFairness and calibration

Recommendation: Accept / Minor / Major / Reject-resubmit / Reject.

If the paper is already published, the recommendation vocabulary is retrospective: read Major as "would have required major revision", not as a live editorial decision.

Do not resolve a mismatch between the findings and the recommendation by regrading a finding downward. If the recommendation feels too severe for the paper, that is a reason to re-examine the recommendation, not to soften a fault that survived the evidence. State the mismatch rather than dissolving it.

State explicitly what would change the recommendation upward or downward.

Output

# Peer Review: [title]

## Summary
- **Paper:** [title, authors, venue/status, date]
- **Genre / field:**
- **Recommendation:** Accept / Minor / Major / Reject-resubmit / Reject
- **Verdict:** [one sentence]

## Paper Map
[research question, methods, results, conclusions, load-bearing claims, inference gap]

## What the Paper Does Well
[2-5 genuine strengths]

## Fatal Findings
[Each: quoted passage -> named fault -> why fatal -> fix if possible -> warrant]

## Major Findings
[Each: quoted passage -> named fault -> severity rationale -> required action -> warrant]

## Minor Findings

## Optional Suggestions

## Methodology Audit
[field-calibrated checks from Phase 2]

## Statistical And Causal Audit
[specialist outputs or fallback checks; quote specific numbers/language; state reverse-causation status for causal claims]

## Sample And Responder Denominators
| Group | n | Affected/responded | Unaffected/non-responded | Exclusions/missing | Claim language fits denominator? | Severity | Warrant |
|---|---|---|---|---|---|---|---|

## Quantified Effect Decomposition
| Claim | Population/timeframe | Events / total by group | Baseline risk | Comparison risk | Absolute effect | Relative effect / metric | Uncertainty | Practical meaning | Severity | Warrant |
|---|---|---|---|---|---|---|---|---|---|---|

## Citation Verification
| Paper claim | Cited source | Source actually says | Verdict | Severity | Warrant |
|---|---|---|---|---|---|

## Framework-To-Operation Traceability
| Framework / standard claim | Requirement | Directly shown in this paper | Citation-borrowed support | Material / dataset / intervention identity | Controls / exclusions | Status | Severity | Warrant |
|---|---|---|---|---|---|---|---|---|

## Active-Agent / Material Isolation
| Claim | Claimed active agent / construct | What was actually used | Carrier matrix / co-administered material | Preparation / processing | Identity verification | Exclusion controls | Adequate? | Severity | Warrant |
|---|---|---|---|---|---|---|---|---|---|

## Targeted Cited-Study Reviews
| Cited study | Why central | Method/statistical adequacy | Reproduction/replication status | Can bear cited load? | Severity impact | Warrant |
|---|---|---|---|---|---|---|

## Literature Context And Deployment Gap
| Downstream use / literature claim | What this paper actually supports | Inside-lane / Lane-stretched / Outside-lane | Warrant |
|---|---|---|---|

## Reproducibility, Transparency, And COI
[data/code/materials/protocol/COI/funder-role findings]

## Specialist Checks
[which skills were invoked or should be invoked, and what paper-level dependency they affect]

## Sources & Warrants
| Review finding / citation verdict | Source | URL | Access date | Publication date | Warrant | Funding / ownership / mandate / alignment |
|---|---|---|---|---|---|---|

## Confidence & Severity
- **Counts:** Fatal / Major / Minor / Optional / Praise
- **Confidence in this review:**

## What Would Change This
- Upward:
- Downward:

## Self-Audit
- **Symmetry test:** Would the same recommendation have been reached if the paper's conclusion ran the other way? Execute the flip; do not merely assert. Name the specific judgements (which findings were graded Fatal rather than Major, the citation verdicts, the recommendation threshold) where the verdict is most sensitive to the prior — asserting symmetry flatly, without identifying where it could break, claims the property rather than showing it. If no — explain. If you can't tell — say so.
- **Standards applied:** [genre and field]
- **Difference in approach vs actual fault separated:**
- **Reviewer priors named:** [topic/authors/funder direction, if any]

## Limits
- Domain expertise required for:
- Sources not accessible:
- Citations unverifiable:
- Specialist review still needed:

Quick Reference

PatternRiskDefault move
Title names one standard, methods apply anotherClaim-standard mismatchGrade title/abstract overclaim; decompose both standards
Abstract causal, design observationalCausal overreachMajor/Fatal; calibrate language or redesign
Reverse causation not addressedDirectionality failureDowngrade to association or require design/analysis that rules out reverse direction
Named framework invoked but requirements not checkedFramework launderingBuild criterion-to-operation table; verify each requirement
Framework criterion satisfied only by citationsCitation-borrowed warrantVerify cited source and separate current-paper evidence from borrowed support
Cited source supports weaker claimCitation overstatementQuote both; grade by centrality
Central claim rests on cited studyUnreviewed dependencyRun targeted cited-study peer review
Input/material under-characterisedActive-agent or construct under-determinationIdentify carrier matrix, preparation, identity checks, and exclusion controls; Major/Fatal if central
Claimed active agent bundled with carrier, culture, vehicle, batch, dataset, or proxyMatrix/confound ambiguityRequire mock/vehicle/blank/carrier-only or equivalent controls
Broad literature supports conclusion but paper overstates its own proofLocal warrant boundary breachPreserve conclusion context but grade this paper's claim separately
p < .05 without effect size/CIStatistical opacityRequire effect sizes and uncertainty
Small sample carries central claimFragility / low powerState per-group n; downgrade unless effect is robust and replicated
Half or more subjects unaffectedResponder-generalisation overreachSeparate responder result from general claim; grade by centrality
Relative risk reduction without baselineRelative-effect launderingReport ARR/RRR and event rates separately
Odds/hazard/rate ratio described as riskMetric substitutionRename metric or justify conversion from data
Many tests, one headline resultMultiple-testing riskRequire correction or exploratory framing
Data/code "available on request"Reproducibility weaknessDowngrade transparency
Sample scope narrower than conclusionGeneralisation overreachRestrict conclusion
Downstream citation uses paper for stronger claimDeployment gapSeparate paper fault from misuse
COI disclosure absent in stake-heavy fieldDisclosure failureRequire disclosure and source-network check
Prestigious venue used as evidenceStatus launderingIgnore venue as warrant; inspect methods/reproduction
Reproducible preprint vs unreproducible journal articleEvidence beats venueWeight reproducibility and replication higher

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.