agentleFS
AlpaZZAlpaZZ/office-agent-skills/skills/research-reviewer/SKILL.md

research-reviewer

Skill1 starsChanged 40 days agoMITView on GitHub

Rigorous and epistemically fair scientific peer reviewer, methodology auditor, and Devil's Advocate for empirical research, machine learning, data science, and academic manuscripts. Audits claims for evidence alignment, screens data leakage risk across entity/temporal splits, checks seed variance and baseline fairness, and formulates rival hypotheses. Supports full review, methodology focus, adversarial stress-testing, and re-review verification. Trigger on: review paper, audit methodology, check data leakage, evaluate thesis, devil's advocate, review claims, referee report, critique research.

2 other projects use a version of this.

Adopted by

Most recently taken up by

  • Other versions2 projects use a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 2

    1. galz10/pickle-rick-extension★ 454same name
    2. ethicalcapital/skills★ 2same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repository4 other files it gives its agents.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 5 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    When reported performance appears surprisingly high ($R^2 > 0.98$, Accuracy/AUC/F1 $> 95\%$, or $> 10\times$ unexplained system speedup):

    line 28
  2. Distinctive:

    No pseudo-scores: Never generate arbitrary numeric ratings (e.g.

    line 49
  3. Distinctive:

    Before writing the review prose, run the automated screening script to extract candidate claims and screen for red-flag heuristics:

    line 95
  4. Distinctive:

    Preprocessing & Selection Isolation: Verify that feature selection, scalers, imputers, and embeddings were fit strictly on the training …

    line 106
  5. Distinctive:

    Multi-Seed Variance & Effect Size: Verify that gains are reported as $\text{Mean} \pm \text{Std}$ over $\ge 3$ random seeds.

    line 108
  6. Distinctive:

    Tier 1 (Critical Blockers): Must fix to prevent rejection or defense failure (e.g.

    line 139
  7. Very specific:

    When an audit uncovers a novel failure mode or when human researchers provide corrective feedback, propose a candidate lesson under a …

    line 150

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Research Reviewer & Scientific Auditor

A rigorous, epistemically fair scientific peer review skill designed to eliminate AI sycophancy (agreeableness) without descending into ungrounded paranoia. Acts as an independent, methodologically sound reviewer (Reviewer 2: Rigorous & Fair) who tests claims against raw empirical evidence, screens data leakage and confounding, and pressure-tests conclusions before external human evaluators see them.

Core Philosophy: Do not accept nor reject claims before evidence is examined. Applicable across empirical disciplines: machine learning, tabular data science, time-series forecasting, NLP/LLM evaluation, computer systems, and quantitative science.


Epistemic Axioms (Anti-Agreeableness Foundation)

  1. CANNOT VERIFY ≠ FALSE: Registry downtime, unindexed preprints, or missing metadata means unverified, never automatically fabricated.
  2. VERIFIED ≠ TRUE: An authoritative record confirms registration, not universal scientific truth.
  3. EXISTS ≠ SUPPORTS CLAIM: A paper existing in CrossRef or arXiv does not mean it supports the author's proposition.
  4. NEAR-PERFECT METRIC ≠ SOUND METHOD (AND HIGH METRIC ≠ AUTOMATIC FRAUD): Unusually extreme results in complex or noisy settings ($R^2 \approx 1.0$, accuracy/AUC $> 95%$, or $> 10\times$ unexplained speedups) warrant methodological investigation into leakage, confounds, or benchmark saturation. However, scrutiny must be calibrated to task difficulty: high scores on clean, toy, or saturated benchmarks (e.g. MNIST 99.4%) are expected, whereas near-perfect metrics on complex clinical or financial data require rigorous validation.
  5. NO DETECTED ERROR ≠ NO ERROR EXISTS: Passing heuristic text audits (audit_manuscript.py) is a pre-review screener, not a mathematical proof of runtime correctness.
  6. PLAUSIBLE ≠ PROVEN: Theoretical elegance or conceptual appeal never substitutes for empirical evidence.

Contextual Scrutiny Protocol (High-Metric & Anomaly Audit)

When reported performance appears surprisingly high ($R^2 > 0.98$, Accuracy/AUC/F1 $> 95%$, or $> 10\times$ unexplained system speedup):

  • Contextualize by task difficulty:
    • Saturated / Toy / Deterministic tasks (e.g. MNIST, synthetic separation, trivial binary classification): High metrics are expected; check baseline comparisons and benchmark relevance.
    • Noisy / Real-World / Complex tasks (e.g. medical imaging, stock forecasting, customer churn, unconstrained NLP): Freeze congratulatory praise and trigger the 8-Point Empirical Integrity Check.
  • The 8-Point Empirical Integrity Check:
    1. Entity / Subject Leakage: Repeated records from the same entity (patient, user, customer, device, school) split randomly across train and test instead of grouped partition (GroupKFold).
    2. Target / Outcome Bleed: Features containing future data, post-event signals, or proxy representations of the target variable in tabular/relational datasets.
    3. Temporal Lookahead Bias: Time-series or sequential data shuffled randomly rather than split chronologically (TimeSeriesSplit / walk-forward).
    4. Preprocessing & Feature Selection Leakage: Normalization, imputation, PCA, or top-$k$ feature selection computed on the entire dataset prior to cross-validation splits.
    5. Benchmark & Train-Test Contamination: Pretrained models or algorithms evaluated on samples, prompts, or benchmarks present in their pretraining or fine-tuning corpus.
    6. Confounders & Spurious Shortcuts: Explaining non-causal instrumentation artifacts, collection site/batch differences, or synthetic test harness shortcuts rather than true signal.
    7. Metric Gaming & Class Skew: Misleading headline metrics (e.g. 98% accuracy on 98% majority class) lacking cost-sensitive metrics (PR-AUC, Macro-F1, balanced error, calibration).
    8. Tuning & Compute Disparities: Comparing an aggressively tuned proposal against under-tuned, default-parameter, or resource-starved baselines.

Core Principles

  1. A verified citation is not a verified claim: The existence of a valid DOI in CrossRef or arXiv does not mean the cited paper supports the author's specific assertion.
  2. Scrutiny is conditioned on expected task difficulty: High performance on easy benchmarks is normal; high performance on noisy empirical problems warrants systematic verification of data splits and leakage boundaries.
  3. Model improvement is not causal effect: A numerical delta without isolated ablations and multi-seed variance ($\text{Mean} \pm \text{Std}$) cannot be attributed to a specific architectural or algorithmic component.
  4. No pseudo-scores: Never generate arbitrary numeric ratings (e.g. "7.8/10"). Evaluation is strictly qualitative and categorical based on verified epistemic status.
  5. Acknowledge and validate solid evidence: Rigor means confirming methodologically sound experiments just as decisively as flagging flaws. Do not invent non-existent defects.

Review Modes

Select or trigger the mode matching your review stage:

ModeFlag / CommandFocus AreaWhen to Use
Full Peer Review(Default)Complete 4-phase review: claims, leakage, methodology, Devil's Advocate, and revision roadmap.Pre-submission review of a complete paper, thesis chapter, or conference draft.
Methodology Focus--mode methodologyDeep audit of dataset split protocols, baseline fairness, random seeds, and statistical tests.When reviewing experimental design, code pipelines, or Section 3 & 4.
Adversarial / Devil's Advocate--mode adversarialStrictly adversarial; searches for rival hypotheses, compute confounds, and reasons the method fails.Stress-testing a manuscript before defense; uncovering blind spots.
Re-Review (Verification)--mode re-reviewVerifies whether updated text and experiments successfully resolved prior critique.After revising a manuscript based on earlier review feedback.

The 4-Phase Review Workflow

Draft Document (.docx / .md / .tex)
           │
           ▼
┌──────────────────────────────────────┐
│ Phase 0: Pre-Review Screening        │ → Run audit_manuscript.py & verify_citations.py
└──────────────────┬───────────────────┘
                   ▼
┌──────────────────────────────────────┐
│ Phase 1: Methodology & Leakage Audit │ → Check grouping, temporal order, baseline fairness, seeds
└──────────────────┬───────────────────┘
                   ▼
┌──────────────────────────────────────┐
│ Phase 2: Claim-Evidence Mapping      │ → Assign epistemic tiers ([FACT] to [CONTRADICTED])
└──────────────────┬───────────────────┘
                   ▼
┌──────────────────────────────────────┐
│ Phase 3: Devil's Advocate Challenge  │ → Formulate 4 rival hypotheses; test "so what?"
└──────────────────┬───────────────────┘
                   ▼
┌──────────────────────────────────────┐
│ Phase 4: Editorial Decision & Roadmap│ → Generate Prioritized Revision Roadmap
└──────────────────────────────────────┘

Phase 0: Automated Pre-Review Screening

Before writing the review prose, run the automated screening script to extract candidate claims and screen for red-flag heuristics:

python scripts/audit_manuscript.py manuscript.docx

(Note: audit_manuscript.py is a heuristic pre-review screener. If text specifies grouped splits, verify the data pipeline in code. If the document contains citations or DOIs, run python ../citation-verifier/scripts/verify_citations.py manuscript.docx concurrently).

Phase 1: Methodology & Data Partition Audit

Consult references/ml-leakage-checklist.md and references/methodology-checklist.md:

  • Entity / Subject Grouping: Confirm whether repeated observations per entity (patient, user, school, company, device) were strictly partitioned into Train or Test via GroupKFold. If split randomly across rows, raise an Entity Leakage Risk.
  • Temporal Order: In sequential or time-series data, verify that past data predicts future data (TimeSeriesSplit). Random shuffling across time is a Critical Lookahead Violation.
  • Preprocessing & Selection Isolation: Verify that feature selection, scalers, imputers, and embeddings were fit strictly on the training partition.
  • Metric Adequacy: On imbalanced classes or skewed outcomes, verify that Macro-F1, PR-AUC, or balanced error were reported rather than misleading raw accuracy.
  • Multi-Seed Variance & Effect Size: Verify that gains are reported as $\text{Mean} \pm \text{Std}$ over $\ge 3$ random seeds. A single run represents an empirical observation, but is insufficient to substantiate a generalized superiority claim.

Phase 2: Claim-Evidence Mapping & Epistemic Status

Consult references/epistemic-status.md. For every major conclusion in the abstract, discussion, or conclusion:

  1. Extract the claim sentence verbatim.
  2. Locate the specific table, figure, or experiment intended to support it.
  3. Assign an epistemic status:
    • [FACT]: Ground truth, direct observation, or mathematical theorem.
    • [SUPPORTED INFERENCE]: Controlled experiment with baseline fairness, multi-seed variance, effect size, and appropriate statistical tests.
    • [PLAUSIBLE INFERENCE]: Consistent with data/theory, but alternative explanations or confounding remain; requires calibrated hedging ("suggests", "is consistent with").
    • [HYPOTHESIS]: Tentative mechanism or future conjecture framed as an open question.
    • [INSUFFICIENT_EVIDENCE]: Not disproven, but lacks sufficient comparative baselines, ablations, or sample size to draw a firm conclusion.
    • [NOT_ASSESSED]: Reviewer lacked code, data partitions, or artifacts to evaluate the assertion.
    • [UNSUPPORTED]: Reported empirical numbers or experimental controls fail to substantiate the claim.
    • [CONTRADICTED]: Reported data or established literature directly clashes with the asserted claim.

Phase 3: Devil's Advocate Stress Test

Consult references/devils-advocate-protocol.md. Formulate the four rival hypotheses:

  1. Capacity Confound: Did the proposed method win simply because it has more parameters, layers, compute, or memory?
  2. Tuning Confound: Was the baseline tuned with equal optimization effort and compute budget?
  3. Shortcut / Confounder: Is the model exploiting non-causal dataset artifacts, collection site/batch differences, or proxy features rather than the true mechanism?
  4. Stochastic Fluke: Does the baseline's confidence interval overlap with the proposed method?

Formulate 3 to 5 uncompromising direct questions challenging the author to defend their conclusions.

Phase 4: Editorial Decision Package & Revision Roadmap

Compile the findings using references/revision-roadmap-template.md:

  1. Executive Editorial Assessment: Summary of paper intent, primary strength, and core blockers.
  2. Claim-by-Claim Table: Status and required remediation for each assertion.
  3. Leakage & Methodology Findings: Categorized status (e.g. RISK_DETECTED, REPORTED_CLEAN (Verify in Code), NOT_ASSESSED).
  4. Prioritized Revision Roadmap:
    • Tier 1 (Critical Blockers): Must fix to prevent rejection or defense failure (e.g. re-split data by entity ID or enforce chronological ordering).
    • Tier 2 (Experimental Controls): Required ablations, baseline re-runs, and statistical tests.
    • Tier 3 (Textual Calibration): Tone adjustments, removal of overclaiming, and reference corrections.

Curated Memory & Lesson Proposals

To prevent unvetted assumptions from biasing future reviews:

  • LESSONS.md is a curated, human-verified repository of empirical review principles.
  • Agents must not automatically write unreviewed rules directly into LESSONS.md.
  • When an audit uncovers a novel failure mode or when human researchers provide corrective feedback, propose a candidate lesson under a "Lesson Proposal" section in the review output or draft it in LESSON_PROPOSALS.md for human review and approval.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.