agentleFS
natexai2026natexai2026/2030-skills/skills/grade-the-evidence/SKILL.md

grade-the-evidence

Skill0 starsChanged 4 months agoMITView on GitHub

Attach an explicit evidence grade to factual claims and keep "how confident" separate from "how well-supported." Use whenever assessing research, a study, a news claim, a report, a vendor's data, a statistic, or your own draft conclusions — any time someone asks "is this true / how solid is this / can we rely on this." Triggers on "how strong is the evidence," "is this study legit," "fact-check this," "what do we actually know," due-diligence reading, literature reviews, and before asserting any empirical claim as settled.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 3 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈3

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    If you catch yourself writing a confident sentence, check the tier of what it rests on.

    line 63
  2. Distinctive:

    If asked for a bottom line, give it — but built from the graded claims, not a vibe.

    line 76
  3. Distinctive:

    Don't grade rhetoric, grade the load-bearing claims: Not every sentence needs a tier — only the ones a decision rests on.

    line 111

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Grade the Evidence

Most bad analysis fails the same way: it states a shaky claim in the same confident voice as a rock-solid one. The reader can't tell which is which, so the whole thing gets trusted — or distrusted — as a block. This skill fixes that by forcing two separate judgments on every empirical claim: how strong is the evidence, and — independently — how confident should we be. High agreement among experts about thin data is not the same as strong data, and collapsing the two is how false certainty gets manufactured.

The core move: two axes, never one

Judge every load-bearing claim on both axes and report them separately:

  1. Evidence strength — what kind of evidence backs this, on the tier scale below.
  2. Confidence — how much the claim should move a decision, after accounting for the evidence tier, disagreement among credible sources, and the time horizon.

They come apart constantly. "Nearly every expert agrees, but no one has measured it" is high-agreement / low-evidence — report it that way, not as high confidence. Borrowed from how the IPCC keeps its evidence/agreement axis separate from its confidence axis, and from GRADE in medicine.

The six-tier evidence scale

Tag each empirical claim inline with the strongest tier it honestly earns:

  1. RCT / quasi-experimental — randomized trials, well-identified natural experiments, diff-in-diff with credible identification. The gold standard; still fallible.
  2. Observational-quantitative — correlational studies, large-N surveys, usage telemetry, economic indicators without causal identification. Real signal, no clean causation.
  3. Expert-consensus — convergent judgment across independent experts absent decisive data. Strong on agreement, weak on measurement.
  4. Theoretical / mechanistic — derived from theory or a formal model, untested at the relevant scale.
  5. Analogical — reasoning from precedent; must state the points of disanalogy (see the analogy-check skill).
  6. Speculative — coherent extrapolation with no direct support. Permitted only in explicit scenario/what-if framing, never as a load-bearing claim, and always visually marked as speculation.

Tiers 1–3 can anchor a confident claim. Tiers 4–6 cannot carry a verdict on their own.

The ⚠ Contested / Thin flag

Any claim resting on tier 4–6 evidence — or tier 1–3 evidence with material disagreement among credible sources — gets a ⚠ Contested/Thin flag plus one line saying which kind of problem it is:

  • Data doesn't exist yet (unmeasured) → resolvable by waiting for studies.
  • Data exists but is disputed (contested) → credible sources genuinely disagree.
  • Definitionally hard to measure (e.g. "died well," "understands") → may never resolve cleanly.

The remedy differs for each, so naming the type is not decoration — it tells the reader whether to wait, to weigh both sides, or to stop expecting a number.

The language rule that makes it bite

A verdict built on majority thin evidence may not use confident directional language. Ban "AI will…", "this proves…", "X causes Y." Use hedged, mechanism-anchored constructions instead:

  • "Early evidence suggests…" (tier 2, thin)
  • "The mechanism implies but has not confirmed…" (tier 4)
  • "Experts broadly expect, though no study has measured…" (tier 3)

If you catch yourself writing a confident sentence, check the tier of what it rests on. If the tier can't carry the confidence, the sentence is wrong — soften the claim or find stronger evidence.

Watch for confidence that outruns competence

A fluent, assertive register is not evidence. This is the single most common tell of a weak claim dressed up strong — in AI output and in human writing alike. Concretely: strong confidence markers ("clearly," "definitely," "obviously") correlate poorly with accuracy, and the dominant error is rarely a fabricated fact — it's interpretive overconfidence: stating an attributed opinion as settled fact, or generalizing a narrow finding into a broad one. When you see certainty markers, that's a prompt to check the tier underneath, not to relax.

Procedure

  1. Extract every load-bearing empirical claim (the ones a decision actually rests on — skip the throat-clearing).
  2. Tag each with its evidence tier (1–6) and a one-clause reason.
  3. Flag any tier 4–6 or disputed tier 1–3 with ⚠ Contested/Thin + the problem type.
  4. Set confidence per claim, explicitly noting where it's lower than the tier alone would suggest (disagreement, long horizon, small/unreplicated sample) — a single unreplicated study is tier 1 evidence but not yet a settled finding.
  5. Rewrite any sentence whose confidence outruns its tier.
  6. If asked for a bottom line, give it — but built from the graded claims, not a vibe.

Output template

CLAIM: <the specific empirical claim>
  Evidence: Tier <n> — <what kind, one clause>
  Flag: <none | ⚠ Contested/Thin — unmeasured|disputed|hard-to-measure>
  Confidence: <low|moderate|high> — <why, esp. if below the tier>
  Rewrite: <hedged version, if the original overclaimed>

Example

Input: "Studies show AI tutoring boosts learning outcomes 30%."

Graded:

CLAIM: AI tutoring raises learning outcomes ~30%.
  Evidence: Tier 1 for scaffolded systems (Harvard Kestin RCT, 0.73–1.3 SD);
            Tier 2/mixed for unscaffolded "just ask ChatGPT" use.
  Flag: ⚠ Contested/Thin — the "30%" collapses two different interventions.
        Scaffolded tutoring that withholds answers helps; unscaffolded
        answer-giving shows measured skill LOSS under proctored testing.
  Confidence: moderate, and only for scaffolded designs. Near-zero for the
              blanket "AI tutoring" version — the effect flips by design.
  Rewrite: "Scaffolded AI tutoring that withholds answers shows large RCT
           gains; unscaffolded answer-giving does not, and can reduce
           unaided performance."

The claim wasn't true or false. It was two claims wearing one number — and only grading the evidence separated them.

Gotchas

  • Don't grade rhetoric, grade the load-bearing claims. Not every sentence needs a tier — only the ones a decision rests on.
  • A single RCT is tier 1 but not settled. Unreplicated, small-sample, or single-site results get tier-1 strength but capped confidence. Say so.
  • Vendor benchmarks are self-reported. They're often tier 2 at best and conflicted; route capability/performance claims through the validate-the-claim skill.
  • "Experts agree" is tier 3, not tier 1. Agreement about unmeasured things is exactly the case this skill exists to catch.
  • Pairs with disaggregate. A single graded claim can still hide opposite effects across groups — grade and disaggregate for anything consequential.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.