source-reliability
Skill1 starsChanged 4 months agoMITView on GitHub
Rank evidence by study design and track record so the strongest source drives the answer, and disagreements between sources are surfaced rather than averaged away. Use whenever an agent weighs multiple sources of differing quality.
In this repository8 other files it gives its agents.
- citation-faithfulnessSkill
- claim-extractionSkill
- conflict-detectionSkill
- evidence-coverageSkill
- meta-analysisSkill
- prisma-searchSkill
- query-planningSkill
- rag-injection-defenseSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 1 is distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 39
When high-quality sources point different directions, the honest output is not a blended number — it is the split, shown:
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Source Reliability
Not all evidence is equal, and treating it as equal is how a preprint ends up outvoting a Cochrane review. This skill gives the agent an explicit hierarchy, so the weight of a claim tracks the strength of what backs it — and so conflicts between sources are shown to the reader, not silently resolved.
When to use
Whenever the answer rests on more than one source and those sources differ in quality, recency, or conclusion.
The rule
Weight each source by its design and record. When strong sources disagree, surface the disagreement — do not average it into a false consensus.
A working hierarchy (strongest first)
- Systematic reviews / meta-analyses of RCTs (e.g. Cochrane)
- Individual well-powered RCTs
- Cohort and case-control studies
- Case series / case reports
- Mechanistic, in-vitro, or animal studies
- Expert opinion, narrative reviews
- Preprints and unreviewed sources — usable, but flagged as unreviewed
Design is the starting point, not the whole story. Adjust for:
- Directness — does the study answer this question, in this population?
- Recency — has newer, stronger evidence superseded it?
- Record — retractions, failed replications, or known conflicts of interest lower trust.
- Consistency — does it agree with the rest of the strong evidence, or stand alone?
Surfacing conflict
When high-quality sources point different directions, the honest output is not a blended number — it is the split, shown:
The strong evidence is divided:
• Meta-analysis of RCTs (2024) → benefit (RR 0.85 [0.78–0.93])
• Large RCT (2026, newer) → no effect (RR 0.99 [0.90–1.09])
Likely reason: the newer trial used a different endpoint / population.
A reader should weigh recency and directness, not an average of the two.Averaging these would erase the most important fact about the evidence: that it is unsettled.
What to put on the answer
Attach a compact trust signal to each claim or to the answer as a whole:
- the tier of the strongest supporting source,
- whether refuting evidence exists and how strong it is,
- and a one-line reason the top source was trusted.
The reader should be able to see why the answer leans the way it does, and change their mind if they weigh the sources differently.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.