agentleFS
Liberty91LTDLiberty91LTD/cti-skills/skills/quality-of-information-check/SKILL.md

quality-of-information-check

Skill35 starsChanged 48 days agoMITView on GitHub

Use when evidence needs grading before anyone reasons from it: \"how reliable is this?\", \"grade this source\", \"check this report\", \"is this corroborated?\", or a URL, pasted report or Liberty91 event handed over ahead of an ACH or assessment. Resolves each document to its originating source, splits it into claims and grades each on access level and claim support with a rationale and hard caps; not an Admiralty rating. Returns a graded claim table, gaps, a deception screen and JSON. Run first by /ach, /threat-assessment and /writing-assessments.

  • Reads credentials

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    If you know something about the incident that is not in the documents, it does not enter the grade.

    line 42
  2. Very specific:

    Run /source-provenance for each input.

    line 50
  3. Distinctive:

    Write the human-readable output and the JSON envelope.

    line 156
  4. Very specific:

    When /cti-orchestrator routes a bare URL here, or the user wants a quick read, return the summary, the graded claim table and the weakest …

    line 222

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

quality-of-information-check

Grade the evidence before you reason about it.

A structured analytic technique run by prompt alone produces output that has the shape of the technique and none of its discipline. An ACH matrix built on whatever the model felt like trusting is worse than no matrix, because it looks rigorous. This skill is the step that comes first: it establishes what the evidence is, where it came from, and how much weight each claim can bear.

Composition. Invokes /source-provenance, then /claim-extraction, then applies the rubric, then hands to /quality-control. Optionally invokes /lookup-* skills for corroboration and /lookup-liberty91 for platform data. Its output is the preferred input of /ach, /threat-assessment and /writing-assessments, and is consumed by /key-assumptions-check, /intelligence-writing, /stix-bundle and /ioc-export.

Three principles

  1. Grade provenance chains, not outlets. The grade belongs to this claim arriving by this chain. The same outlet carries a vendor's own telemetry finding and an attacker's boast in the same week.
  2. Grade claims, not events. One report contains observations, attributions and assessments of very different standing. One grade for all of them is meaningless.
  3. Data does the research, the model applies the rubric. The chain is resolved by the platform or by a script. Corroboration is counted from data. What is left for the model is labelled as model judgement.

Inputs

  • One or more URLs, pasted text, or Liberty91 event IDs.
  • First-party observations: hits from /lookup-sentinel or the user's own EDR. These enter the claim table directly, graded direct and established (rubric rule R12), with provenance_basis: first-party. Steps 1 to 4 do not apply to them.
  • Optional, and worth asking for: the question or hypothesis the evidence is meant to serve. With it, the skill can mark which claims are load-bearing.

What the model may and may not do

Allowed (model judgement, labelled as such)Not allowed
Judge transmission fidelityAssert a primary the script or platform did not resolve
Judge claim typeAssert corroboration not established by data or lookups
Apply the rubricInfer access type from an organisation's reputation
Write the rationaleInvent or paraphrase vendor confidence language
Analyse gapsAssign a grade above a cap
Run the deception screenApply a track record modifier outside Tier 1

If you know something about the incident that is not in the documents, it does not enter the grade. It can go in the gaps section as a lead to check.

Procedure

The order is fixed. Do not grade before the chain is resolved. Do not resolve corroboration after grading.

1. Resolve provenance

Run /source-provenance for each input. Record provenance_basis:

TierBasisMeaning
1platform-resolvedLiberty91 returned provenance fields for the event
2script-resolvedresolve_provenance.py resolved the chain for this article
3model-judgedPasted text, or a page that could not be fetched. Every provenance field is unverified

Tier 1 applies only when the platform response carries the provenance fields listed in /lookup-liberty91 (primaries[], independent_primaries, chains[]). A configured key is not enough. The platform also returns ratings of its own: an event-level credibility and verification status, a source_count, and a reliability letter and stance per source. These are not provenance fields and they are not grades. Do not copy them into grading, do not read source_count as independent_primaries, and do not treat a source with stance corroborates as corroboration. If they are all the platform returned, the run is Tier 2.

State the basis at the top of the output. When inputs differ, the run takes the worst. Items observed in the user's own telemetry carry first-party, which ranks above all three tiers.

For Tier 3, reconstruct only what the text itself says ("according to a report by X"). Record the organisation as named, the URL as null, and flag unresolved_provenance.

2. Extract claims

Run /claim-extraction:

  • on the primary document, where one was resolved and read;
  • on the article only for what it adds: attributed quotes, references to other primaries, and unattributed assertions (flagged press_originated).

If /source-provenance returned other_primaries or primary_cites, the claims found in or relayed from each belong to that organisation. Run a separate chain for any that matter to the question. A claim the primary relays from an organisation in primary_cites is graded on that organisation's document, not on the primary's access.

The cached text keeps table rows as lines with cells separated by |. A table row can be an anchor.

3. Judge transmission fidelity

For each claim from the primary, compare the article's wording with the primary's anchor sentence. Record one value per hop:

FidelityMeaning
faithfulSame assertion, same strength, same scope
caveats_droppedThe hedge, confidence level or scope limit is gone ("possibly linked to" became "attributed to")
embellishedThe outlet added strength, scale or detail the primary does not contain
not_carriedThe outlet does not repeat this claim. Not a fault, and no flag
unresolvedThe hop could not be read, so the comparison could not be made

Quote both wordings in the hop's notes. Where any hop is caveats_dropped or embellished, set flag caveats_dropped_in_chain and grade on the primary's wording.

The primary's own hop is always faithful: it is the origin. A user who gives you the primary directly has a one-hop chain with fidelity: faithful.

4. Establish corroboration

Per claim. Corroboration is counted by independent primaries with separate access, never by the number of articles.

TierHow
1Read independent_primaries from the platform. basis: platform.
2 and 3Default is independent_primaries: 1, basis: unchecked.

Not corroboration:

  • Vendor A citing vendor B.
  • A government advisory that restates vendor reporting without adding evidence.
  • Two outlets covering the same report.
  • A second primary linked from the same article, until you have established it had its own access.
  • A platform's event-level rating. verification: corroborated, credibility: 1 or a source_count above 1 says that several reports exist, not that independent primaries agree.

In Tier 2 and 3, offer to check. Where a claim contains indicators, run the applicable lookups (/lookup-virustotal, /lookup-otx, /lookup-misp, /lookup-opencti, /lookup-crowdstrike, /lookup-reversinglabs) and count a confirmation only where a lookup shows a different organisation reporting the same indicator from its own data. Record basis: lookups and list each source. An OTX pulse that cites the same vendor report is the same source.

Matching names across vendors. Hashes, CVEs and infrastructure match exactly, in every tier. Actor, malware and tool names do not: one vendor's APT28 is another's Sednit.

AvailableBehaviouralias_source
LIBERTY91_API_KEYResolve names through the Threat Library via /lookup-liberty91, including vendor-cluster merge recordsliberty91
references/aliases.ymlUse the user's own alias fileuser
NeitherNames are not matched across vendors. Corroboration for name-based claims stays at 1, uncheckednone

alias_source records what was used for the claim. Set it to none on every claim where no name matching was done, whatever is configured.

With neither, say so in the gaps section and name both options. Do not match names from memory. The file format is in references/aliases.example.yml.

5. Apply the rubric

Apply references/grading-rubric.md as written. The grade is two words, written out in full. It is not the Admiralty scale: never write it as a letter and a number, and never convert it to one. For each claim:

  1. Access level, from the stated access for the claim. Claim type does not change it.
  2. Claim support, from corroboration, the primary's stated confidence and consistency. For attribution and assessment claims, R13 fixes the level from the stated confidence.
  3. The caps, R1 to R13.
  4. A rationale of one to two sentences that names the access type, the corroboration count and basis, the stated confidence, and any cap that applied.

Tier 1 only: if the platform returns a track record score for the organisation and claim type, report it next to the grade and record track_record_applied. It moves the claim's footing by one band and never changes the access level. In Tier 2 and 3 the field is none.

A rationale that says only that the vendor is reputable is a failed rationale. It grades the name.

6. Compute recency and flags

age_days from date_observed to today. Where date_observed is unknown, from the primary's publication date. Set every flag in the schema's vocabulary that applies. single_source is set whenever independent_primaries is 1.

7. Run the deception screen

Apply references/deception-screen.md: MOM, POP, MOSES, EVE. Answer each briefly. Always run it in full for actor_claim items and for attribution that rests on artefacts an adversary could plant. Set deception_indicators_present on the claims a raised flag touches.

8. Derive the event aggregate

Compute event_qoi_summary from the items. Never assign it. Identify the weakest link: the lowest-graded claim among those the bottom line depends on.

9. Write gaps and next steps

  • What the evidence cannot tell us.
  • What corroboration was not checked, and why.
  • What would raise or lower each load-bearing grade.
  • Which lookups would corroborate which claims. Whether to wait for a vendor follow-up.

10. Emit and validate

Write the human-readable output and the JSON envelope. Save the JSON and run:

python3 skills/quality-of-information-check/scripts/validate_evidence.py <qoi.json> \
  --source-text <primary.txt> [--source-text <article.txt>]

The source text files are the ones /source-provenance wrote to the local cache. Exit 0 is valid. Exit 1 lists violations: fix them and rerun. Do not return output that fails validation. Then hand to /quality-control.

Output

# Quality of Information Check

**Provenance basis:** script-resolved
**Assessed:** 2026-09-27
**Question served:** Who is behind the intrusions at regional water utilities?
**Documents:** <article title> (outlet, date) → <primary title> (organisation, date)
**TLP:** CLEAR

## Summary
Two to four sentences in plain English. What is well supported, what is judgement,
what is unverified, and what the weakest link is.

## Graded claims
Load-bearing claims first.

| ID | Claim | Type | Primary | Stated access | Access level | Claim support | Corrob. | Flags |
|---|---|---|---|---|---|---|---|---|
| C02 | ... | observation | Northwind | ir_engagement | direct | firm | 1 (unchecked) | single_source |

### Rationale
- **C02: direct, firm.** ...

### Chain and fidelity
| Hop | Organisation | Published | Fidelity | Note |
|---|---|---|---|---|

## Event aggregate
| Field | Value |
|---|---|
| Claims graded | 9 |
| Independent primaries | 1 |
| Most recent observation | 2026-08 |
| Grade range | direct and firm, down to untraced and tentative |
| Summary line | core facts direct and firm, attribution direct and tentative, trend statement press-originated |
| Weakest link | C04 |

## Gaps
## Deception screen
## Next steps

## What in this output is model judgement
Claim typing, transmission fidelity, rationale wording, the deception screen and the
gap analysis are model judgements. The provenance chain, publication dates, the
primary's access and confidence sentences, and the classification of sources were
resolved by <platform | script>. Grades follow a fixed rubric; caps were checked by
validate_evidence.py.

The last section is mandatory and its content follows the tier. In Tier 3 it must say that the provenance chain is also a model reconstruction.

The JSON envelope is specified in references/evidence-item-schema.md.

Short form

When /cti-orchestrator routes a bare URL here, or the user wants a quick read, return the summary, the graded claim table and the weakest link, and offer the full output. Every step still runs, and the JSON is still validated. Only the presentation is shortened.

Common errors

ErrorCorrection
One grade for the whole reportGrade each claim
Grading the outletGrade the primary by access; grade the outlet on fidelity only
"Widely reported" treated as corroborationCount independent primaries
A vendor's attribution graded establishedAttribution is a judgement. R4 rules out established without a second primary, and R13 sets claim support from the vendor's stated confidence
Lowering the access level because the claim is a judgementAccess level follows access. The judgement is recorded in claim support
Copying the platform's credibility or reliability valueThose rate the event and the outlet. Grade the claim from the rubric
Writing the grade as a letter and a numberThe grade is two words. It is not an Admiralty rating and has no Admiralty equivalent
Actor's data-volume figure repeated as factactor_claim, graded adversary and unverified, until corroborated
Own telemetry left out of the table as "not a report"It is the top of the scale. Enter it as an observation graded direct and established
CVSS or EPSS entered as evidenceThey are attributes of a CVE, not claims about the event
Access assumed from the vendor's product lineAccess is what the document states. Otherwise undisclosed, which gives access level indirect at best
Filling an unresolved chain from memoryUnresolved is a finding. Cap and report

References

  • references/evidence-item-schema.md: the contract with every downstream skill.
  • references/grading-rubric.md: the rubric and the caps.
  • references/deception-screen.md: MOM, POP, MOSES, EVE with CTI prompts.
  • references/worked-examples.md: three full runs, one per tier.
  • references/doctrine.md: where each part of the method comes from. Use it when asked "why this grade".
  • references/aliases.example.yml: format of the user alias file.

Related skills

  • /source-provenance, /claim-extraction: the two steps this skill orchestrates.
  • /ach, /threat-assessment, /writing-assessments: prefer graded evidence and run this skill first by default. They run without it when the user asks, in a labelled ungraded mode capped at Moderate confidence.
  • /quality-control: validates the output.
  • /source-assessment: the Admiralty scale reference. The evidence grade is a different instrument and is not converted to or from it.
  • /confidence-levels, /likelihood-language: for the judgements built on this evidence.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.