ast05-untrusted-external-instructions
Skill2 starsChanged 3 months agoApache-2.0View on GitHub
Detect and triage OWASP AST05 Untrusted External Instructions — skills that fetch external documentation (URLs, runbooks, schemas) and follow it as instruction rather than treat it as data, including Author Rug-Pull, Reviewer Bait-and-Switch, transitive reference chaining, and relay-node amplification in multi-skill chains. Use when a skill's referenced content has no hash pin, when a reviewed skill's external dependency could have changed since review, when triaging a skill-chain injection that no single node's review caught, or when deciding whether a finding belongs here versus AST01 or AST02.
In this repository10 other files it gives its agents.
The file
We split this skill into the rules it gives an agent
4 rules. About 0 are common, 2 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈2
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 58
Read *incomplete review* as a qualifier on the verdict, never as leave to withhold one.
- Very specific: line 168
vs AST01: if the payload sits in the shipped SKILL.md, it's AST01.
- Very specific: line 208
The instruction-channel name list is narrow on purpose: Only names that denote the channel itself (prompt, messages, instructions, …
- Distinctive: line 250
Only the dataflow check sees that one, and only if the first NEVER above was honoured.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AST05 - Untrusted External Instructions
Unofficial community implementation of the OWASP® Agentic Skills Top 10 standard, v1.0. NOT an OWASP project; no OWASP endorsement. OWASP is a registered trademark of the OWASP Foundation, used here descriptively to identify the standard implemented. Source: https://owasp.org/www-project-agentic-skills-top-10/
Pattern: Knowledge. The category exists because textual external references have none
of the tooling that code dependencies already have — no hash-pin field, no lockfile
for prose, and signing the skill package says nothing about what a URL returns at
runtime. Mechanism lives in scripts/detector.py: a call-site dataflow scan that
follows a fetched response body to the agent's instruction channel or to an
executable sink, a gated check for a declared instruction-versus-data convention,
and two reads of how wide the declared fetch surface is. Every one of them is an
artifact_signal proxy, never coverage of a named AST05 scenario — frozen scenario
tiers and the reason for that live in coverage-matrix.md.
Orientation — read this much first
Fires when a skill fetches external documentation and treats it as instruction, when referenced content carries no hash pin, when a reviewed skill's external dependency could have changed since review, or when a chain injection slipped past every single-node review.
Decides: nothing at scenario level. Zero of AST05's six scenarios are statically decidable
from one package, so this category publishes no scenario-level F1. One of the six,
AST05-S05 Malicious Instructions Embedded in Documents, is tiered agent-judgable — its
evidence is in the package, but deciding it needs semantic judgement rather than a rule,
so it yields no automatable number either. What ships instead is five
enabling-precondition checks, each labelled artifact-signal-only, and that label travels
with every number they produce.
Does not decide: what a URL serves now, what it will serve at run time, or what it serves to someone else. That is the whole category — read Why review-time inspection cannot close this category before you argue with it.
How much freedom this page grants, and the one place it grants none. Nearly all of AST05 is judgement — a property of the category, not a gap in this file — so the exception is worth naming before the rules rather than after them:
| Work | Freedom | Why |
| Labelling a number one of these checks produced | Low — the label is looked up, not argued | CHECK_COVERAGE marks all five checks artifact-signal-only, STATIC_DETECTABLE is empty, and scenarios/registry.yaml places every AST05 scenario outside the static tier. Promoting a proxy to coverage is the one error this category cannot absorb, because no scenario-level number exists underneath it to contradict the claim later. Where this file and coverage-matrix.md disagree about a tier, the matrix wins and this file is the bug. |
| Deciding what a result means | High — this is the work, and rules 1–6 supply its questions, not its answers | Whether a fetch surface is too wide for the function, whether a named boundary does anything, what a clean run over a package that fetches through a shell is worth: nothing in this repository closes any of them. A page that implied a rule here would be manufacturing certainty the category does not have. |
A clean run of the five checks with no manual step beside it is an incomplete review, not a negative finding — and the steps that bound it belong to the same deliverable, not to a follow-up. Walk Where the shipped checks go quiet and name every bullet that applies to the code in front of you: a single named step leaves the remaining blind spots reported as surfaces the checks covered, which is the same over-claim as reporting the clean run bare.
Read incomplete review as a qualifier on the verdict, never as leave to withhold one.
The answer still ends in the verdict the question asked for; a reply that names the manual
steps and stops has reported this page's machinery and kept the finding, which is the bare
clean run's failure arriving from the other side and reads downstream as a pass either
way. Name the steps against the code, too — the boundary function by its own name and the
file it is defined in, the manifest's own network block — because a limit stated in this
file's vocabulary describes the check, and only the package's own strings let a reader
check the check.
Route first. Reasoning for each cross-category call is in Distinguishing AST05 from its neighbors; this table is only the jump.
| What you are holding | Go to |
| A payload in the shipped SKILL.md | AST01 — wrong skill; there is nothing here yet to review |
| The identical payload in content the skill only references | Stay — rules 1–4 |
| A skill chain where no single node's review caught it | Rule 5 |
| A hash pin checked once at install | Rule 1 — a snapshot, not a control |
| An unverified integrity claim over a code dependency | AST02 — signed package graph, not referenced prose |
| The skill's own version changing | AST07 — same drift, different object |
| Content a scanner may never observe at all | AST08 — this category widens their gap |
A sanitize() the static check accepted | Where the shipped checks go quiet, first bullet — highest-yield manual step on this page |
An audit tier column reading static-detectable for these five checks | Why review-time inspection cannot close this category, last paragraph — that column reports mechanism, not coverage |
Stop after Decision rules if you are writing a control recommendation; rules 1–6 are the controls. Read Where the shipped checks go quiet only when a check came back clean and you must decide whether that clean result means anything. NEVER is the last stop before a number or a clean result leaves this page.
Layer 3 — load on condition, never by default.
coverage-matrix.md(337 lines) — load before re-tiering a scenario, quoting a coverage claim, or reporting any number for this category (the proxy corpus figure lives there, and it is not scenario coverage). Do NOT load it to route a finding.scripts/detector.py(688 lines) — load before reproducing a dataflow false positive or negative, or extending the boundary-name or sink-name sets. Do NOT load it to learn what a check covers:CHECK_COVERAGEis reproduced incoverage-matrix.md, and reading the code to infer coverage is exactly how a proxy gets promoted.- Do NOT load either to decide whether a referenced URL is malicious. Neither file can answer that, and no amount of reading will change it.
Why review-time inspection cannot close this category
The reviewed object and the executed object are different objects, and no control in the skill format connects them. A package can be reviewed, signed and hash-pinned; the documents it points at cannot, because the format has no hash-pin field for prose and no lockfile for a URL. A skill that passed review yesterday and points at a URL its author still controls is silently rewritten today with no version bump, no re-signature, and no diff anywhere in the artifact — the Author Rug-Pull shape.
That is why this category's static-detectable tier is empty and why this package
publishes no scenario-level F1 at all. It is not a gap in the detector. Every one of
AST05's six scenarios needs evidence the artifact cannot hold: the referenced document
at two points in time, or from several fetch vantage points, or the chain actually
followed, or the pipeline's per-node backbone models. The five checks that do ship read
enabling preconditions and are declared artifact-signal-only — a label that travels
with every number they produce, so no consumer can quietly promote a proxy into
coverage.
Two commands here answer two different questions about those same five checks, and taking
one for the other is how the promotion happens anyway. cli.js audit <path> reports a
check's mechanism: each one is a deterministic rule over the package's own bytes, so
its tier column reads static-detectable for all five, and the footer beneath the table
says these are checks and not scenario coverage. cli.js coverage reports this
category's position — static tier 0, artifact-signal-only 1.00 (n=6). Both are
true of the same code: one says how a check works, the other says what it may claim. Quote
coverage.
Decision rules
- A hash pin without a re-verify-on-every-load step is a snapshot, not a control. Record a content hash for referenced material at review time, then re-check it immediately before model ingestion on every subsequent load — refusing content that is unpinned or has drifted. A pin checked only once, at install, degrades to the exact "reviewed skill is never the skill that runs" failure this category exists to prevent.
- A redirect to an unreviewed resource is a verification failure, not a pass-through. Treat a missing reference the same way — log the resolved source, version, and content digest at the moment of failure so the specific drift is auditable, not just "content unavailable."
- Bait-and-switch defeats review by keying on the reviewer, not the content. A URL can serve clean documentation to reviewers, scanners, and crawlers (by IP, user-agent, or timing) while serving malicious instructions to a live agent run. The decision consequence: a scan result is evidence about what the scanner's fetch saw, not evidence about what any given agent's fetch will see — a passing scan is not proof of a clean live path.
- Reference-following must be complete or explicitly bounded — a partial audit is a false negative generator. Transitive Reference Chaining means an attacker needs to control only one link buried deep in a chain the agent follows, past where a shallow review stopped. If reference auditing has a depth limit, that limit is itself a declared scope boundary that must be recorded, not a silent truncation.
- A skill chain's injection resistance is the minimum over its nodes, and it does not compose across hops. When one skill's output feeds the next (intake → triage → an action-taking skill), each node re-parses upstream output with its own instruction-vs-data boundary and its own backbone model. Some nodes neutralize an injected payload; others pass it on. The attacker needs exactly one weak relay. The consequence for review: certifying every endpoint of a chain does not certify the chain — the minimum-resistance node, not the average, sets the chain's actual exposure, and that node can differ per model swap even with the skill definitions held fixed.
- Prefer inlining a snapshot over fetching live, and treat "must stay current" as a distribution problem, not a fetch problem. Snapshotting external documentation into the signed skill package at publish time makes it reviewable and pinnable like any other package content. When content genuinely needs to stay current, route updates through a controlled, auto-updating marketplace channel with its own review and provenance — not by pointing a skill at an arbitrary, unmanaged URL.
Distinguishing AST05 from its neighbors
- vs AST01: if the payload sits in the shipped SKILL.md, it's AST01. If a malicious author places the identical payload in content the skill merely references, keeping the shipped skill body clean, that's AST05 specifically because the skill passes install-time review — there is nothing malicious to review yet at that point.
- vs AST02 (Supply Chain Compromise): AST02's integrity controls (hash pins, signed digests) reach code dependencies the skill declares and installs. AST05 is the documentation a skill points to at runtime — a surface those same integrity tools do not reach, because the referenced content was never part of the signed package graph in the first place.
- vs AST07 (Update Drift): AST07 is the skill's own version changing. AST05 is the identical drift phenomenon applied to referenced content the skill points to, which can change while the skill itself stays pinned, unchanged, and passing every version-based integrity check.
- vs AST08 (Poor Scanning): externally referenced content can be absent, or served differently, at scan time versus run time (the Reviewer Bait-and-Switch mechanism above) — this widens AST08's detection gap specifically to content a scanner may never actually observe, regardless of how good the scanner's own analysis is.
Where the shipped checks go quiet
The two dataflow checks parse Python with ast and match call nodes — never source
text. That choice is load-bearing: a regex for requests.get matches this module's own
pattern tables and every fixture literal quoted in the tests beside it, so a text-based
version of these checks convicts the scanner itself. It also fixes the boundaries of
what they can see.
- A declared boundary and an implemented boundary are indistinguishable here. The
taint is cleared by a call to any name in the boundary set (
sanitize,quarantine,as_reference_data,tag_untrusted, …) or by a provenance marker in a nearby string literal. A function namedsanitize()whose body isreturn textclears the finding completely. At the static layer, naming the control is the control — verifying that it does anything is a manual step, and it is the highest-yield one on this whole page. - Taint does not cross a function boundary or a file. Fetch in one helper, sink in another, and both checks clear. Same for a fetch whose response is stored on an object attribute and read back later.
- Python only, and an unparsed file is an INCOMPLETE that reads as a pass. A
JavaScript or shell fetcher produces no nodes at all. A
.pyfile that fails to parse is counted and named in the evidence string, but the verdict is still negative — read the evidence, not the boolean. - The instruction-channel name list is narrow on purpose. Only names that denote the
channel itself (
prompt,messages,instructions,system_prompt, …) are sinks.rules,context,systemandpersonawere removed after they fired on a correctly-written clean fixture. A skill whose channel is spelled anything else is uncovered — precision was bought with recall here, deliberately. json.loads(response.text)is not a sink and must not be made one. Parsing a body as data is the correct handling; convicting it would fire on every well-written HTTP client in the corpus and destroy the check's usefulness.- The absent-boundary check is gated on a Python fetch call site. A package that fetches only through a shell command never opens the gate, so the boundary question is never asked of it.
Scope and out-of-artifact boundary
Whether a given referenced URL currently serves clean or malicious content is a live,
time-varying fact about an external host — not a property this artifact can determine
once and cache as ground truth. None of AST05's six named scenarios is decidable from
a single skill package, so this category publishes no scenario-level F1 at all. A
detector can check whether a control exists — an unpinned reference, a fetched body
reaching an instruction or executable sink, an absent boundary convention, an unbounded
fetch surface — without being able to certify what the referenced content currently
says on the open internet, or that the author will edit it after review. Those checks
are declared artifact-signal-only for exactly that reason. The exact split, and the
number the proxy corpus does publish, are fixed in coverage-matrix.md.
NEVER — how a proxy result becomes a claim it cannot carry
- NEVER accept a boundary you have not read. Taint clears on a call to any name in
the boundary set (
sanitize,tag,escape,as_reference_data, …), matched by attribute name and never resolved, and it clears again on any string literal in the same expression that the marker regex matches — whose alternatives include the bare wordsuntrustedandprovenance.prompt += "untrusted" + fetchedcomes back clean; so doessanitize(fetched)whose body isreturn text, and assigning through that name un-taints the value for the executable-sink check downstream as well. At the static layer, naming the control is the control. Skip the manual read and the category's two strongest checks become a spelling test the author always passes. - NEVER read a clean
AST05-absent-instruction-boundaryas adherence to a convention. The check concatenates every.mdin the package and clears on one regex hit anywhere in it: aCHANGELOG.mdline reading "fixed a typo in the word provenance" clears the whole package. Presence of a convention is weaker than adherence to one, andcoverage-matrix.md's open debt 2 names the shape that separates them — a package whose prose declares the control its code then ignores. Only the dataflow check sees that one, and only if the first NEVER above was honoured. - NEVER read the two fetch-surface checks as observations of what the package
fetches.
AST05-unrestricted-network-fetchandAST05-wildcard-domain-allowlistreadmanifest.permissions.networkandpermissions.tools, and nothing else. A package with no permissions block at all returns clean with the evidence "no network block to widen (network=None)" while a bundled script callsrequests.getfreely — an undeclared fetcher is the quietest possible pass. Declaration against code is AST04'sdetect_permission_understating; if the finding is not routed there, no check in this repository reads the call sites against the manifest. - NEVER read a clean
AST05-wildcard-domain-allowlistas a valid allowlist. The check reports*and bare-TLD entries such as*.com, and deliberately spares*.example.com, which at least bounds an organisation. Butvalidators/usf.py'shost_errorsrejects every wildcard under USF v1 host-only matching — a wildcard subdomain "would never match and reads as broader access than is actually granted". The same field can therefore fail validation and pass this check, so quoting the check's "allow-list scoped" evidence as conformance reports the opposite of the validator's answer. - NEVER say this category's checks looked at the hash pin. Rule 1's control is a pin
plus re-verification on every load, and no check in
scripts/detector.pyreads one:scenarios/registry.yamlroutes AST05-S01'sartifact_signaltoAST01-content-hash-missing, a check that lives in AST01's module. The registry also closes the inverse shortcut in its own words — a pinned reference can still be rug-pulled once the operator accepts the new hash, and an unpinned one may never be edited at all. Unread here and insufficient there: a clean AST05 run has verified nothing about pinning, and reporting otherwise retires the one control rule 1 asks for. - NEVER cite
AST05-absent-instruction-boundaryabout a relay hop. Its gate is a network fetch call site. A skill consuming a prior skill's output over stdin or a handoff file — the relay shape of rule 5 and of AST05-S04 — never opens the gate and returns "package declares no fetch call site, so there is no external-content hop to bound". The scope choice was deliberate (coverage-matrix.mdopen debt 3: an ungated stdin source would have fired onskills/advisory, which readssys.stdin), which makes the clean result correct and the citation wrong. The minimum-resistance node in a chain is precisely what this check never looks at. - NEVER let
artifact-signal-only 1.00 (n=6)stand for all five checks. Three of them have a labeled pair.AST05-unrestricted-network-fetchandAST05-wildcard-domain-allowlistfire on none of the six fixtures — every AST05 fixture declares a bounded, single-host allowlist — and their two-sided evidence lives only inscripts/test_ast05_detector.py. A number measuring three checks, reported as the category's, is the same promotion the freedom table forbids, arriving through the denominator instead of the label, in the one category whose entire claim to honesty is that its numbers say what they measured.
References
Full attack-scenario catalog (including Malicious Instructions Embedded in Documents, resource-exhaustion DoS) and the complete preventive-mitigation list are the whitepaper's own AST05 section (source: the OWASP Agentic Skills Top 10 publication, section AST05 (no local copy: this repository points at the publication rather than bundling it, so the authority chain cannot drift from a stale copy; the whitepaper is CC BY-SA 4.0)). This file is the delta on top of it.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

