llm-agent-security-audit
Skill3 starsChanged 32 days agoMITView on GitHub
Use when auditing an LLM, RAG, chatbot, or tool-using agent for prompt injection, data exfiltration, system-prompt leakage, or cross-tenant access.
In this repository22 other files it gives its agents.
- AGENTS.mdAGENTS.md
- skills/general/code-reviewer/AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- fastify-best-practicesSkill
- agent-toolkit-maintainerSkill
- onboard-repoSkill
- docker-patternsSkill
- accessibilitySkill
- code-reviewerSkill
- thermo-nuclear-code-quality-reviewSkill
- brag-slimSkill
- biome-formattingSkill
- instagram-public-researchSkill
- research-data-collectionSkill
- api-security-auditSkill
- cicd-security-auditSkill
- cloud-misconfiguration-auditSkill
- js-secrets-auditSkill
- security-auditSkill
- source-leak-auditSkill
- ariadne-prdSkill
- ariadneSkill
The file
We split this skill into the rules it gives an agent
6 rules. About 0 are common, 6 are distinctive, and 0 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈6
Distinctive · estimate
This author's own choices. Worth a look.
0
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 11
Test trust boundaries around model instructions, retrieved content, tools, and tenant data.
- Distinctive: line 25
Use synthetic canary data and a disposable account.
- Distinctive: line 27
Test indirect instructions in documents and web content for instruction/data confusion.
- Distinctive: line 34
Test cross-tenant retrieval with a value independently known to belong to a second synthetic tenant.
- Distinctive: line 36
Check tool permissions, argument validation, URL allowlists, confirmation gates, output encoding, and audit logs.
- Distinctive: line 53
Stop if a test reaches real data, sends an uncontrolled external request, or would mutate a real system.
≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
LLM and Agent Security Audit
Test trust boundaries around model instructions, retrieved content, tools, and tenant data. A refusal or unsafe sentence alone is not a vulnerability; require reproducible impact at the data or action layer.
When to Use
- Reviewing an agent with browser, shell, filesystem, email, or API tools.
- Assessing RAG/document ingestion, plugins, MCP tools, or multi-tenant chat.
- Investigating prompt injection, system-prompt leakage, or tool exfiltration.
Procedure
- Map system/developer/user/retrieved instructions, tool schemas, credentials, memory, tenant boundaries, and approval gates.
- Use synthetic canary data and a disposable account. Label every test input and keep external callbacks under the auditor's control.
- Test indirect instructions in documents and web content for instruction/data confusion. Verify whether the model can cause a tool action, not merely say it would.
- Apply the run-twice rule for alleged prompt or system-message leakage. A finding needs reproducible non-guessable content or a verifiable action.
- For exfiltration claims, require an authorized OOB callback containing only the synthetic canary. Rendered Markdown or a model claim is not proof.
- Test cross-tenant retrieval with a value independently known to belong to a second synthetic tenant. Do not use real customer data.
- Check tool permissions, argument validation, URL allowlists, confirmation gates, output encoding, and audit logs. Recommend architectural controls, not prompt-only mitigations.
Evidence Gate
Confabulation, jailbreak compliance, a model repeating user input, and a generic refusal bypass are not findings without a trust-boundary consequence.
Report
Include model/build, synthetic input, tool/data boundary, reproducibility, observable impact, logs or callback metadata, and mitigation. Never include system prompts, customer content, credentials, or harmful payloads verbatim.
Stop Conditions
Stop if a test reaches real data, sends an uncontrolled external request, or would mutate a real system. Revoke test credentials and preserve minimal logs.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.