LLM / AI Application Defense

Defense-in-depth for LLM-powered applications. Use during design and code review, or to remediate findings from llm-redteam. Assume any text the model reads may be adversarial and any tool it can call may be triggered by that text.

Core principle

The model is not a security boundary. Never rely on instructing the model ("do not reveal…", "ignore malicious requests") as your only control. Enforce security outside the model with real authorization, mediation, and least privilege.

1. Input mediation (untrusted-context handling)

• Clearly separate system instructions from untrusted data (user input, retrieved docs, tool outputs, files, web content). Delimit and label untrusted data; don't concatenate it into the instruction channel.
• Filter/normalize inputs for known injection patterns as a speed bump, not a guarantee.
• For indirect sources (RAG, web, email, files): validate provenance, strip active content, and treat all of it as data — never instructions.

2. Output mediation (insecure-output-handling defense)

• Treat model output as untrusted before it reaches any sink.
• Contextually encode/validate for the destination: HTML-encode before browser render, parameterize before SQL, never pass to a shell/eval, validate before a downstream API call.
• Enforce output schemas (structured output / function-call schemas) and reject non-conforming responses.

3. Least-privilege tools & agency (excessive-agency defense)

• Give the model the minimum tools and scopes needed; no ambient credentials.
• Allow-list actions; deny by default. Parameter allow-lists on each tool.
• Human-in-the-loop confirmation for high-impact/irreversible actions (payments, deletes, external messages, code execution).
• Per-action authorization checked in the tool implementation against the end user's permissions, not the agent's.
• Sandbox code execution and file/network access; time and resource caps per action.
• Bound agent loops (max steps, max cost) to prevent runaway execution.

4. Data protection (sensitive-disclosure defense)

• Minimize sensitive data placed in context; retrieve only what's needed, scoped to the user.
• Per-user/tenant isolation across history, memory, and RAG (see references/deployment-checklist.md).
• Output filters for secrets/PII; redact before returning.
• Keep secrets and sole security logic out of the system prompt (system-prompt-leakage defense).

5. RAG & knowledge-base hygiene

• Access-control retrieval per user/tenant at query time.
• Validate and provenance-track ingested documents; isolate user-contributed content.
• Treat retrieved content as data; monitor retrieval for anomalies. (Details in the llm-redteam skill's rag-security.md.)

6. Rate limiting & cost controls (unbounded-consumption defense)

• Per-user token/request quotas, output length caps, and concurrency limits.
• Cost monitoring and circuit breakers; alert on spikes (denial-of-wallet).
• Loop/recursion caps on agents and tool chains.

7. Monitoring & detection

• Log prompts, tool calls, retrievals, and outputs (with PII/secret redaction) and correlation IDs.
• Alert on: injection canary hits, unusual tool-call patterns, cross-tenant retrieval attempts, cost spikes, and output-filter triggers.
• Feed incidents back into the red-team eval suite.

8. Assurance: red-team CI

• Run the llm-redteam eval harness in CI on every prompt/model/tool change.
• Gate deploys on attack-success-rate thresholds; keep the regression suite at 0%.
• Re-run after any dependency/model upgrade (supply-chain defense).

9. Supply chain (LLM03 defense)

• Pin and verify models, adapters, plugins, and datasets; track provenance.
• Monitor third-party components for compromise; re-test on updates.

Standards mapping

• OWASP LLM Top 10 (2025): controls above map to LLM01–LLM10.
• NIST AI RMF: Govern (policy, roles), Map (context/threats), Measure (eval harness/ASR), Manage (monitoring, response).
• MITRE ATLAS: use for threat vocabulary and to ensure defensive coverage of known techniques.

See references/defense-checklist.md and references/deployment-checklist.md.

<sub>© 2026 noahfranklin · Part of the Gideon security suite · MIT License · Original work.</sub>