agentleFS
Sign inStart free
vinayaklatthevinayaklatthe/microsoft-security-skills/skills/azure-ai-content-safety/SKILL.md

azure-ai-content-safety

Skill179 starsChanged 5 months agoMITView on GitHub

Guidance for Azure AI Content Safety — programmatic content moderation for text, images, multimodal, and generative AI guardrails. Covers Content Safety categories (hate, violence, sexual, self-harm) with severity levels, Prompt Shields for jailbreak and indirect prompt injection detection, groundedness detection (hallucination check vs grounding sources), protected material detection (text and code), custom categories, blocklists, Content Safety Studio for tuning, integration with Azure OpenAI as input/output filters (built-in plus custom), latency/cost trade-offs, agent guardrails, and deployment patterns (sidecar in app, integrated with Azure OpenAI). WHEN: Azure AI Content Safety, prompt shields, jailbreak detection, indirect prompt injection, groundedness check, protected material detection, content moderation, generative AI guardrails, harmful content categories, custom blocklist AI, agent guardrail. DO NOT USE for AI workload threat detection / SOC alerts (use defender-for-cloud-ai), end-user AI usage governance (use purview-ai-hub), or general AI app design unrelated to safety.

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 2 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Start with default Azure OpenAI content filters: Every Azure OpenAI deployment has a default Content Safety filter at *medium severity* …

    line 45
  2. Distinctive:

    Groundedness detection for RAG apps: After the model produces a response, call groundedness detection with the source chunks.

    line 60

≈ “Common” is an estimate: at least 1 in 5 of the 308,232 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Azure AI Content Safety

Azure AI Content Safety is the platform for real-time content moderation and generative AI safety. It analyzes text, images, and multimodal content for harm categories, blocks prompt-injection / jailbreak attempts (Prompt Shields), detects ungrounded responses (Groundedness), and flags protected text/code reproduction.

It pairs with Azure OpenAI as built-in content filters (default safety net) and as configurable guardrails for any AI/LLM application (including non-Microsoft models).

When to use

Building or hardening any generative AI application — chat assistant, agent, RAG app, multimodal product — that needs guardrails on inputs and outputs.

Do not use this skill for SOC threat detection on AI workloads (defender-for-cloud-ai), end-user AI usage governance (purview-ai-hub), or generic AI application architecture.

Capability map

CapabilityWhat it doesWhere to apply
Text moderationClassifies text into hate, violence, sexual, self-harm (severity 0–7)Input + output
Image moderationSame categories on imagesUser uploads, generated images
Multimodal moderationText+image combined contextMultimodal chat
Prompt Shields — User PromptDetects jailbreak attempts in the user promptInput
Prompt Shields — IndirectDetects injection in untrusted grounding content (RAG, tool results, web pages)Pre-grounding step
Groundedness detectionChecks LLM response against grounding sources for hallucinationOutput
Protected material — textDetects verbatim copyrighted textOutput
Protected material — codeDetects reproduction of public-repo codeCode-generation output
Custom categoriesCustomer-defined harm categories (e.g., off-brand topics)Input + output
BlocklistsExact-match + regex blocklistsInput + output

Approach

  1. Start with default Azure OpenAI content filters. Every Azure OpenAI deployment has a default Content Safety filter at medium severity across all four harm categories. Don't deploy without it.
  2. Tune severity thresholds per use case.
    • Children's product: low threshold (block at severity 2).
    • Adult creative app: high threshold (block only at severity 6). Author a custom content filter in Azure OpenAI Studio; assign to deployment.
  3. Enable Prompt Shields for any user-facing chat or agent.
    • User Prompt Shield on the incoming prompt.
    • Indirect Prompt Shield on grounding content (RAG document chunks, tool outputs, fetched web pages) before they hit the model. Indirect injection is the higher-impact, lower-visibility risk.
  4. Groundedness detection for RAG apps. After the model produces a response, call groundedness detection with the source chunks. Flag/reject ungrounded claims. Adds latency (one extra model call) but is the main defense against hallucination in high-stakes apps (medical, legal, financial).
  5. Protected material detection for products that generate text or code at scale:
    • Text: SaaS copywriting tools.
    • Code: developer assistants (Copilot-style). Flag and either suppress or annotate the output.
  6. Custom categories + blocklists for brand- or domain-specific risks (competitor mentions, regulated disclosure language, internal codenames). Train custom categories via Content Safety Studio with ~100 examples per category.
  7. Architecture pattern (sidecar / middleware).
    • Request → Prompt Shield → (LLM call) → Output classifiers (harm, groundedness, protected material) → Response.
    • Fail-closed: if Content Safety call errors, do not return uncontrolled LLM output.
    • Log every decision (allowed/blocked/flagged) for audit and tuning.
  8. Performance. Each guardrail adds 50–300ms. Parallelize input and output checks; short-circuit on first hard block. Budget for the latency in user expectation.
  9. Operate. Weekly review of blocked-content samples (with privacy controls). Tune thresholds, custom categories, and blocklists. Track false positive / false negative rates per category.

Guardrails

  • Default filters are baseline, not a complete safety strategy. Prompt Shields, groundedness, and custom categories close the real gaps.
  • Don't expose harm-category scores to end users. A "you said X violence score 6" message helps attackers tune jailbreaks.
  • Fail-closed on Content Safety errors. A 500 from the service should not result in unmoderated LLM output reaching users.
  • Indirect prompt injection is the under-appreciated risk. A user's prompt is visible; a malicious string in a fetched web page or document is not. Always shield grounding content.
  • Groundedness detection requires you to send the grounding sources. RAG apps natively have them; pure chat apps don't.
  • Custom categories are not a substitute for product policy. They detect; they don't prevent the user from generating harmful content with another tool.
  • Protected material detection is best-effort. Copyright/IP review is a legal process, not an API result.
  • Content Safety is per-call billed. High-traffic apps need cost modeling; not all features run on every call (groundedness is expensive — gate on use case).
  • Pair with Defender for Cloud AI for SOC-side detection of attempts and patterns (defender-for-cloud-ai).

Common anti-patterns

  • "Default filter is enough" — no jailbreak/injection protection.
  • "Indirect prompt shield skipped because 'all our sources are trusted'" — they're trusted to be there; not trusted to be uncompromised. RAG corpora get poisoned.
  • "Groundedness on every chat call regardless of grounding" — burns cost on pure chat without grounding sources to check against.
  • "Logged user prompts in plaintext for tuning" — privacy/PII exposure. Pseudonymize.
  • "Returned the Content Safety severity score in error messages" — gives attacker a tuning oracle.
  • "Custom categories trained on 5 examples" — no model lift. Need ~100 minimum.
  • "Blocklists used as the main strategy" — misses paraphrases. Combine with classifier categories.
  • "Content Safety only on output" — input attacks (jailbreaks) reach the model.

Example prompts

  • Architect Content Safety guardrails for a RAG-based customer support assistant on Azure OpenAI.
  • Configure Prompt Shields for a Foundry agent with internet-grounded tools.
  • Tune content filter severity thresholds for a teen-audience product.
  • Add groundedness detection to flag hallucinations in our medical Q&A assistant.
  • Train a custom category for "competitor mention" and integrate as output blocklist.
  • Cost and latency model: Content Safety calls per chat turn on a 10K-user app.
  • IR runbook: how SOC handles "prompt shield blocked attempted jailbreak" pattern (paired with Defender for Cloud AI alert).

Microsoft Learn

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

Reviews can't be read right now.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free