ai-foundry-posture-check
Skill70 starsChanged 4 months agoMITView on GitHub
Assess security, reliability, and cost posture of Azure OpenAI and Microsoft Foundry deployments. Detects critical anti-patterns: API keys instead of Managed Identity, public endpoints without firewall, disabled content filtering, missing diagnostic settings, deprecated model versions, over-provisioned PTU, absence of AI Gateway (APIM) for rate limiting, shared dev/prod deployments, quota exhaustion risk, 429 throttling rates, runaway token consumption, and single-region deployment. Use when asked about AI workload security, OpenAI best practices, Foundry health check, model deployment review, AI cost optimization, quota monitoring, or throttling analysis.
In this repository7 other files it gives its agents.
The file
We split this skill into the rules it gives an agent
10 rules. About 0 are common, 7 are distinctive, and 3 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈7
Distinctive · estimate
This author's own choices. Worth a look.
3
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 49
If no results, try:
- Distinctive: line 57
If no Cognitive Services accounts exist, report "No Azure OpenAI or AI Foundry resources found" and end assessment.
- Distinctive: line 185
Check if OpenAI accounts exist in more than one region (disaster recovery / latency optimization):
- Very specific: line 223
Use execute_kusto_query with the Log Analytics workspace connected to the OpenAI account's diagnostic settings.
- Distinctive: line 276
Only check if PTU/Provisioned deployments exist:
- Distinctive: line 284
If PTU deployments exist, check utilization via metrics:
- Very specific: line 345
Use execute_kusto_query with the Log Analytics workspace.
- Distinctive: line 416
If the user provides a list of accepted exceptions, do not flag those items.
- Distinctive: line 426
When exceptions are provided:
- Very specific: line 467
Use GetAzCliHelp to validate the command syntax before suggesting
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
AI Foundry & OpenAI Posture Check
Purpose
Assess the security, reliability, and cost efficiency of Azure OpenAI and Microsoft Foundry deployments. Detects the most common anti-patterns that startups make when building AI-powered products — from exposed endpoints to runaway token costs.
Based on the Azure Well-Architected Framework for AI workloads and Microsoft Foundry operational best practices.
When to use this skill
- User asks "is our OpenAI deployment secure?"
- User asks about AI cost optimization or token consumption
- Review before going to production with an AI feature
- User asks about content filtering, model versions, or rate limiting
- Periodic AI workload health check
Pre-check
Confirm with the user:
- Which Azure OpenAI / Cognitive Services accounts to assess (or "all in subscription")
- Whether they use PTU (Provisioned Throughput) or Standard deployments
- Whether they have production AI workloads already live
Assessment procedure
Step 0: Discover AI resources
az cognitiveservices account list \
--subscription <sub-id> \
--query "[?kind=='OpenAI' || kind=='AIServices'].{name:name, kind:kind, rg:resourceGroup, location:location, sku:sku.name}" \
-o tableIf no results, try:
az cognitiveservices account list \
--subscription <sub-id> \
--query "[].{name:name, kind:kind, rg:resourceGroup, location:location}" \
-o tableIf no Cognitive Services accounts exist, report "No Azure OpenAI or AI Foundry resources found" and end assessment.
For each account found, run the following checks:
🔐 CATEGORY 1 — Security (Critical)
Check 1.1 — Managed Identity enabled (not API keys only)
az cognitiveservices account show \
--name <account> --resource-group <rg> \
--query "{identity:identity.type, disableLocalAuth:properties.disableLocalAuth}" \
-o json| Finding | Severity | Score |
identity.type = SystemAssigned/UserAssigned AND disableLocalAuth = true | ✅ Pass | 12 pts |
identity.type set but disableLocalAuth = false | ⚠️ Partial | 6 pts |
identity.type = None or null | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity
Check 1.2 — Network isolation (firewall or private endpoint)
az cognitiveservices account show \
--name <account> --resource-group <rg> \
--query "{publicAccess:properties.publicNetworkAccess, defaultAction:properties.networkAcls.defaultAction, privateEndpoints:properties.privateEndpointConnections}" \
-o json| Finding | Severity | Score |
publicNetworkAccess = Disabled + private endpoints exist | ✅ Pass | 13 pts |
defaultAction = Deny + IP/VNet rules configured | ✅ Pass | 13 pts |
defaultAction = Allow (open to internet) | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/cognitive-services-virtual-networks
Check 1.3 — Content filtering (RAI policies)
az rest --method get \
--url "https://management.azure.com/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<account>/raiPolicies?api-version=2024-10-01"Then check deployments have a policy assigned:
az cognitiveservices account deployment list \
--name <account> --resource-group <rg> \
--query "[].{name:name, model:properties.model.name, raiPolicy:properties.raiPolicyName}" \
-o table| Finding | Severity | Score |
| All deployments have RAI policy assigned with filters enabled | ✅ Pass | 10 pts |
| Some deployments missing policy or using minimal filters | ⚠️ Partial | 5 pts |
| No RAI policies or content filtering disabled | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/content-filters
⚙️ CATEGORY 2 — Reliability & Operations
Check 2.1 — Model version pinned and not deprecated
az cognitiveservices account deployment list \
--name <account> --resource-group <rg> \
--query "[].{name:name, model:properties.model.name, version:properties.model.version}" \
-o tableCross-reference with available models:
az cognitiveservices account list-models \
--name <account> --resource-group <rg> \
--query "[].{model:model.name, version:model.version, lifecycle:model.lifecycleStatus}" \
-o table| Finding | Severity | Score |
| All deployments on GA versions with explicit version pinned | ✅ Pass | 7 pts |
| Deployments on versions nearing retirement (< 3 months) | ⚠️ Attention | 3 pts |
| Deployments on Legacy/Deprecated versions | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/model-retirements
Check 2.2 — Diagnostic settings enabled
# Get resource ID
RESOURCE_ID=$(az cognitiveservices account show \
--name <account> --resource-group <rg> --query "id" -o tsv)
az monitor diagnostic-settings list --resource $RESOURCE_ID -o json| Finding | Severity | Score |
| Diagnostic settings sending RequestResponse + Audit to Log Analytics | ✅ Pass | 7 pts |
| Diagnostic settings exist but incomplete (only metrics, no logs) | ⚠️ Partial | 3 pts |
| No diagnostic settings at all | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/monitoring
Check 2.3 — Resource Lock on production account
az lock list --resource-group <rg> \
--query "[?contains(id, '<account>')].{name:name, level:level}" \
-o table| Finding | Severity | Score |
| CanNotDelete lock exists on the OpenAI account | ✅ Pass | 5 pts |
| Lock exists on RG but not specifically on account | ⚠️ Partial | 3 pts |
| No locks | ❌ Fail | 0 pts |
📖 https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/lock-resources
Check 2.4 — Multi-region resilience (failover capability)
Check if OpenAI accounts exist in more than one region (disaster recovery / latency optimization):
az cognitiveservices account list \
--subscription <sub-id> \
--query "[?kind=='OpenAI' || kind=='AIServices'].{name:name, location:location, rg:resourceGroup}" \
-o tableGroup by location and count unique regions:
| Finding | Severity | Score |
| OpenAI accounts in 2+ different regions | ✅ Pass | 5 pts |
| All OpenAI accounts in a single region | ⚠️ Risk | 2 pts |
| Only 1 OpenAI account total (single point of failure) | ❌ Risk | 0 pts |
Context from the field: Customers with single-region deployments face outages during regional capacity events. Multi-region + APIM load balancing (Check 4.1) is the recommended pattern for production AI workloads.
Check 2.5 — 429 throttling rate (last 24 hours)
Requires diagnostic settings sending logs to Log Analytics (Check 2.2). If diagnostics are not configured, skip and note dependency.
AzureDiagnostics
| where TimeGenerated > ago(24h)
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where Category == "RequestResponse"
| summarize
TotalRequests = count(),
ThrottledRequests = countif(resultSignature_d == 429),
ThrottleRate = round(100.0 * countif(resultSignature_d == 429) / count(), 2)
by Resource, properties_s
| order by ThrottleRate descUse execute_kusto_query with the Log Analytics workspace connected to the OpenAI account's diagnostic settings.
| Finding | Severity | Score |
| Throttle rate < 1% in last 24h | ✅ Pass | 6 pts |
| Throttle rate 1–10% (occasional bursts) | ⚠️ Attention | 3 pts |
| Throttle rate > 10% (active capacity problem) | ❌ Fail | 0 pts |
| No diagnostic data available (Check 2.2 failed) | ⚠️ Skip | 0 pts — flag dependency |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/quota
Context from the field: Customers assume PTU eliminates throttling — it does not. Bursty traffic and concurrency limits can cause 429s even on PTU. Check APIM retry/spillover pattern (Check 4.1) as mitigation.
💰 CATEGORY 3 — Cost & Efficiency
Check 3.1 — Rate limits configured per deployment (not max)
az cognitiveservices account deployment list \
--name <account> --resource-group <rg> \
--query "[].{name:name, model:properties.model.name, sku:sku.name, capacity:sku.capacity}" \
-o table| Finding | Severity | Score |
| Each deployment has explicit TPM capacity set (not maximum) | ✅ Pass | 5 pts |
| Single deployment consuming all available quota | ⚠️ Risk | 2 pts |
| Unable to determine (no deployments) | N/A | 5 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/quota
Check 3.2 — Model selection diversity (not GPT-4o for everything)
az cognitiveservices account deployment list \
--name <account> --resource-group <rg> \
--query "[].{name:name, model:properties.model.name}" \
-o table| Finding | Severity | Score |
| Mix of models (e.g., gpt-4o + gpt-4o-mini, or batch deployments) | ✅ Pass | 3 pts |
| Only premium models deployed (no cost-efficient alternative) | ⚠️ Info | 1 pt |
| Single deployment only | N/A | 3 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models
Check 3.3 — PTU utilization (if applicable)
Only check if PTU/Provisioned deployments exist:
az cognitiveservices account deployment list \
--name <account> --resource-group <rg> \
--query "[?sku.name=='ProvisionedManaged'].{name:name, capacity:sku.capacity}" \
-o tableIf PTU deployments exist, check utilization via metrics:
az rest --method get \
--url "https://management.azure.com<resource-id>/providers/microsoft.insights/metrics?api-version=2019-07-01&metricnames=ProvisionedManagedUtilizationV2×pan=P7D&interval=PT1H&aggregation=Average"| Finding | Severity | Score |
| PTU utilization 70–85% average (well-sized) | ✅ Pass | 7 pts |
| PTU utilization < 50% (over-provisioned — burning money) | ⚠️ Waste | 2 pts |
| PTU utilization > 95% (under-provisioned — users getting 429s) | ⚠️ Risk | 4 pts |
| No PTU deployments (using Standard — fine for most startups) | N/A | 7 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/provisioned-throughput
Check 3.4 — Quota utilization (TPM used vs limit)
az cognitiveservices account list-usage \
--name <account> --resource-group <rg> \
-o jsonAlternative if the above returns empty (common for newer deployments):
az rest --method get \
--url "https://management.azure.com/subscriptions/<sub-id>/providers/Microsoft.CognitiveServices/locations/<location>/usages?api-version=2024-10-01"| Finding | Severity | Score |
| Quota usage < 70% of limit across all models | ✅ Pass | 5 pts |
| Quota usage 70–90% (nearing limit — plan increase) | ⚠️ Attention | 3 pts |
| Quota usage > 90% (at risk of rejection — request increase NOW) | ❌ Risk | 0 pts |
| Unable to retrieve usage data | ⚠️ Info | 3 pts |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits
Context from the field: Approved TPM quotas sometimes don't appear in Foundry due to subscription/project mismatches or the new Global/Data Zone quota pooling model. If usage data looks incorrect, verify the quota was assigned to the correct subscription and region.
Check 3.5 — Token consumption trend (tokens per day per deployment)
Requires diagnostic settings sending logs to Log Analytics (Check 2.2). If diagnostics are not configured, skip and note dependency.
AzureDiagnostics
| where TimeGenerated > ago(7d)
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where Category == "RequestResponse"
| extend model = tostring(parse_json(properties_s).modelName)
| extend promptTokens = toint(parse_json(properties_s).promptTokens)
| extend completionTokens = toint(parse_json(properties_s).completionTokens)
| summarize
DailyPromptTokens = sum(promptTokens),
DailyCompletionTokens = sum(completionTokens),
DailyTotalTokens = sum(promptTokens) + sum(completionTokens),
RequestCount = count()
by bin(TimeGenerated, 1d), Resource, model
| order by TimeGenerated desc, DailyTotalTokens descUse execute_kusto_query with the Log Analytics workspace.
| Finding | Severity | Score |
| Token consumption stable or declining (no runaway growth) | ✅ Pass | 5 pts |
| Token consumption growing > 20% day-over-day (investigate) | ⚠️ Attention | 2 pts |
| Token consumption spiking > 50% day-over-day (possible prompt leak or loop) | ❌ Risk | 0 pts |
| No diagnostic data available (Check 2.2 failed) | ⚠️ Skip | 0 pts — flag dependency |
📖 https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/monitoring
Context from the field: Unmonitored token growth is the #1 surprise cost driver for startups using OpenAI. A single prompt injection loop or chatbot without max_tokens can burn through thousands of dollars in a weekend.
🏗️ CATEGORY 4 — Architecture Patterns
Check 4.1 — AI Gateway / APIM in front of OpenAI (for production)
az apim list --subscription <sub-id> \
--query "[].{name:name, sku:sku.name, rg:resourceGroup}" \
-o table| Finding | Severity | Score |
| APIM exists with OpenAI-related APIs configured | ✅ Pass | 5 pts |
| No APIM — calls go directly to OpenAI endpoint | ⚠️ Risk | 0 pts |
| N/A (dev/test environment only) | N/A | 5 pts |
Check 4.2 — Environment separation (dev ≠ prod for AI resources)
az cognitiveservices account list \
--subscription <sub-id> \
--query "[?kind=='OpenAI'].{name:name, rg:resourceGroup, tags:tags}" \
-o json| Finding | Severity | Score |
| Separate accounts or deployments for dev/prod with tags | ✅ Pass | 5 pts |
| Single account with environment tag but shared deployments | ⚠️ Partial | 2 pts |
| Single account, no separation, no tags | ❌ Fail | 0 pts |
Scoring
| Category | Checks | Max Points |
| 🔐 Security (Identity, Network, Content Filter) | 1.1–1.3 | 35 |
| ⚙️ Reliability (Model version, Diagnostics, Locks, Multi-region, 429 monitoring) | 2.1–2.5 | 30 |
| 💰 Cost & Efficiency (Rate limits, Model mix, PTU, Quota, Token trend) | 3.1–3.5 | 25 |
| 🏗️ Architecture (Gateway, Env separation) | 4.1–4.2 | 10 |
| TOTAL | 15 checks | 100 |
Maturity levels
| Score | Level | Meaning |
| 0–39 | 🔴 Critical | Major security and operational gaps — not production-ready |
| 40–69 | 🟡 Developing | Some controls in place but significant risks remain |
| 70–89 | 🟢 Solid | Good posture — minor improvements recommended |
| 90–100 | 🏆 Exemplary | Following WAF best practices for AI workloads |
Accepted exceptions (optional)
If the user provides a list of accepted exceptions, do not flag those items. Instead, note them in the report as Accepted Exception with the reason provided.
Example format the user may provide:
| Check | Reason |
| 1.2 Network isolation | Public endpoint required for third-party SaaS integration |
| 1.3 Content filtering | Custom RAI policy approved by security team for internal tool |
| 3.3 PTU utilization | PTU capacity pre-provisioned for upcoming product launch |
When exceptions are provided:
- Skip the flagged checks in scoring
- List them in a separate "Accepted Exceptions" section at the end of the report
- Recalculate the overall score excluding excepted checks
Expected output
Report header (mandatory — use this exact format)
AI Foundry & OpenAI Posture Check Report
| Field | Value |
| Score | XX / 100 |
| Level | 🟡 Developing |
| Accounts Assessed | N (list names) |
| Deployments Found | M (model details) |
| Assessment Date | YYYY-MM-DD |
Per-account findings
For each OpenAI/AI Services account:
| Check | Status | Finding | Recommendation |
| Managed Identity | ❌ | API keys enabled, no MI | Enable System MI + set disableLocalAuth=true |
| Network | ⚠️ | Public access enabled, no firewall | Set defaultAction=Deny + add VNet rules |
| ... | ... | ... | ... |
Critical findings (fix immediately)
Security issues that expose the AI endpoint or violate responsible AI principles.
Cost optimization opportunities
Token waste, PTU over-provisioning, or model selection improvements with estimated savings.
Architecture recommendations
Patterns to improve resilience: AI Gateway, retry policies, multi-region deployment.
Remediation guidance
For each ❌ or ⚠️ finding, include in the output:
- The specific
azCLI command to remediate (suggest only — do not execute) - Use
GetAzCliHelpto validate the command syntax before suggesting - Include the 📖 documentation link from the check that failed
References
- WAF for AI: https://learn.microsoft.com/en-us/azure/well-architected/ai/
- OpenAI Security: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/managed-identity
- Content Filtering: https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/content-filters
- Model Lifecycle: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/model-retirements
- PTU Planning: https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/provisioned-throughput
- PTU Calculator: https://ptucalc.com
Sample output
The following is a redacted example of what the report looks like when run against a subscription.
AI Foundry & OpenAI Posture Check Report
| Field | Value |
| Score | 61 / 100 |
| Level | 🟡 Developing |
| Accounts Assessed | 2 (oai-contoso-prod, oai-contoso-dev) |
| Deployments Found | 5 (gpt-4o, gpt-4o-mini, text-embedding-ada-002, gpt-4o x2) |
| Assessment Date | 2026-07-15 |
Per-account findings (sample — oai-contoso-prod)
| Check | Status | Finding | Recommendation |
| 1.1 Managed Identity | ⚠️ | System MI enabled but disableLocalAuth=false | Set disableLocalAuth=true to block API key access |
| 1.2 Network isolation | ❌ | publicNetworkAccess=Enabled, no firewall rules | Set defaultAction=Deny + add VNet rules |
| 1.3 Content filtering | ✅ | All deployments have RAI policy contoso-standard assigned | — |
| 2.1 Model version | ⚠️ | gpt-4o deployment on version 2024-05-13 retiring in 60 days | Upgrade to 2024-08-06 |
| 2.2 Diagnostic settings | ✅ | RequestResponse + Audit logs → law-contoso-prod | — |
| 2.3 Resource lock | ❌ | No CanNotDelete lock on oai-contoso-prod | Add lock |
| 2.4 Multi-region | ✅ | Accounts in East US + West Europe | — |
| 3.1 Rate limits | ✅ | Each deployment has explicit TPM capacity set | — |
| 3.2 Model diversity | ✅ | Mix of gpt-4o + gpt-4o-mini for cost efficiency | — |
| 4.1 AI Gateway (APIM) | ❌ | No APIM instance found | Deploy APIM with OpenAI backend |
Cost optimization opportunities (sample)
| Finding | Current Cost | Potential Savings | Action |
gpt-4o used for simple classification tasks | ~$420/mo | ~$340/mo (80%) | Route to gpt-4o-mini |
Dev account oai-contoso-dev idle on weekends | ~$85/mo | ~$25/mo (30%) | Reduce dev TPM allocation |
Remediation commands (sample)
# Disable local API key authentication
az cognitiveservices account update --name oai-contoso-prod --resource-group rg-ai-prod --set properties.disableLocalAuth=true
# Set network firewall to deny by default
az cognitiveservices account update --name oai-contoso-prod --resource-group rg-ai-prod --set properties.networkAcls.defaultAction=Deny
# Add resource lock
az lock create --name CanNotDelete --resource-group rg-ai-prod --resource oai-contoso-prod --resource-type Microsoft.CognitiveServices/accounts --lock-type CanNotDeleteDoes it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

