agentleFS
alex-voloshin-devalex-voloshin-dev/ai-skills/.agents/skills/analyze-prod/SKILL.md

analyze-prod

Skill2 starsChanged 6 months agoMITView on GitHub

Analyze production environment — collect Kubernetes pod status, managed database health, logs, metrics, networking, and diagnose issues. Supports GCP, Azure, and AWS via the `cloud-platforms` skill. Applies SRE or DevOps roles. Use standalone or as part of production incident investigation.

The file

We split this skill into the rules it gives an agent

7 rules. About 0 are common, 4 are distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Cloud platform detection: Read AGENTS.md to identify which cloud platform is used.

    line 18
  2. Distinctive:

    If invoked as part of a bugfix/incident flow — extract context from the parent conversation instead of asking.

    line 32
  3. Distinctive:

    Before collecting data, establish the environment context:

    line 53
  4. Distinctive:

    Run the following read-only diagnostic commands.

    line 78
  5. Very specific:

    Run platform-specific database diagnostic commands from cloud-platforms skill:

    line 127
  6. Very specific:

    Run platform-specific monitoring commands from cloud-platforms skill to list dashboards and check metrics.

    line 160
  7. Distinctive:

    After any fix, re-run relevant diagnostic commands from Step 4 to verify resolution.

    line 226

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Analyze Production Environment

Systematic analysis of the production environment. Collects cluster status, pod health, database metrics, logs, and diagnoses issues. Works standalone or as an entry point for production bugfixing.

Cloud platform detection: Read AGENTS.md to identify which cloud platform is used. Consult cloud-platforms skill and load the corresponding reference module (GCP, Azure, or AWS) for platform-specific CLI commands.

⚠️ SAFETY: This workflow runs READ-ONLY commands only. No mutations (scale, delete, restart, deploy) without explicit user approval at Step 6.

1. Clarify the Scope

Ask the user:

  • What is the problem or question? (e.g., "pods crashlooping", "high latency on /api/orders", "DB connections exhausted", "general health check")
  • Which service(s) are affected? (deployment name, namespace, or "all")
  • Cloud environment? (project/subscription/account + cluster name)
  • When did the issue start? (deploy, config change, traffic spike, or unknown)
  • Is there an active incident? (P1–P4 severity, or no incident)

If invoked as part of a bugfix/incident flow — extract context from the parent conversation instead of asking.

2. Apply Appropriate Role

Select and apply the role based on the problem type:

Problem TypePrimary RoleRationale
Pod crashes, restarts, health check failuressre-engineer roleReliability, K8s troubleshooting, SLO impact
High latency, error rate spikes, SLO burnsre-engineer roleObservability, SLI/SLO analysis
Managed DB issues (connections, replication, CPU)sre-engineer roleDatabase reliability, capacity
Networking, ingress, DNS, connectivitysre-engineer role + devops-engineer roleNetwork diagnostics + infra config
Deployment failures, rollback neededdevops-engineer roleCI/CD, Helm, K8s deployment
Terraform/infra drift, resource configdevops-engineer roleIaC, cloud resource management
Application errors visible in logsStack-specific rolejava-engineer role, python-engineer role, frontend-engineer role
General / unclearsre-engineer roleSRE debugging methodology as default for prod

Announce the applied role(s) to the user. If this is a P1/P2 incident, always apply sre-engineer role.

3. Establish Context

Before collecting data, establish the environment context:

3a. Cloud Platform Context

Detect platform from AGENTS.md and verify authentication. Consult cloud-platforms skill for platform-specific auth verification commands:

  • GCP: gcloud config get-value project + gcloud auth list
  • Azure: az account show + az aks list
  • AWS: aws sts get-caller-identity + aws eks list-clusters

Confirm the active project/subscription/account matches the user's target.

3b. Kubernetes Cluster

// turbo
kubectl config current-context
kubectl cluster-info

Also run platform-specific cluster list command from cloud-platforms skill to verify cluster status.

Record: Cluster name, location, version, node count, current kubectl context.

4. Collect Production Snapshot

Run the following read-only diagnostic commands. Present results as a structured summary.

4a. Node Health

// turbo
kubectl get nodes -o wide
kubectl top nodes

Flag: Nodes in NotReady state, high CPU/memory utilization (>80%), version skew between nodes.

4b. Pod Status

For the affected namespace (or all namespaces if unspecified):

// turbo
kubectl get pods -n <namespace> -o wide --sort-by='.status.startTime'
kubectl get pods -n <namespace> --field-selector=status.phase!=Running

Flag:

  • Pods in CrashLoopBackOff, Error, Pending, ImagePullBackOff
  • Pods with high restart counts (>3 in last hour)
  • Pods not matching expected replica count (check kubectl get deployments)

4c. Pod Details (for problematic pods)

kubectl describe pod <pod-name> -n <namespace>
kubectl logs --tail=200 --timestamps <pod-name> -n <namespace>
kubectl logs --previous --tail=50 <pod-name> -n <namespace>

Look for: OOMKilled (exit code 137), application exceptions, connection errors, startup failures, readiness probe failures.

4d. Resource Usage

// turbo
kubectl top pods -n <namespace> --sort-by=memory
kubectl get hpa -n <namespace>

Flag: Pods near memory limits, HPA at max replicas, CPU throttling.

4e. Managed Database Health

Run platform-specific database diagnostic commands from cloud-platforms skill:

  • GCP: gcloud sql instances list + gcloud sql instances describe
  • Azure: az postgres flexible-server show or az sql server list
  • AWS: aws rds describe-db-instances

Key metrics (from cloud monitoring or CLI):

  • CPU utilization, memory utilization, disk usage
  • Active connections vs max connections
  • Replication lag (if HA/read replicas)
  • Failed connections count

4f. Networking and Ingress

// turbo
kubectl get ingress -n <namespace>
kubectl get svc -n <namespace>
kubectl get endpoints -n <namespace>

Flag: Services with 0 endpoints (no healthy backends), Ingress with no address, port mismatches.

4g. Recent Events

// turbo
kubectl get events -n <namespace> --sort-by='.lastTimestamp' --field-selector type!=Normal

Record: Warning/Error events — especially FailedScheduling, OOMKilled, FailedMount, Unhealthy, BackOff.

4h. Cloud Monitoring (if available)

Run platform-specific monitoring commands from cloud-platforms skill to list dashboards and check metrics.

Guide the user to check relevant dashboards for:

  • SLI metrics: Error rate, latency (p50/p95/p99), availability
  • Error budget burn rate: Is the SLO at risk?
  • Alerting policies: Which alerts fired?

5. Analyze Findings

Using the applied role's expertise:

  1. Correlate the user's problem with collected data
  2. Identify root cause using the role's debugging methodology
  3. Assess SLO impact (if sre-engineer role active): error budget consumed, burn rate

<common_prod_issues>

  • Pod CrashLoopBackOff: OOM (check limits), app startup failure (check logs), missing config/secret, failed dependency (DB not reachable)
  • High latency: Database slow queries, connection pool exhaustion, CPU throttling (check requests vs limits), upstream dependency timeout
  • Managed DB connection exhaustion: Too many connections from pods, connection leak in app, pool size misconfigured, proxy/pooler not configured
  • Pods Pending: Insufficient cluster resources (node autoscaler max reached), node affinity/taint mismatch, PVC not bound
  • 5xx errors: Application exception (check logs), upstream timeout, readiness probe failing (traffic to unhealthy pod), resource exhaustion
  • Deployment stuck: Image pull error (wrong tag, registry auth), failed readiness probe, PDB blocking rollout, insufficient quota
  • Network unreachable: NetworkPolicy blocking traffic, service selector mismatch, DNS resolution failure, VPC/firewall rule </common_prod_issues>

6. Present Diagnosis

Structure the diagnosis:

## Production Environment Summary
- Cloud: [GCP/Azure/AWS] | Project/Sub/Account: [id] | Cluster: [name] ([location])
- Nodes: [count] ([healthy]/[total]) | K8s version: [version]
- Managed DB: [instance] ([state], [tier])

## SLO Impact (if applicable)
- SLO: [target] | Current: [actual] | Error budget: [remaining]%
- Burn rate: [Nx] | Time to budget exhaustion: [duration]

## Findings
### [Issue 1: title]
- **Symptom**: what was observed
- **Evidence**: specific log lines, metrics, pod status
- **Root cause**: why it's happening
- **Severity**: P1 (outage) / P2 (degraded) / P3 (minor) / P4 (cosmetic)
- **Blast radius**: affected users, services, regions

## Recommendations
1. [Immediate mitigation] — [command] ⚠️ REQUIRES APPROVAL
2. [Root cause fix] — [change description]
3. [Prevention] — [long-term improvement]

## Environment Health: [HEALTHY | DEGRADED | OUTAGE]

7. Fix or Escalate

⚠️ All production mutations require explicit user approval.

Based on the diagnosis:

  • Immediate mitigation (rollback, restart, scale): Present the exact command. Wait for user approval. Execute and verify
  • Application code fix: Transition to stack-specific role. Fix code, test locally (optionally analyze-local skill), then deploy through normal CI/CD
  • Infrastructure fix (Terraform, Helm, K8s config): Apply with devops-engineer role patterns via PR, never direct apply
  • Root cause unclear: Propose additional diagnostics (increase log verbosity, tracing, profiling, query analysis)

After any fix, re-run relevant diagnostic commands from Step 4 to verify resolution.

8. Summary

Present the completed analysis:

  • Problem: original report / incident ID
  • Severity: P1–P4
  • Role(s) applied: which roles were used
  • Root cause: what was found
  • SLO impact: error budget consumed
  • Fix applied: what was changed (or "investigation only — no fix applied")
  • Verification: confirmation the issue is resolved
  • Action items: postmortem (if P1/P2), prevention measures, monitoring improvements

Integration

  • Called by: bugfix skill (production environment diagnostics)
  • Roles: sre-engineer role, devops-engineer role
  • Skills: cloud-platforms skill (platform-specific CLI commands, managed service diagnostics)

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.