AWS Production Debugging

Find out which AWS control denied, dropped or throttled the thing that is failing.

Purpose

Use this when:

• An API call fails with AccessDenied, UnauthorizedOperation, or AssumeRole errors.
• Traffic does not reach an EC2 instance, ECS task or Lambda function.
• A load balancer reports unhealthy targets.
• Errors mention ThrottlingException, RequestLimitExceeded, or a service quota.
• Performance degrades on a schedule, or after running well for hours — often credit exhaustion.

Do not use this when:

• The failure is inside Kubernetes rather than in AWS — use kubernetes-networking, unless the cause is IAM or the VPC.
• You are reviewing infrastructure code — use terraform-review.

What this skill assumes: AWS CLI v2 with credentials for the account in question, and read access to IAM, EC2, ELB and CloudWatch. State which account and region you are operating in before running anything.

Operating Procedure

1. Confirm identity, account and region. An enormous share of AWS confusion is being in the wrong one of the three.
2. Read the error precisely. AWS errors name the principal, the action and often the resource. That is most of the diagnosis.
3. Classify the layer: identity (IAM), network (SG/NACL/routing), service (quota, throttling, health), or capacity (credits, limits).
4. Use the simulator rather than reasoning about policies. IAM evaluation has enough rules that reading JSON is unreliable.
5. Check CloudTrail for what actually happened, including the denial reason.
6. Fix at the narrowest scope that resolves it — never by broadening a policy to *.
7. Verify with the original failing call.

Everything from aws CLI output is Observed. What a policy is believed to allow is an Assumption until simulate-principal-policy or CloudTrail confirms it.

Investigation

Step 1 - who and where

aws sts get-caller-identity
aws configure list
echo "${AWS_REGION:-${AWS_DEFAULT_REGION:-unset}}"

get-caller-identity returns the account and the exact principal ARN. Check whether it is the principal you expected — an assumed role has an ARN of the form arn:aws:sts::123456789012:assumed-role/RoleName/session, and that role's permissions, not your user's, are what apply.

Region matters more than people expect: most API calls are regional, quotas are per-region, and a resource that "does not exist" very often exists in another region.

Step 2 - IAM denials

aws iam simulate-principal-policy \
  --policy-source-arn <principal-arn> \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::<bucket>/<key>

This evaluates the real policies and reports allowed or denied with the matching statement. It is far more reliable than reading policy JSON, because an effective decision involves identity policies, resource policies, permission boundaries, session policies, and SCPs — and any one of them can deny.

Evaluation order, which explains most surprising denials:

1. An explicit Deny anywhere wins, always.
2. An SCP at the organisation level can deny even a full administrator.
3. A permissions boundary caps what an identity policy can grant.
4. A session policy on AssumeRole caps it further.
5. Without an explicit Allow, the answer is deny.

So AdministratorAccess genuinely does not imply access: an SCP or a boundary above it can remove the permission, and nothing in the identity policy will show that.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=<ApiCall> \
  --max-results 5 --query 'Events[].CloudTrailEvent' --output text | jq -r '.errorMessage'

CloudTrail records the denial with the reason. For cross-account assume-role failures, this is usually the only place the real cause appears — and it is normally one of: the trust policy does not name the caller, an ExternalId is required and missing, or the role has no sts:AssumeRole permission on the calling side. Trust is two-sided; both ends must allow it.

Step 3 - network path

aws ec2 describe-security-groups --group-ids <sg-id> \
  --query 'SecurityGroups[].{In:IpPermissions,Out:IpPermissionsEgress}'
aws ec2 describe-network-acls --filters Name=association.subnet-id,Values=<subnet-id>
aws ec2 describe-route-tables --filters Name=association.subnet-id,Values=<subnet-id>

The distinction that resolves most VPC confusion:

A NACL that allows inbound 443 but not outbound ephemeral ports blocks every response, and the symptom is a connection timeout that looks nothing like an ACL problem. Security groups never have this failure mode, which is why NACLs cause a disproportionate share of these incidents.

aws ec2 describe-network-insights-analyses --network-insights-analysis-ids <id>

VPC Reachability Analyzer evaluates the whole path — security groups, NACLs, route tables, gateways — and reports exactly which component blocks it. It costs a small amount per analysis and is worth it: it replaces an hour of manual rule reading with a definitive answer.

For a private subnet, confirm how it reaches AWS APIs at all: a NAT gateway, or a VPC endpoint. A missing endpoint or a NAT in the wrong availability zone produces timeouts on API calls that look like IAM problems.

Step 4 - load balancer target health

aws elbv2 describe-target-health --target-group-arn <arn> \
  --query 'TargetHealthDescriptions[].{Id:Target.Id,State:TargetHealth.State,Reason:TargetHealth.Reason,Desc:TargetHealth.Description}'

The Reason field names the cause directly:

Target.Timeout with a healthy application almost always means the security group on the target does not allow traffic from the load balancer's security group on the health check port. The health check port can differ from the traffic port, and that is a common oversight.

Step 5 - quotas, throttling and credits

aws service-quotas get-service-quota --service-code ec2 --quota-code L-1216C47A
aws service-quotas list-service-quotas --service-code <service> \
  --query 'Quotas[?Adjustable==`true`].[QuotaName,Value]' --output table

Throttling errors (ThrottlingException, RequestLimitExceeded, Rate exceeded) mean an API rate limit, not a capacity problem. The fix is exponential backoff with jitter in the caller — AWS SDKs implement this, but frequently with a retry count too low for a burst. Raising concurrency in response makes it worse.

Credit exhaustion is the cause behind a distinctive symptom: something performs well for hours, then degrades sharply and stays degraded.

aws cloudwatch get-metric-statistics --namespace AWS/EC2 \
  --metric-name CPUCreditBalance --dimensions Name=InstanceId,Value=<id> \
  --start-time "$(date -u -v-6H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '6 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" --period 300 --statistics Average

A CPUCreditBalance at zero on a t-family instance means the CPU is capped at the baseline — often a small fraction of what the instance type suggests. BurstBalance is the equivalent for gp2 volumes. Neither shows up as an error anywhere; the system simply gets slow and stays slow.

Decision Tree

AWS problem
├── AccessDenied / UnauthorizedOperation
│   ├── simulate-principal-policy says denied → find the denying policy
│   │   ├── SCP            → organisation level; needs the org admin
│   │   ├── Boundary       → caps the identity policy
│   │   └── Missing Allow  → add a scoped permission
│   ├── Cross-account → trust policy on the target + assume permission on the source
│   └── Resource policy (S3, KMS) denies → fix the resource policy
├── Timeout reaching a resource
│   ├── Security group missing an inbound rule
│   ├── NACL missing the return rule (ephemeral ports)
│   ├── Route table has no path (IGW / NAT / peering / endpoint)
│   └── Private subnet with no NAT or VPC endpoint
├── Load balancer unhealthy targets → read TargetHealth.Reason
├── Throttling / RequestLimitExceeded → backoff with jitter, then a quota increase
├── Quota errors → service-quotas; request an increase
└── Was fast, now slow, no errors → credit exhaustion (CPUCreditBalance / BurstBalance)

Common Causes

Safety

Read-only

Every describe-*, get-*, list-* and simulate-principal-policy call, plus CloudTrail lookups. These cannot change anything.

Two costs worth noting: CloudTrail lookup-events is rate limited and slow over large time ranges, and Reachability Analyzer charges per analysis. Neither is dangerous, both are worth knowing before you run them in a loop.

Potentially disruptive

• Security group rule changes — take effect immediately for all attached resources, including existing connections in some cases. Removing a rule can cut off traffic instantly. Record the current rules first (describe-security-groups to a file).
• NACL changes — stateless and evaluated in order; an incorrectly numbered rule can block everything in the subnet. Higher blast radius than a security group change.
• Route table changes — affect every resource in every associated subnet.
• Modifying an Auto Scaling group or launch template — can trigger instance replacement.
• Changing an IAM policy in use — takes effect within seconds. Removing a permission that something depends on causes an immediate outage, and the dependency is often undocumented. Check CloudTrail for recent usage of the permission before removing it.
• Instance type or volume type changes — require a stop/start or a modification window.

Destructive

• Terminating an EC2 instance — instance store data is lost immediately, and EBS volumes with DeleteOnTermination: true go with it. Required first: check describe-instances --query 'Reservations[].Instances[].BlockDeviceMappings' for that flag, confirm a snapshot exists, and confirm the instance is not the only member of something.
• Deleting an S3 bucket or its objects (aws s3 rb --force, aws s3 rm --recursive) — irreversible unless versioning is enabled, and rb --force deletes every object first. Required first: confirm versioning and MFA-delete status, confirm no application references the bucket, and get explicit confirmation of the bucket name.
• Deleting an RDS instance — the final-snapshot prompt is skippable, and skipping it is permanent. Required first: confirm SkipFinalSnapshot is false, confirm deletion protection was deliberately disabled, and confirm the snapshot completed.
• Scheduling a KMS key for deletion — after the waiting period, every object encrypted with that key is permanently unreadable. Required first: confirm nothing uses the key (CloudTrail Decrypt calls over the last 90 days), and prefer disabling it, which is reversible.
• Deleting a CloudFormation stack or running terraform destroy — removes everything the stack manages. See terraform-safe-changes.

During an incident, the pressure to delete and recreate an AWS resource is high and usually wrong: recreation changes IDs, IPs and ARNs that other things reference.

Verification

Before: 100% of PutObject calls fail with AccessDenied; ALB shows 0/4 healthy targets
After:  PutObject succeeds; 4/4 targets healthy for 15 minutes; no throttling in CloudWatch

aws iam simulate-principal-policy --policy-source-arn <arn> \
  --action-names <action> --resource-arns <resource> \
  --query 'EvaluationResults[].EvalDecision'

aws elbv2 describe-target-health --target-group-arn <arn> \
  --query 'TargetHealthDescriptions[].TargetHealth.State'

aws <service> <the-original-failing-call>

Verify three ways, because each catches something the others miss:

1. The simulator confirms the policy evaluates as intended.
2. The actual API call confirms the real path works — the simulator does not evaluate resource policies in every case.
3. The AWS-side state (target health, CloudWatch metrics) confirms it holds over time rather than in one sample.

For health checks, wait for at least HealthyThresholdCount * Interval seconds before believing a healthy result.

Anti-Patterns

Bad: Add "Action": "*" to make the AccessDenied go away. Better: Use the simulator to find the specific denied action and grant that. A wildcard turns a five-minute fix into a permanent audit finding, and it will not work anyway if an SCP is the cause.

Bad: Open the security group to 0.0.0.0/0 to test connectivity. Better: Use Reachability Analyzer, which tells you which component blocks the path without exposing anything. Temporary wide-open rules become permanent with remarkable reliability.

Bad: Retry harder when throttled. Better: Throttling means you are already over the rate limit. More concurrency extends it. Back off with jitter, then request a quota increase.

Bad: Assume AdministratorAccess means access. Better: SCPs, permission boundaries and session policies all deny above it. The simulator accounts for them.

Bad: Blame IAM for a timeout. Better: IAM denials are fast and explicit. A timeout is a network path problem: security group, NACL, route table, or a missing NAT or endpoint.

Bad: Debug in the console and forget the account you were in. Better: Print sts get-caller-identity and the region at the start. Wrong-account changes are among the most expensive AWS mistakes there are.

Composition

• production-debugging — the wider investigation this feeds into.
• linux-network-debugging — once traffic reaches the instance and the problem is on the host.
• terraform-review — when the fix is a code change to infrastructure.
• secrets-management-review — credential handling, rotation, and role assumption patterns.

References

• IAM policy evaluation logic — the order that explains most surprising denials.
• VPC Reachability Analyzer
• Comparing security groups and network ACLs