Terraform Troubleshooting

Diagnose Terraform errors by working out which of the four layers is wrong: configuration, state, provider, or the real world.

Purpose

Use this when:

• terraform plan or apply fails with an error.
• An apply hangs, or fails with Error acquiring the state lock.
• The plan shows changes that make no sense, or never converges.
• An apply failed partway and you do not know what state is in.
• Error: Invalid for_each argument / Provider produced inconsistent final plan / Cycle:.

Do not use this when:

• The plan is correct and you are deciding whether to apply it — use terraform-review.
• You are executing a normal change — use terraform-safe-changes.

What this skill assumes: the exact error text. Terraform errors are specific and the message usually names the resource and the attribute; paraphrased errors are not diagnosable.

Operating Procedure

1. Capture the exact error. Full text, resource address, and the command that produced it.
2. Identify the layer. Configuration (HCL is wrong), state (Terraform's belief is wrong), provider (the plugin or its credentials are wrong), or reality (the cloud disagrees).
3. Reproduce read-only. terraform validate, then plan, before touching anything.
4. Check the obvious environmental causes first: wrong workspace, expired credentials, stale provider cache, a concurrent run.
5. Fix at the right layer. Configuration problems are fixed in code. State problems are fixed with state operations — and only after a state backup.
6. Re-plan to confirm convergence.
7. Record the cause; most Terraform errors recur across a team.

TF_LOG=DEBUG output is Observed evidence; your reading of an error message is an Assumption until the log or the provider API confirms it.

Investigation

Step 1 - establish the basics

terraform version
terraform workspace show
terraform validate
terraform providers

validate checks syntax and internal consistency without contacting any provider. If it fails, the problem is entirely in the configuration and nothing else matters yet.

terraform providers shows which provider versions are actually selected — frequently different from what the code appears to request, because the lockfile wins.

Step 2 - the error catalogue

Step 3 - state locks specifically

terraform plan -lock-timeout=10m

Wait first. Most locks are a colleague's running apply, and force-unlocking one produces two concurrent writers and a corrupt state file — a far worse problem than waiting.

If nothing is running, confirm it before unlocking. The error message prints the lock ID, who created it, and when:

Lock Info:
  ID:        7a4e9c1e-...
  Operation: OperationTypeApply
  Who:       ci-runner@build-4821
  Created:   2026-09-11 09:14:02 UTC

Check whether that CI job is still running. Only when you have positively confirmed it is not:

terraform force-unlock 7a4e9c1e-...

Always pass the specific ID. There is no safe "unlock whatever is stuck".

Step 4 - when the plan will not converge

A plan that shows the same change every time, even immediately after a successful apply, means something rewrites the attribute after Terraform sets it.

terraform apply
terraform plan -detailed-exitcode   # exit 2 means it still wants to change something
terraform state show <address> | grep <attribute>

Causes, in order of frequency:

1. Another system owns the attribute — an autoscaler changes desired_count, a controller adds tags, a service mesh mutates annotations. Fix: lifecycle { ignore_changes = [...] } on that specific attribute.
2. Provider normalisation — the API returns a canonical form (JSON key order, ARN case, a trailing newline in a policy) that differs textually from your input. Fix: use the provider's normalising function, for example jsonencode on IAM policy documents.
3. A genuine provider bug — check the provider's issue tracker for your exact version.

ignore_changes is the right answer only for case 1. Using it to silence case 2 or 3 hides real drift.

Step 5 - recovering a failed apply

A failed apply leaves state reflecting whatever succeeded before the failure. Terraform is designed for this: it is not corruption, and the correct first move is to look, not to repair.

terraform state pull > "state-backup-$(date -u +%Y%m%dT%H%M%SZ).json"
terraform plan

The new plan shows what remains to be done. In most cases, fixing the underlying cause and re-running apply is the complete answer. Reach for state surgery only when the plan itself proves state and reality disagree — for example, Terraform wants to create something that already exists.

Step 6 - provider-level debugging

TF_LOG=DEBUG TF_LOG_PATH=./tf-debug.log terraform plan
grep -E 'HTTP Response|StatusCode|error' tf-debug.log | head -40

This shows the actual API calls and responses. Use it when the error message is generic (context deadline exceeded, unexpected status 400) and you need to know what the provider asked for and what the cloud said back.

The debug log contains request bodies and may contain credentials and secret values. Treat tf-debug.log as sensitive: do not attach it to a public issue without redacting it.

Decision Tree

terraform command failed
├── terraform validate fails → configuration only; nothing else is relevant
├── Credential or auth error → provider layer; re-auth, check profile/region/role
├── "state lock" → wait; confirm no active run; force-unlock the named ID only
├── "already exists" → reality has it, state does not → import
├── "inconsistent final plan" / panic → provider bug → check version, upgrade or pin
├── Cycle / for_each / count errors → configuration; dependencies or unknown values
├── Apply failed partway
│   ├── Re-plan shows the remaining work → fix the cause, apply again
│   └── Re-plan wants to create existing resources → import them
└── Plan never converges
    ├── Another system mutates the attribute → ignore_changes on that attribute
    ├── Formatting/normalisation difference → use the canonical encoding
    └── Neither → provider bug

Common Causes

Safety

Read-only

terraform version, validate, providers, plan (takes the lock), show, state list, state show, state pull, and TF_LOG debugging. None of these change infrastructure.

Potentially disruptive

• terraform init -reconfigure / -migrate-state — changes where state lives. -migrate-state copies state to a new backend; if it is interrupted you can end up with two partial states. Back up first.
• terraform apply to recover a failed apply — real changes. Re-review the plan; the situation has changed since the original review.
• lifecycle { ignore_changes } — silences a real diff permanently. If the attribute is not genuinely owned by another system, you have hidden drift rather than fixed it.
• terraform force-unlock <id> — see below; it is only safe with positive confirmation.

Destructive

• terraform force-unlock <id> run while another apply is genuinely in progress produces two concurrent writers and can corrupt the state file, which is the hardest Terraform failure to recover from. Required first: positively confirm the holding process is dead (check the CI job, ask the named user), and use the exact ID from the error.
• terraform state rm <address> — removes the resource from state and leaves it running, unmanaged. Required first: terraform state pull > backup.json, and a concrete plan to re-import it. Never use this to make an error message go away.
• terraform state push — overwrites remote state wholesale. Required first: a backup of the current remote state and certainty that no one else has applied since your copy was taken. A bad push silently orphans every resource created since.
• terraform apply -replace=<address> on a stateful resource — destroys and recreates it, losing the data. Required first: a verified restore.

None of these should ever be the first response to an error. Every one of them is recovery, and every one needs the state backup taken before it runs.

Verification

Before: apply fails with "Error acquiring the state lock", 3 resources pending
After:  apply completes; terraform plan -detailed-exitcode returns 0

terraform plan -detailed-exitcode
terraform state list | wc -l

Exit code 0 from -detailed-exitcode is the strongest single signal that the problem is resolved: configuration, state and reality agree. Exit 2 means something still differs — the fix is incomplete, even if the error is gone.

For state operations, additionally confirm in the provider API that the resource count matches what state claims, and that nothing was orphaned.

Anti-Patterns

Bad: force-unlock because the lock has been there a few minutes. Better: Find the holder. The error prints who and when. A CI apply that takes 20 minutes is normal; breaking its lock corrupts state.

Bad: terraform state rm to clear an error. Better: Understand the error. Removing from state abandons live infrastructure that will still be billed, will not be managed, and will collide with the next apply.

Bad: Add ignore_changes to make a perpetual diff go away. Better: Find out what rewrites the attribute. If nothing does, ignore_changes is hiding real drift and you will not see the next unauthorised change either.

Bad: Delete .terraform and the lockfile when anything goes wrong. Better: .terraform is a cache and deleting it is usually harmless; deleting .terraform.lock.hcl silently upgrades every provider, which can turn a small error into a plan full of replacements.

Bad: Re-run apply repeatedly hoping it converges. Better: Each failed apply can leave more partial state. Read the error, fix the cause once.

Composition

• terraform-review — once the error is fixed, review the resulting plan before applying.
• terraform-safe-changes — for the apply itself, imports and moved blocks.
• aws-production-debugging — when the provider error reflects a real cloud-side problem such as quotas, IAM or service limits.

References

• Terraform: State — locking and recovery
• Terraform: Debugging — TF_LOG levels and what each contains.
• Terraform: terraform state commands