Terraform Safe Changes

Apply infrastructure changes so that the worst outcome is a failed apply, not a deleted database.

Purpose

Use this when:

• Applying a Terraform change to production.
• Refactoring modules, renaming resources, or moving resources between states.
• Importing existing infrastructure into Terraform.
• A plan wants to replace or destroy something that must survive.
• A previous apply failed partway and state no longer matches reality.

Do not use this when:

• You are only assessing a plan — use terraform-review first. Review precedes execution, always.
• Terraform itself is erroring — use terraform-troubleshooting.

What this skill assumes: a remote backend with locking and versioning, credentials for the target environment, and the ability to take a backup of whatever is at risk. If state is local and unversioned, fix that before making any production change.

Operating Procedure

Terraform has no undo. The procedure exists because of that.

1. Confirm the target. Workspace, backend, account, region. Print them; do not assume.
2. Back up state. A copy of state before the change is your only recovery path for state corruption.
3. Plan to a file and review it (terraform-review). Never apply an unreviewed plan.
4. Classify the change: additive, in-place, replacement, or destroy. The rest of the procedure depends on which.
5. Guard what must not die. prevent_destroy on stateful resources, before the apply.
6. Stage the change. Non-production first. Then a canary slice of production if the resource type allows it.
7. Apply the reviewed plan file, not a fresh plan.
8. Verify in the provider, not only in Terraform.
9. Record what changed and how to reverse it.

At every step, state what is Observed from command output. Never guess an account ID, a region or a workspace — print it.

Investigation

Step 1 - confirm where you are pointing

terraform workspace show
terraform version
aws sts get-caller-identity 2>/dev/null || gcloud config get-value project 2>/dev/null
grep -rn 'backend "' -A8 --include='*.tf' .

Wrong-environment applies are the most common serious Terraform incident, and they are entirely preventable by printing these four things. A workspace named default in a repository that has a prod workspace is a red flag on its own.

Step 2 - back up state

terraform state pull > "state-backup-$(date -u +%Y%m%dT%H%M%SZ).json"

Keep it outside the repository and outside the state bucket. If the backend is S3, confirm versioning is on — it is the difference between a five-minute recovery and rebuilding by hand.

aws s3api get-bucket-versioning --bucket <state-bucket>

If this returns empty, versioning is off. Enable it before making changes, not after losing state.

Step 3 - guard stateful resources

resource "aws_db_instance" "main" {
  # ...
  lifecycle {
    prevent_destroy = true
  }
}

prevent_destroy turns an accidental destroy into a plan-time error. Apply it to databases, state buckets, data volumes, KMS keys and anything else whose loss is unrecoverable. Add it in its own small change, ahead of the change you actually want to make.

It is a guard, not a policy: it blocks terraform destroy and replacements of that resource, but it does not protect against terraform state rm followed by a destroy, and it must be removed before a genuinely intended deletion.

Step 4 - rename and refactor without replacing

moved {
  from = aws_instance.web
  to   = aws_instance.frontend
}

A moved block (Terraform 1.1+) tells Terraform that a resource was renamed, so the plan shows no change instead of destroy-and-create. This is the correct tool for renames, module extraction, and count to for_each migrations. It is declarative, reviewable, and lives in Git — unlike terraform state mv, which is an untracked operator action.

moved {
  from = aws_instance.web[0]
  to   = aws_instance.web["primary"]
}

The same applies when migrating an indexed resource to keyed for_each, which otherwise replaces everything.

Step 5 - import existing infrastructure

import {
  to = aws_s3_bucket.logs
  id = "my-existing-logs-bucket"
}

Import blocks (Terraform 1.5+) are planned and reviewed like any other change, so the import shows up in the plan before it happens. Generate the matching configuration with:

terraform plan -generate-config-out=generated.tf

Review the generated configuration before keeping it: it contains every attribute including defaults and read-only fields, and will need trimming. After importing, a plan must show no changes — a plan that wants to modify a freshly-imported resource means your configuration does not match reality, and applying it will change live infrastructure.

Step 6 - stage the apply

terraform apply tfplan

For high-risk changes, reduce the blast radius before the full apply:

• Apply to a non-production workspace first and verify the result there.
• For a fleet, change one instance or one region first and observe before continuing.
• Split a large plan into sequential PRs rather than using -target.

-target exists for recovery, not for staging. Using it routinely leaves state and configuration out of sync, and the message Terraform prints after a targeted apply says exactly that.

Decision Tree

What kind of change is this?
├── Add new resources only
│   └── Low risk. Apply normally. Verify nothing existing changed.
├── In-place update (~)
│   ├── Attribute is not in the request path → apply normally
│   └── Attribute affects live traffic (LB rules, SG, DNS) → change window + rollback plan
├── Replacement (-/+)
│   ├── Resource holds data → STOP. Verified backup + agreed downtime, or avoid via moved/import
│   ├── Resource is in the request path → require create_before_destroy
│   └── Stateless → confirm the forcing attribute is intended, then apply
├── Destroy
│   ├── Stateful → STOP. See the Destructive section.
│   └── Stateless and unreferenced → confirm no dependents, then apply
└── Refactor (rename, module move, count→for_each)
    └── Use moved blocks. A refactor that shows any destroy is not a refactor.

Common Causes

Safety

Read-only

terraform plan, show, state list, state show, state pull, workspace show, validate, and providers. Note that plan takes the state lock and refreshes state by default.

Potentially disruptive

• terraform apply on in-place changes — changes live infrastructure. Rollback is to revert the code and apply again, which takes as long as the original change. There is no instant undo.
• terraform import — writes to state. Recoverable with terraform state rm, but an import that binds the wrong real resource to an address means the next apply modifies the wrong thing.
• terraform apply -target — leaves state partially converged; the next full plan may show surprising changes.
• terraform force-unlock <id> — only when you have confirmed no other apply is running. Breaking a lock while a colleague applies produces concurrent writes and a corrupted state file.

Destructive

Every operation below can cause irreversible loss. Each requires: a verified backup, an explicit human confirmation naming the resources, and a stated blast radius. Never propose one as a first suggestion or as a way to get past an error.

• terraform destroy — destroys every resource in the state, not just the one you are thinking about. Before running it, print exactly what it will remove and have a person confirm the list:

  terraform plan -destroy -out=destroy.tfplan
  terraform show -json destroy.tfplan | jq -r '.resource_changes[].address'

  Required first: a verified backup or snapshot of every stateful resource in that list, a named approver, and confirmation that this is a non-production environment or an agreed teardown.

• Applying a plan containing -/+ on a database, disk, bucket or volume — this deletes the data. Required first: a restore that has actually been tested, not a backup that exists.
• terraform state rm — removes a resource from state while leaving it running. The resource becomes unmanaged and invisible; a later apply may create a duplicate. Required first: terraform state pull > backup.json, and a written plan for re-importing it.
• Deleting or overwriting the state file or its bucket — this loses the mapping between configuration and real infrastructure. Recovery means importing every resource by hand.

If a plan shows an unexpected destroy, the correct response is to stop and find out why, never to add -auto-approve.

Verification

Before: 6 instances behind the LB, 0 errors, plan: 1 to change
After:  6 instances behind the LB, 0 errors, re-plan exit code 0 (no changes)

terraform plan -detailed-exitcode   # 0 = converged, 2 = still drifting, 1 = error
terraform state list | wc -l
aws elbv2 describe-target-health --target-group-arn <arn> \
  --query 'TargetHealthDescriptions[].TargetHealth.State' --output text

Verify in three places, because each can lie independently:

1. Terraform — a clean re-plan proves convergence.
2. The provider API — proves the resource exists and is configured as expected.
3. The application — proves the change did not break what uses the resource.

A successful terraform apply means Terraform's API calls returned success. It does not mean the load balancer has healthy targets or that DNS has propagated.

Anti-Patterns

Bad: terraform apply -auto-approve in an interactive production session. Better: Apply a reviewed plan file. -auto-approve belongs in CI, where the plan was reviewed in the PR, and nowhere else.

Bad: Fix drift by running terraform apply and letting it converge. Better: Find out what drifted and why with -refresh-only. Drift is often a deliberate emergency change, and blindly converging reverts it.

Bad: Use terraform state rm to get past an error. Better: State surgery abandons a live resource. Understand the error first; most "stuck" states are a lock, a stale plan, or a provider bug with a documented fix.

Bad: Rename a resource and accept the replacement because "it will just recreate". Better: Use a moved block. "It will just recreate" is true right up until the resource is a database or has a stable IP that something else depends on.

Bad: One giant apply covering networking, compute and data in a single PR. Better: Split by blast radius. Small applies fail small, and a failed apply in the middle of a 300-resource plan leaves you reconciling state under time pressure.

Bad: Keep prevent_destroy off because it gets in the way. Better: It is supposed to get in the way. That is the entire feature.

Composition

• terraform-review — always run first; this skill assumes the plan has been reviewed.
• terraform-troubleshooting — state locks, provider errors, failed applies.
• incident-triage — if an apply has already caused an outage, stabilise before continuing.

References

• Terraform: moved blocks and refactoring
• Terraform: import blocks
• Terraform: The lifecycle meta-argument
• Terraform: State — locking, backends and recovery.