Infrastructure as Code

Overview

Tool-agnostic IaC best practices covering the decision framework, testing pyramid, CI/CD integration, state management, and policy enforcement. Includes Terraform-focused examples as the most common tool.

Tool Selection Decision Matrix

Recommendation: Default to Terraform for multi-cloud or large teams. Use CDK/Pulumi if team strongly prefers general-purpose languages. Use CloudFormation if AWS-only and team already knows it.

Project Structure

Terraform Layout

infrastructure/
├── modules/                    # Reusable modules
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   ├── outputs.tf
│   │   └── README.md
│   ├── eks/
│   ├── rds/
│   └── lambda/
├── environments/               # Per-environment configs
│   ├── dev/
│   │   ├── main.tf            # Module composition
│   │   ├── variables.tf
│   │   ├── terraform.tfvars
│   │   ├── backend.tf         # S3 + DynamoDB state config
│   │   └── providers.tf
│   ├── staging/
│   └── prod/
├── policies/                   # OPA/Sentinel policies
│   └── deny-public-s3.rego
└── tests/                      # Integration tests
    └── vpc_test.go

Key rules:

• Separate state per environment (separate backend.tf)
• Shared modules in modules/, consumed by environments/
• Pin module versions: source = "../modules/vpc" or version = "~> 2.0"
• Pin provider versions in providers.tf
• Never commit .tfstate or .tfvars with secrets

Module Design

# modules/vpc/variables.tf
variable "name" {
  description = "VPC name"
  type        = string

  validation {
    condition     = can(regex("^[a-z][a-z0-9-]*$", var.name))
    error_message = "Name must be lowercase alphanumeric with hyphens."
  }
}

variable "cidr_block" {
  description = "VPC CIDR block"
  type        = string
  default     = "10.0.0.0/16"

  validation {
    condition     = can(cidrhost(var.cidr_block, 0))
    error_message = "Must be a valid CIDR block."
  }
}

variable "availability_zones" {
  description = "List of AZs to use"
  type        = list(string)
}

variable "tags" {
  description = "Tags to apply to all resources"
  type        = map(string)
  default     = {}
}

Module rules:

• One concern per module (VPC, EKS, RDS — not "everything")
• All inputs as variables with descriptions, types, and validations
• All outputs documented
• Use for_each over count for stable resource identity
• Use locals for computed values
• Generate docs with terraform-docs
• Semantic versioning when publishing

IaC Testing Pyramid

Run in order, fail fast:

Layer 1: Format & Validate (seconds)

# Pre-commit hook or CI step 1
terraform fmt -check -recursive
terraform validate

Layer 2: Lint (seconds)

# Enforce best practices
tflint --recursive
# Check for deprecated syntax, naming conventions, unused variables

Layer 3: Security Scan (seconds)

# Detect misconfigurations
checkov -d .
# or
tfsec .
# Catches: public S3 buckets, open security groups, unencrypted resources, missing tags

Layer 4: Plan & Cost (minutes)

# Preview changes — mandatory before apply
terraform plan -out=tfplan

# Estimate cost impact (Infracost)
infracost breakdown --path=tfplan --format=json
# Post cost diff as PR comment
infracost diff --path=tfplan --compare-to=infracost-base.json

Layer 5: Policy-as-Code (seconds)

# OPA (Open Policy Agent) — custom guardrails
terraform show -json tfplan > tfplan.json

# Preflight: parse-check the policies FIRST. `opa check` exits 1 on a syntax
# error, so a broken policy is caught here instead of silently never running.
opa check policies/

# -i/--input takes a FILE PATH, not stdin (use --stdin-input to pipe instead).
# --fail-defined exits 1 when the query is defined, i.e. when deny has entries;
# index with [_] so the query stays undefined (exit 0) when there are none —
# a bare 'data.terraform.deny' is an empty set, which is always defined (exit 1).
opa eval -d policies/ -i tfplan.json --fail-defined --format pretty 'data.terraform.deny[_]'; rc=$?

# Exit contract — FAIL CLOSED. Measured on OPA 1.19.0:
#   0 = no violations   1 = violations   2 = error (parse failure, missing input/policy dir)
# Never key the gate on `-eq 1` alone: exit 2 means the policy never evaluated,
# and reading that as "clean" lets an unparseable policy wave every change through.
case $rc in
  0) echo "policy: clean" ;;
  1) echo "policy: violations"; exit 1 ;;
  *) echo "policy: ERROR — did not evaluate"; exit 1 ;;
esac

# Example policy: deny public S3 buckets
# policies/deny-public-s3.rego
# Rego v1: OPA 1.0+ requires `contains` for partial set rules and `if` before the
# body. `import rego.v1` is a no-op on 1.x and keeps the file valid on 0.59+.
package terraform
import rego.v1

deny contains msg if {
  resource := input.resource_changes[_]
  resource.type == "aws_s3_bucket"
  resource.change.after.acl == "public-read"
  msg := sprintf("S3 bucket %s must not be public", [resource.address])
}

Layer 6: Integration Tests (minutes)

// tests/vpc_test.go (Terratest)
func TestVPCCreation(t *testing.T) {
    terraformOptions := &terraform.Options{
        TerraformDir: "../modules/vpc",
        Vars: map[string]interface{}{
            "name":               "test-vpc",
            "cidr_block":         "10.0.0.0/16",
            "availability_zones": []string{"us-east-1a", "us-east-1b"},
        },
    }
    defer terraform.Destroy(t, terraformOptions)
    terraform.InitAndApply(t, terraformOptions)

    vpcID := terraform.Output(t, terraformOptions, "vpc_id")
    assert.NotEmpty(t, vpcID)
}

Or use native Terraform tests (1.6+):

# tests/vpc.tftest.hcl
run "create_vpc" {
  command = apply

  variables {
    name               = "test-vpc"
    cidr_block         = "10.0.0.0/16"
    availability_zones = ["us-east-1a", "us-east-1b"]
  }

  assert {
    condition     = output.vpc_id != ""
    error_message = "VPC ID must not be empty"
  }
}

CI/CD for Infrastructure

GitHub Actions Workflow

name: Infrastructure
on:
  pull_request:
    paths: ['infrastructure/**']
  push:
    branches: [main]
    paths: ['infrastructure/**']

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Format Check
        run: terraform fmt -check -recursive
        working-directory: infrastructure

      - name: Validate
        run: |
          for env in infrastructure/environments/*/; do
            terraform -chdir="$env" init -backend=false
            terraform -chdir="$env" validate
          done

      - name: Lint
        run: tflint --recursive
        working-directory: infrastructure

      - name: Security Scan
        run: checkov -d infrastructure/ --quiet

  plan:
    needs: validate
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    environment: plan
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Plan
        run: |
          terraform -chdir=infrastructure/environments/prod init
          terraform -chdir=infrastructure/environments/prod plan -out=tfplan -no-color
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

      - name: Cost Estimate
        uses: infracost/actions/setup@v3
      - run: infracost diff --path=infrastructure/environments/prod/tfplan --format=json --out-file=/tmp/infracost.json
      - uses: infracost/actions/comment@v3
        with:
          path: /tmp/infracost.json
          behavior: update

  apply:
    # Depends on validate, NOT plan: plan is pull_request-only, and a skipped
    # dependency skips its dependents — needs:plan would make apply unreachable.
    needs: validate
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    environment: production  # Requires manual approval
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      # Apply a PLAN, never a bare `apply -auto-approve`: with no plan file Terraform
      # re-resolves state at apply time, so what executes is not what anyone reviewed —
      # the PR plan above is advisory only. Planning and applying in the SAME job at least
      # makes the applied plan the one printed in this log. Stronger: upload the plan as an
      # artifact keyed by commit SHA and download it here instead of re-planning.
      - name: Plan and apply
        run: |
          terraform -chdir=infrastructure/environments/prod init
          terraform -chdir=infrastructure/environments/prod plan -out=tfplan
          terraform -chdir=infrastructure/environments/prod apply tfplan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

GitOps Flow

Developer → PR with infra changes
  → CI: fmt + validate + lint + security scan (automated)
  → CI: terraform plan + cost estimate (automated, posted as PR comment)
  → Review: team reviews plan output and cost impact
  → Approve: merge to main
  → CD: terraform apply (automated with manual approval gate for prod)

Rules:

• Never terraform apply from local machines in production
• Plan output must be reviewed by a human before apply
• Production apply requires explicit approval (GitHub environment protection)
• All state operations logged for audit

State Management

Remote State (Terraform + AWS)

# backend.tf
terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "prod/vpc/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

State rules:

• Always use remote backend (never local state in production)
• Enable encryption at rest
• Enable state locking (DynamoDB for AWS, GCS for GCP)
• Enable versioning on state bucket (disaster recovery)
• Separate state per component AND per environment
• State key pattern: <env>/<component>/terraform.tfstate
• Never store state in Git
• Use terraform state mv for refactoring, with backup

Drift Detection

# Schedule in CI (weekly or daily)
terraform plan -detailed-exitcode
rc=$?
case $rc in
  0) echo "no drift" ;;
  2) echo "DRIFT: changes detected — investigate manual changes"; exit 1 ;;
  *) echo "BROKEN: drift check did not run (exit $rc)"; exit 1 ;;
esac

Alert on exit 1 as loudly as on exit 2. Alerting only on 2 is a fail-open: this job is scheduled and unattended, so expired credentials, an unreachable backend or a syntax error exit 1 and produce silence — which is indistinguishable from "no drift" to everyone watching. The failure mode is a drift detector that has quietly not run for weeks. Never let "did not run" share an outcome with "nothing to report".

Common AWS Patterns

VPC with Public/Private Subnets

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = "${var.project}-${var.environment}"
  cidr = var.vpc_cidr

  azs             = var.availability_zones
  private_subnets = var.private_subnet_cidrs
  public_subnets  = var.public_subnet_cidrs

  enable_nat_gateway   = true
  single_nat_gateway   = var.environment != "prod"  # Cost: single for dev, per-AZ for prod
  enable_dns_hostnames = true

  tags = merge(var.tags, {
    "kubernetes.io/cluster/${var.cluster_name}" = "shared"
  })

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb"           = 1
    "karpenter.sh/discovery"                    = var.cluster_name
  }

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }
}

Tagging Strategy

# providers.tf — apply default tags to ALL resources
provider "aws" {
  region = var.region

  default_tags {
    tags = {
      Environment = var.environment
      Project     = var.project
      ManagedBy   = "terraform"
      Team        = var.team
      CostCenter  = var.cost_center
    }
  }
}

Required tags (enforce via OPA/Config Rules):

• Environment: dev/staging/prod
• Project: application name
• ManagedBy: terraform/manual/cdk
• Team: owning team
• CostCenter: billing allocation

Anti-Patterns

• Monolithic state file for entire infrastructure (one blast radius)
• terraform apply from laptops (no audit trail, no review)
• No state locking (corruption from concurrent runs)
• Hardcoded values (regions, account IDs, CIDRs)
• No security scanning in pipeline
• Using count for resources with side effects on index changes
• terraform destroy without confirming scope
• Committing .tfvars with secrets
• No drift detection (manual changes go unnoticed)
• Skipping terraform plan review before apply

References

• Anton Babenko's terraform-skill: comprehensive Terraform-specific patterns
• HashiCorp official agent skills: Terraform and Packer best practices
• Pulumi agent skills: Pulumi-specific patterns and migration guides