Maintain infrastructure

a-novel/infra is a standard GitOps OpenTofu repository:

• a pull request shows each root's plan from a read-only identity;
• merging to master applies the roots the merge changed since the last successful deploy: bootstrap, then foundation, then the services in parallel;
• a daily drift check re-plans everything.

Services deploy independently. When one needs a change from another, that change merges first, in its own pull request.

Infrastructure only moves forward, and no procedure or workflow undoes a change. A faulty change is fixed by a new pull request, which may revert it, and a bad release by a hotfix. Damaged data goes back through recovery. Where a fix is slow, speed up the hotfix path rather than adding an undo.

Keep it that way. Prefer OpenTofu, provider features, Kubernetes and its operators, and native GitHub Actions over custom code. Custom code exists only for the workflow rules that can stop or misdeploy production, in tested .github/scripts, and for the health check every service shares, modules/smoke-check/smoke.sh.

Load prefer-small-solutions throughout, write-github-actions for workflows, plan-feature for architecture, and choose-dependency before adding a tool. Prose follows document-code.

The repository

• Production is one workload project and one cluster, and Kubernetes keeps the services apart. Each service's namespace runs under Pod Security's restricted level, two admission policies and default-deny network policies, with its own service accounts, budget, database and secrets. The deploy checks after each apply that admission still refuses what it must. Never propose a project per service.
• Inputs live in each root's committed terraform.tfvars. Secret values never do: ops add them with gcloud, and External Secrets generates those the cluster chooses itself, as Trust boundaries describes.
• Images are pinned as ghcr.io/…:tag@sha256:…. Every plan verifies each release image it changes: its release tag's own run built it, the release App published that tag as an immutable GitHub Release, and its version is not below the one deployed. The cluster's deploy then copies them to Artifact Registry with skopeo --preserve-digests.
• Releases pin themselves. Once all its images are published, a service or platform release dispatches pin-release.yaml through the shared publish-actions/deploy-release action. That opens their pull request as the Dependencies App, so it merges like a Renovate update. Renovate leaves ghcr.io/a-novel/** alone.
• Bot pull requests merge alone only when they are safe to. The auto-approver runs strict here: a release pin, or an update Renovate itself put on auto-merge that only moves versions. Renovate arms every minor and patch update except OpenTofu's and the Google providers', and a plan needing allow-downtime or allow-resource-deletion still holds it for a maintainer. OpenTofu and Google provider updates, majors, a moved action tag, and anything else a bot opens wait for a maintainer's or an admin's review on GitHub.
• Secrets are pinned as numeric versions.
• The docs are in docs/ops-basics.md, docs/architecture.md and docs/runbooks/.

Authority

• Agents never apply, dispatch or re-run production workflows, or change IAM by hand. Push a branch and let the pull request plan it. A failed deploy job that needs a re-run goes to the developer as gh run rerun <run ID> --repo a-novel/infra --failed.
• Use gcloud only for reads, and only when the user allows it in the session and holds ops access (check-access). A useful read: tofu plan -refresh=false -lock=false with GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token). Without ops access, prove plans offline as Change safety shows.
• Name every gcloud target explicitly, in a command you run and in one you hand to the developer: --project, and --zone, --region or --location whenever the command takes one, as literal values. Never rely on the active gcloud config, a CLOUDSDK_* variable, or a shell variable such as the runbooks' $PROD that the reader may not have set. A default left by other work runs the command on the wrong project or cluster. The values are the *_project_id, cluster_region and cluster.zones inputs in environments/production/foundation/terraform.tfvars.
• Ops hold read access, to the project and to the services' namespaces but never their Secrets, and secret-version rights. The named operators run Authentication's maintenance Jobs. Other developers hold none of them. Everything else goes through deploy.yaml, downtime.yaml for planned downtimes, or recovery.yaml for drills.
• Write runbook commands for the identity that actually runs them. A command ops cannot run belongs in a workflow.
• Deleting, replacing or forgetting a resource, or weakening its protection, needs the allow-resource-deletion label on the pull request, applied by a maintainer or an admin.
• A PostgreSQL major upgrade needs the allow-downtime label, applied the same way, and that service's planned downtime, scheduled by core, in progress.

Route the work

• Adding a service, or changing resources, state layout, releases, databases, network policies, service TLS or recovery: read Change safety.
• Adding a service or a platform, scraping a chart's metrics, or a release that changes the spans, tables or promises Grafana reads: read Grafana coverage.
• Workflows, tests, linting or tooling: read Tooling and tests.
• Creating, archiving or configuring a repository, or changing its rulesets, teams, labels, environments or secret access: edit the GitHub root, as infra's Repository config describes. A repository is an entry of its terraform.tfvars, and its class sets the rest. Creating one takes steps the root cannot: follow the stack's create-repository runbook. Hand the developer its release App install, ready to run, before the creating PR merges. A scoped secret joins that file only after an admin switches the org secret to selected repositories: one still visible to every repository fails each grant with HTTP 409. Each plan of the root spends the Plan App's hourly API budget, which every pull request shares, so plan-github plans only a pull request that changes the root (a-novel/infra#1380). Count the calls a new data source makes. github_repository_file pages through each file's commit history, so discovery reads files through github_rest_api, about 400 calls a plan (a-novel/infra#1385).
• Admitting a repository to the Drive visual-test identities: follow Subscribe a repository to Drive visual tests, whose later steps run after the merge.

Deliver

Prove a refactor is a no-op before review. Report each root's expected plan in the pull request: imports, in-place changes and deletions, with the reason for each. Separate stages that need the deletion or downtime label from those that do not. Stack them when they depend on one another. A stacked pull request plans only after its parent merges, on its next push or reopen. In the handoff, list what must be done by hand and who holds it: the label (a maintainer), then the admin-bypass merge after a required-check rename and the GitHub settings the root does not manage (core).

master requires an approval the author's account cannot give. When the user is an admin of a-novel/infra and opened the pull request, they review, then record it with gh workflow run approve-pr.yaml --repo a-novel/infra -f pull_request=<PR URL>. The workflow refuses anyone else's pull request, a bot's included: those wait for a maintainer's or an admin's review on GitHub. Never dispatch it yourself: that records a review nobody did.