Stakater Reloader Project Memory

Project Purpose

Reloader is a Kubernetes operator that automatically triggers rolling restarts of workloads when the ConfigMaps or Secrets they reference are updated. Without it, Kubernetes does not restart pods when configuration changes — operators must do it manually or rely on GitOps pipelines.

What it watches: ConfigMaps, Secrets, Namespaces, and (optionally) SecretProviderClassPodStatus (CSI-mounted secrets).

Workload types it can reload: Deployment, StatefulSet, DaemonSet, CronJob, Job, Argo Rollout, and OpenShift DeploymentConfig.

How restarts are triggered: Two strategies (selected via --reload-strategy):

1. env-vars (default) — injects an environment variable (STAKATER_{NAME}_{TYPE}) into every container with the SHA1 hash of the resource's data. A change in data changes the env var value, causing Kubernetes to restart pods.
2. annotations — writes the SHA1 hash into the pod template's annotations, which also forces a rollout.

The core problem it solves: ConfigMaps and Secrets are decoupled from pod lifecycle in Kubernetes. Applications reading config at startup see stale data after a config update unless pods are restarted. Reloader closes that gap automatically and selectively.

Potential improvements observed:

• Duplicate reload suppression: If a workload references both a ConfigMap and a Secret that are updated in the same controller reconcile cycle, it may get reloaded twice. Could be solved with a per-workload debounce map keyed by namespace/name/resourceVersion, flushed after a short TTL.
• CronJob/Job reload is destructive: Jobs are deleted and recreated on change, which loses run history. Could instead only annotate the CronJob template without spawning a new Job.
• No per-resource reload rate limiting: A rapid-fire ConfigMap update (e.g., from a CI pipeline) can trigger many restarts. A cooldown window per resource would help.
• CSI integration gap: CSI volumes are watched at the SecretProviderClassPodStatus level, but the link back to the workload is indirect and may miss edge cases. Needs a direct map from SecretProviderClass → workloads that mount it.

Repo Map

Core Runtime Flow

1. Entry — main.go:10 calls app.Run().

2. CLI Init — internal/pkg/app/app.go calls cmd.NewReloaderCommand() which registers all Cobra flags from options/flags.go and runs startReloader().

3. Client Setup — pkg/kube/client.go: builds kube.Clients with:

• kubernetes.Interface — standard k8s client
• appsclient.Interface — OpenShift client (auto-detected by probing deploymentconfigs)
• argorollout.Interface — if --is-Argo-Rollouts=true
• csiclient.Interface — if --enable-csi-integration

4. Controller Creation — startReloader() iterates kube.ResourceMap (configmaps, secrets, namespaces, and optionally secretproviderclasspodstatuses) and calls controller.NewController() for each resource in each watched namespace.

5. Informer/Queue — controller.NewController():

• Creates a cache.NewFilteredListWatchFromClient with label/field selectors.
• Registers Add, Update, Delete event handlers.
• Creates a workqueue.TypedRateLimitingQueue for async processing.

6. Event Detection:

• Add — enqueues only if ReloadOnCreate is enabled (skips during initial sync unless SyncAfterRestart).
• Update — compares SHA of old vs new object data; enqueues only on real changes.
• Delete — enqueues only if ReloadOnDelete is enabled.
• Namespace events update selectedNamespacesCache for namespace-selector filtering.

7. Handler Dispatch — The queue worker calls handler.Handle() on the dequeued item. Three handler types:

• ResourceCreatedHandler (create.go) — fires doRollingUpgrade or sends webhook.
• ResourceUpdatedHandler (update.go) — fires doRollingUpgrade or sends webhook.
• ResourceDeleteHandler (delete.go) — calls invokeDeleteStrategy (removes env vars or clears annotation).

8. Workload Discovery — doRollingUpgrade() (upgrade.go:181) calls rollingUpgrade() for each workload type. For each type, ItemsFunc lists all workloads in the namespace, then pkg/common.ShouldReload() checks annotations to decide which ones need reloading.

9. Reload Execution — invokeReloadStrategy() either:

• env-vars: mutates container env vars; uses JSON patch if SupportsPatch=true, full update otherwise.
• annotations: writes SHA to pod template annotations; same patch/update split.

10. Post-reload — optionally pauses the Deployment via pause_deployment.go, records Kubernetes Events via recorder, updates Prometheus metrics, sends alert webhooks.

HA Mode: if --enable-ha, internal/pkg/leadership/ runs Kubernetes Lease-based leader election. Only the leader runs controllers; losing leadership stops them and marks the pod unhealthy.

HTTP Server: port :9090 serves /metrics (Prometheus) and liveness/readiness probes.

Reload Behavior And Annotations

All annotation names are configurable via CLI flags; the values below are defaults.

Trigger Annotations (on workloads)

Exclude Annotations (on workloads)

Behavior Annotations (on workloads)

Search/Match Pattern

The reloader.stakater.com/search annotation on a workload pairs with reloader.stakater.com/match: "true" on a ConfigMap or Secret. Any workload with search: true will reload when any match: true resource changes.

Global Flag Overrides

• --auto-reload-all — reload all workloads on any ConfigMap/Secret change; annotation not required.
• --resources-to-ignore=configMaps or =secrets — skip one type entirely.
• --ignored-workload-types=jobs,cronjobs — skip Job and CronJob reload.
• --namespaces-to-ignore — comma-separated namespace names to skip.
• --namespace-selector — only watch namespaces with matching labels.
• --resource-label-selector — only watch ConfigMaps/Secrets with matching labels.

Precedence Rules

1. reloader.stakater.com/ignore: "true" wins everything — workload is skipped.
2. Exclude annotations override include annotations for specific named resources.
3. Named annotations (.../reload) are checked before auto annotations.
4. --auto-reload-all is the lowest-priority fallback (only applies if no annotation matches).
5. Annotations are checked on both the workload and its pod template (pod template takes precedence in some paths — verify in pkg/common/common.go:ShouldReload()).

Workload Support

Reload flow per workload: doRollingUpgrade() → rollingUpgrade() per type → ItemsFunc lists workloads → ShouldReload() filters → invokeReloadStrategy() patches or updates → optional pause + metrics + alert.

CSI Support

Enabled by: --enable-csi-integration

What is watched: SecretProviderClassPodStatus resources (from sigs.k8s.io/secrets-store-csi-driver). Resource name constant: constants.SecretProviderClassController = "secretproviderclasspodstatuses".

How it works:

1. The CSI driver injects secrets into pods as volume mounts and tracks injection state via SecretProviderClassPodStatus objects.
2. Reloader watches these objects for version changes.
3. When a version change is detected, it computes a SHA of the object's IDs and versions.
4. It then looks up the referenced SecretProviderClass and treats the event like a Secret update, triggering workload reloads.

Workload annotation: secretproviderclass.reloader.stakater.com/reload: "my-spc" or secretproviderclass.reloader.stakater.com/auto: "true".

Required: CSI CRDs must be installed in the cluster. Reloader auto-detects their presence at startup.

Env var postfix: STAKATER_{NAME}_SECRETPROVIDERCLASS.

Known limitations:

• Only works for secrets mounted as volumes via CSI, not env-var-based CSI injection.
• The link from SecretProviderClassPodStatus → workload is indirect; edge cases may be missed.
• Requires the CSI driver CRDs to be pre-installed; Reloader won't start CSI controller if CRDs are absent.

Build, Test, And Run Commands

Go version: go 1.26.2 (from go.mod)

Docker image: ghcr.io/stakater/reloader — multi-arch (amd64, arm64, arm), distroless nonroot base.

Helm chart: deployments/kubernetes/chart/reloader/ — install via Helm or kubectl apply -f deployments/kubernetes/reloader.yaml.

Branch policy: master is in feature freeze, pr-title.yaml only lets fix: and chore: PR titles through. New features go to v2. The master chart stays on 2.x, so its version gate compares against helm search repo stakater/reloader --version '^2', not the newest chart (v2 publishes 3.x to the same repo). Likewise the UBI builder image in pull_request.yaml comes from the highest stable v1.* tag.

Coding Conventions

Package boundaries: Each internal/pkg/<name> package has a single clear responsibility. Cross-package access goes through exported types/functions only.

Error handling: logrus.Errorf(...) for non-fatal, logrus.Fatalf(...) for startup failures. Errors are returned up the call stack and logged at the point of action, not at every layer. Retry uses k8s.io/client-go/util/retry.RetryOnConflict.

Logging: logrus with structured fields. Format controlled by --log-format=json flag. Log level controlled by --log-level. Messages follow the pattern: "Changes detected in '%s' of type '%s' in namespace '%s'".

Kubernetes client patterns: All k8s operations go through the kube.Clients struct. Use context.TODO() for context (no request-scoped contexts). List/watch via informers, not polling.

Callback pattern: Workload-specific logic is encapsulated in callbacks.RollingUpgradeFuncs structs returned by handler.Get*RollingUpgradeFuncs(). Adding a new workload type = add a new RollingUpgradeFuncs factory function and call it in doRollingUpgrade().

Test style: Standard testing.T, testify/assert. Fake k8s objects via testutil/kube.go. Tests live alongside source in the same package. Large integration-style tests in handler/upgrade_test.go.

Naming patterns:

• Annotation variables: XxxUpdateOnChangeAnnotation, XxxReloaderAutoAnnotation
• Callback funcs: GetXxxItem, GetXxxItems, UpdateXxx, PatchXxx
• Handler factories: GetXxxRollingUpgradeFuncs()

Adding new behavior: Add flag to options/flags.go + common.ReloaderOptions struct → wire in cmd/reloader.go → implement logic in handler/ or callbacks/ → add metrics recording → write tests in *_test.go.

Gotchas And Risks

Duplicate reloads: If a workload references multiple ConfigMaps/Secrets and all change simultaneously, each change event fires a separate reload. No deduplication exists within a reconcile window. This can cause unnecessary rolling restarts.

Controller init guard: secretControllerInitialized and configmapControllerInitialized booleans in controller/controller.go prevent processing Add events during the initial list/sync (to avoid reloading everything on startup). If --sync-after-restart is set, both are pre-set to true, bypassing the guard. Be careful when this interacts with --reload-on-create.

Namespace filtering: --namespaces-to-ignore does a name match; --namespace-selector watches namespaces by label and caches them in selectedNamespacesCache. The cache is updated on Namespace Add/Update/Delete events. A race between cache population and first ConfigMap event could cause missed reloads on startup in label-selected deployments.

RBAC: Reloader requires get/list/watch on secrets and configmaps, and get/list/watch/update/patch on all workload types it manages. Missing RBAC silently causes no reloads (not an error — just empty lists). Check ClusterRole in deployments/kubernetes/chart/reloader/templates/.

GitOps drift: If a GitOps tool (Flux, ArgoCD) manages the same Deployments, annotation or env var changes made by Reloader will be detected as drift and reverted. Use --reload-strategy=annotations with care in GitOps setups; env-vars strategy is generally safer since it modifies the pod template rather than workload-level annotations.

Annotation precedence edge case: Annotations are checked first on the workload object, then on the pod template. If both are set to conflicting values, the behavior depends on which path ShouldReload() hits first. Verify in pkg/common/common.go.

CronJob/Job destructive reload: Job recreation deletes the old Job. Any in-flight pod from that Job will be terminated. This is intentional but surprising. There is no protection for long-running jobs.

OpenShift DeploymentConfig: Auto-detected by probing for the deploymentconfigs resource. If the probe fails at startup, OpenShift support is silently disabled. Check pkg/kube/client.go.

Argo Rollouts: Must be explicitly enabled via --is-Argo-Rollouts=true. Without it, Rollout objects are never listed. The SupportsPatch=false means full object updates are used — be aware of potential conflicts with Argo's own controller.

CSI rotation behavior: SecretProviderClassPodStatus is updated by the CSI driver when secrets rotate. Reloader reacts to those updates. However, if the CSI driver updates the status in a way that doesn't change the versions Reloader tracks, the reload will be missed.

Backward compatibility: Annotation names are configurable, so changing defaults would break existing clusters. Never change default annotation values without a migration path.

Tests to update for risky changes: handler/upgrade_test.go (large suite covering all workload types), controller/controller_test.go (event handling), pkg/common/common_test.go (reload decision logic).

Open Questions

• Exact ShouldReload() precedence: The code in pkg/common/common.go checks annotations in a specific order. The exact tie-breaking when both workload-level and pod-template-level annotations are set should be verified by reading that function fully before making annotation behavior changes.
• CSI → workload mapping: How exactly does Reloader map a SecretProviderClassPodStatus change back to workloads? Is it via the SecretProviderClass name matching an annotation on the workload, or via volume reference scanning? Needs confirmation before adding CSI-related features.
• ContainerPatchPathFunc field: RollingUpgradeFuncs has a ContainerPatchPathFunc field, but it is not documented — unclear if/how it differs from ContainersFunc in patch scenarios.
• Webhook vs alert: --webhook-url replaces reloading with a POST request. ALERT_WEBHOOK_URL env var sends an alert after reloading. These are two different mechanisms; the naming is confusing and easy to conflate.
• Load test scenarios S7–S13: Only S1, S4, and S6 are confirmed from CI. The behavior and coverage of the remaining scenarios is unknown without reading test/loadtest/ in full.
• SyncAfterRestart semantics: Flag docs say it "syncs add events after restart" but only if ReloadOnCreate is also true. The interaction between these two flags in HA mode (where controllers restart on leader change) needs verification.

Important Files