agentleFS
k8s-servantk8s-servant/k8s-skills/skills/k8s-flux-troubleshooting/SKILL.md

k8s-flux-troubleshooting

Skill38 starsChanged 17 days agoApache-2.0View on GitHub

Use when Flux does not deploy or update something in Kubernetes: a GitRepository, OCIRepository or HelmRepository that is not Ready ("couldn't find remote ref", "authentication required"), a Kustomization that is not Ready ("kustomization path not found", "kustomize build failed", "dependency ... is not ready", health check timeouts), a HelmRelease that is not Ready (HelmChart "invalid chart reference", InstallFailed, UpgradeFailed, RetriesExceeded), changes in Git that never reach the cluster, or suspended Flux objects. Investigates with kubectl get, describe, events and logs, and proposes fixes. Reads first and asks before changing anything.

The file

We split this skill into the rules it gives an agent

3 rules. About 0 are common, 1 is distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Never read Secrets: not Git or registry credentials, not SOPS keys, not values or postBuild substitutions from Secrets, not Helm release …

    line 37
  2. Very specific:

    If the flux CLI is installed, flux get all -A and flux events -n flux-system show the same state in one view.

    line 106
  3. Distinctive:

    Ask Flux to reconcile an object now instead of waiting for its interval (use the current time as the value.

    line 171

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Flux troubleshooting

Find out why Flux does not deploy or update a workload, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Flux works in a chain. A source (GitRepository, OCIRepository, HelmRepository, Bucket) fetches an artifact. A Kustomization builds and applies a path from that artifact; a HelmRelease installs a chart, through a HelmChart that Flux creates from the HelmRelease. Every object reports Ready, and the first object in the chain that is not Ready holds the cause. Follow the chain from the object that deploys the workload back to its source.

The commands use placeholders: flux-system (where Flux and most of its objects live), my-app (the Kustomization or HelmRelease, and its source), and my-namespace (where the workload runs). Replace them with the real names; Flux objects can live in any namespace.

Never read Secrets: not Git or registry credentials, not SOPS keys, not values or postBuild substitutions from Secrets, not Helm release Secrets. The status of the Flux objects says when credentials are missing or wrong. For the same reason do not use flux tree, flux diff or flux debug ... --show-values / --show-vars: they read Secrets.

When to use

  • A change pushed to Git, an OCI registry or a Helm repository never reaches the cluster.
  • kubectl get kustomizations,helmreleases -A shows READY False or Unknown.
  • A GitRepository, OCIRepository, HelmRepository or HelmChart is not Ready.
  • Objects are SUSPENDED, or a HelmRelease is Stalled after failed installs or upgrades.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • Flux applied the workload, but its pods fail (crash, image, probe, Pending): k8s-workload-troubleshooting.
  • A HelmRelease's install or upgrade fails inside Helm (hooks, ownership, immutable fields): read references/helmreleases.md first; for the Helm side, k8s-helm-troubleshooting.
  • The Flux controllers themselves crash or stay Pending: k8s-workload-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. List the Flux objects and their state. READY, STATUS (the Ready message) and SUSPENDED are the columns that matter.

kubectl get kustomizations,helmreleases -A
kubectl get gitrepositories,ocirepositories,helmrepositories,helmcharts,buckets -A

2. Read the object that deploys the workload: its conditions, the source it points at, and its events.

kubectl describe kustomization my-app -n flux-system
kubectl get kustomization my-app -n flux-system -o jsonpath='{.spec.sourceRef}{"\n"}{.spec.path}{"\n"}{.spec.suspend}{"\n"}{.status.conditions}{"\n"}'
kubectl describe helmrelease my-app -n flux-system
kubectl events -n flux-system --for kustomization/my-app

3. Read its source. A Kustomization's sourceRef, or the HelmChart a HelmRelease uses (named <namespace>-<name>, in the HelmRelease's namespace) and its HelmRepository or OCIRepository.

kubectl describe gitrepository my-app -n flux-system
kubectl get gitrepository my-app -n flux-system -o jsonpath='{.spec.url}{"\n"}{.spec.ref}{"\n"}{.status.artifact.revision}{"\n"}{.status.conditions}{"\n"}'
kubectl describe helmchart flux-system-my-app -n flux-system
kubectl describe helmrepository -n flux-system

4. If the objects look right but nothing happens, check the controllers. Their logs name each object they reconcile.

kubectl get deployments -n flux-system
kubectl logs -n flux-system deployment/source-controller --tail=100 | grep my-app
kubectl logs -n flux-system deployment/kustomize-controller --tail=100 | grep my-app
kubectl logs -n flux-system deployment/helm-controller --tail=100 | grep my-app

If the flux CLI is installed, flux get all -A and flux events -n flux-system show the same state in one view; they are optional, kubectl is enough.

5. Open the matching reference and follow its steps:

What you seeReference
A GitRepository, OCIRepository, HelmRepository or HelmChart not Ready; couldn't find remote ref, authentication required, invalid chart referencereferences/sources.md
A Kustomization not Ready: kustomization path not found, kustomize build failed, dependency ... is not ready, health check timeout, Source artifact not foundreferences/kustomizations.md
A HelmRelease not Ready: SourceNotReady, InstallFailed, UpgradeFailed, RetriesExceeded, driftreferences/helmreleases.md
Nothing happens at all, SUSPENDED True, objects on removed v1beta APIs, controllers downreferences/suspend-and-controllers.md

Read the results

What the common findings mean (Flux 2.9):

  • A Kustomization with Source artifact not found, retrying in 30s (reason ArtifactFailed): its source has no artifact yet. The cause is on the source; read it.
  • A GitRepository with failed to checkout and determine revision: unable to clone '<url>': couldn't find remote ref "refs/heads/<branch>" (reason GitOperationFailed, FetchFailed True): the branch in spec.ref does not exist. authentication required or repository not found mean credentials or the URL.
  • kustomization path not found: stat ... (reason ArtifactFailed): the source is fine, but spec.path names a folder the artifact does not have.
  • kustomize build failed: ... (reason BuildFailed): the folder exists, but kustomize cannot build it; the message says why.
  • A HelmRelease with HelmChart '<ns>/<name>' is not ready: ... (reason SourceNotReady): read the HelmChart. invalid chart reference: ... no '<chart>' chart with version matching '<version>' found (reason InvalidChartReference, Stalled True) means the repository has no such version; Flux will not retry until the spec changes.
  • Stalled True with RetriesExceeded on a HelmRelease: install or upgrade failed and the retries are used up (install retries default to 0). Flux waits for a spec change or a reset.
  • SUSPENDED True: Flux skips the object; its conditions are from the last run before it was suspended and can be stale.
  • A source that is Ready, with the new commit in status.artifact.revision, but the cluster unchanged: look at the Kustomization or HelmRelease that uses it, and its lastAppliedRevision.

State the cause in one sentence, then the evidence you saw (the condition reason and message, the spec field, the revision), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Asking Flux to reconcile now, resuming a suspended object and patching a Flux object's spec are reversible, so they go under "Ask first" and run only after the user says yes. Deleting Flux objects or the workloads they manage, changing credentials, and reinstalling Flux are never run by you: hand them to the user as proposed fixes.

The fix almost always belongs in Git. Flux objects are themselves usually applied by a parent Kustomization (often flux-system), which puts back the version in Git at its next run; a patch on the live object is then only a test. Say so, and propose the change in the repository. Write each undo with the values you found.

Point a Kustomization at the right folder, to test before fixing Git:

Ask first: the agent runs this only after the user approves it.

kubectl patch kustomization my-app -n flux-system --type=merge -p '{"spec": {"path": "./kustomize"}}'
# Undo: kubectl patch kustomization my-app -n flux-system --type=merge -p '{"spec": {"path": "./kustomize/overlays/production"}}'

Ask Flux to reconcile an object now instead of waiting for its interval (use the current time as the value; any new value triggers a run):

Ask first: the agent runs this only after the user approves it.

kubectl annotate --overwrite gitrepository my-app -n flux-system reconcile.fluxcd.io/requestedAt=2026-10-24T12:00:00Z
# Undo: nothing to undo; the annotation only asks for one reconciliation

Resume an object that was suspended (only once the reason for suspending it is gone):

Ask first: the agent runs this only after the user approves it.

kubectl patch kustomization my-app -n flux-system --type=merge -p '{"spec": {"suspend": false}}'
# Undo: kubectl patch kustomization my-app -n flux-system --type=merge -p '{"spec": {"suspend": true}}'

Sources

Checked against the Flux 2.9 documentation and the controllers' source (flux2 v2.9.6, source-controller and kustomize-controller v1.9.6, helm-controller v1.6.5):

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.