agentleFS
k8s-servantk8s-servant/k8s-skills/skills/k8s-kyverno-troubleshooting/SKILL.md

k8s-kyverno-troubleshooting

Skill38 starsChanged 16 days agoApache-2.0View on GitHub

Use when Kyverno blocks, changes or fails to police resources in Kubernetes: "admission webhook \"validate.kyverno.svc-fail\" denied the request", "resource ... was blocked due to the following policies", "Policy ... failed" from a ValidatingPolicy, a policy that should block but only reports (Audit), PolicyReports with failures, a policy that is not Ready, mutate or generate rules that do nothing, UpdateRequests stuck, or "failed calling webhook" for kyverno when its pods are down. Covers ClusterPolicy and Policy (kyverno.io) and the CEL policies (policies.kyverno.io). Investigates with kubectl get, describe, events and logs, and proposes fixes. Reads first and asks before changing anything.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Read first, ask before changing: Run the investigation commands freely.

    line 22
  2. Very specific:

    Every create or update fails with failed calling webhook naming kyverno.

    line 51

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Kyverno troubleshooting

Find out why Kyverno denies, changes or does not police a resource, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Kyverno runs as admission webhooks plus background controllers. Two policy families exist side by side:

  • ClusterPolicy and Policy (kyverno.io/v1): rules with validate, mutate, generate or verifyImages. Deprecated in Kyverno 1.19 and planned for removal in 1.20; the API server prints a deprecation warning for them.
  • CEL policies (policies.kyverno.io/v1): ValidatingPolicy, MutatingPolicy, GeneratingPolicy, DeletingPolicy, ImageValidatingPolicy, and their Namespaced* variants.

A policy is a cluster-wide guardrail someone chose on purpose. The usual fix is to make the resource comply, not to weaken the policy; changing or excepting a policy is a decision for its owners.

The commands use placeholders: my-namespace, my-app, my-policy and my-rule. Replace them with the real names. Never read Secrets (policies can reference them for image verification or context; read their names only).

When to use

  • Applying a manifest, a Helm install or a GitOps sync fails with a Kyverno denial.
  • A resource is admitted although a policy should have blocked it.
  • PolicyReports show FAIL results, or a policy is not Ready.
  • A mutate or generate rule does not change or create what it should.
  • Every create or update fails with failed calling webhook naming kyverno.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • The Kyverno pods crash or stay Pending: k8s-workload-troubleshooting.
  • A different admission controller denies the request (Gatekeeper, a cloud policy add-on): k8s-gatekeeper-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. Read the denial the user got. It names the webhook, the policy and the rule: validate.kyverno.svc-fail (ClusterPolicy and Policy) or vpol.validate.kyverno.svc-fail (ValidatingPolicy). Rules that Kyverno generated for Deployments, StatefulSets, Jobs and similar from a Pod rule are named autogen-<rule> (autogen-cronjob-<rule> for CronJobs).

2. List the policies and their state.

kubectl get clusterpolicies,policies -A
kubectl get validatingpolicies,mutatingpolicies,generatingpolicies,imagevalidatingpolicies
kubectl get namespacedvalidatingpolicies -A

3. Read the policy named in the denial: its match, its failure action, its rules.

kubectl get clusterpolicy my-policy -o yaml
kubectl describe clusterpolicy my-policy
kubectl get validatingpolicy my-policy -o jsonpath='{.spec.validationActions}{"\n"}{.spec.matchConstraints}{"\n"}{.spec.validations}{"\n"}{.status}{"\n"}'

For ClusterPolicy and Policy, the failure action is validate.failureAction on each rule (Enforce blocks, Audit only reports), or the older spec.validationFailureAction. For a ValidatingPolicy it is spec.validationActions (Deny, Audit, Warn).

4. Read the policy reports for the resource: results of background scans and of admission in Audit mode.

kubectl get policyreports -n my-namespace
kubectl get clusterpolicyreports
kubectl get policyreports -n my-namespace -o jsonpath='{range .items[*].results[?(@.result=="fail")]}{.policy}{" "}{.rule}{" "}{.message}{"\n"}{end}'
kubectl events -n my-namespace | grep -i policy

5. If Kyverno itself may be the problem, read its pods, its webhook configurations and its logs.

kubectl get deployments,pods -n kyverno
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations | grep -i kyverno
kubectl logs -n kyverno deployment/kyverno-admission-controller --tail=100
kubectl get configmap kyverno -n kyverno -o jsonpath='{.data.resourceFilters}{"\n"}'

6. Open the matching reference and follow its steps:

What you seeReference
A denial, was blocked due to the following policies, Policy ... failed, a policy that should block but only reports, PolicyReport failuresreferences/validation.md
Mutations that do not happen, generated resources missing, UpdateRequests, cleanup policiesreferences/mutate-and-generate.md
failed calling webhook naming kyverno, a policy not Ready, namespaces Kyverno ignores, controllers downreferences/webhooks-and-controllers.md

Read the results

What the common findings mean (Kyverno 1.19):

  • admission webhook "validate.kyverno.svc-fail" denied the request: resource <Kind>/<ns>/<name> was blocked due to the following policies, followed by <policy>: and <rule>: <message>: a ClusterPolicy or Policy rule in Enforce mode failed. The message after the rule name is the policy's own text plus the path that failed (failed at path /spec/template/metadata/labels/...).
  • admission webhook "vpol.validate.kyverno.svc-fail" denied the request: Policy <name> failed: <message>: a ValidatingPolicy with Deny failed.
  • An autogen- rule name: the policy was written for Pods; Kyverno applies it to the pod template of the controller you applied.
  • Admitted, and a fail result in a PolicyReport: the rule runs in Audit (or a ValidatingPolicy with Audit): Kyverno reports and lets it through.
  • No report and no denial at all: the policy does not match the resource (kinds, namespaces, selectors), the namespace is in the resourceFilters of the kyverno ConfigMap (by default kube-system, kube-public, kube-node-lease and kyverno), or the policy is not Ready.
  • failed calling webhook "validate.kyverno.svc-fail" (or another kyverno webhook): the API server cannot reach Kyverno, and the webhook fails closed. Nothing about the resource is wrong.

State the cause in one sentence, then the evidence you saw (the denial or report, the policy and rule, its failure action, its match), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Make the resource comply: change it in its source (manifest, Helm values, GitOps repository), so the next apply passes. Changing a policy's failure action or match, adding a PolicyException, and touching Kyverno's webhook configurations or ConfigMap change what is enforced for everyone: never run them yourself; hand them to the policy's owners as proposed fixes and say why. Write each undo with the values you found.

Bring back a Kyverno admission controller that was scaled to zero (write the undo with the replicas you found):

Ask first: the agent runs this only after the user approves it.

kubectl scale deployment/kyverno-admission-controller -n kyverno --replicas=1
# Undo: kubectl scale deployment/kyverno-admission-controller -n kyverno --replicas=0

For a resource that must not comply (a real exception), the policy's owners add a PolicyException. Exceptions are turned off by default in Kyverno and must be enabled first (--enablePolicyException); see references/validation.md.

Sources

Checked against the Kyverno 1.19 documentation and source (v1.19.2):

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.