k8s-kyverno-troubleshooting
Skill38 starsChanged 16 days agoApache-2.0View on GitHub
Use when Kyverno blocks, changes or fails to police resources in Kubernetes: "admission webhook \"validate.kyverno.svc-fail\" denied the request", "resource ... was blocked due to the following policies", "Policy ... failed" from a ValidatingPolicy, a policy that should block but only reports (Audit), PolicyReports with failures, a policy that is not Ready, mutate or generate rules that do nothing, UpdateRequests stuck, or "failed calling webhook" for kyverno when its pods are down. Covers ClusterPolicy and Policy (kyverno.io) and the CEL policies (policies.kyverno.io). Investigates with kubectl get, describe, events and logs, and proposes fixes. Reads first and asks before changing anything.
In this repository32 other files it gives its agents.
- AGENTS.mdAGENTS.md
- CLAUDE.mdCLAUDE.md
- k8s-aks-troubleshootingSkill
- k8s-api-troubleshootingSkill
- k8s-argocd-troubleshootingSkill
- k8s-autoscaling-troubleshootingSkill
- k8s-cert-manager-troubleshootingSkill
- k8s-cilium-troubleshootingSkill
- k8s-cloudnative-pg-troubleshootingSkill
- k8s-cluster-reviewSkill
- k8s-coredns-troubleshootingSkill
- k8s-envoy-gateway-troubleshootingSkill
- k8s-envoy-troubleshootingSkill
- k8s-external-secrets-troubleshootingSkill
- k8s-fluent-bit-troubleshootingSkill
- k8s-flux-troubleshootingSkill
- k8s-gatekeeper-troubleshootingSkill
- k8s-gateway-api-troubleshootingSkill
- k8s-grafana-troubleshootingSkill
- k8s-helm-troubleshootingSkill
- k8s-istio-troubleshootingSkill
- k8s-loki-troubleshootingSkill
- k8s-network-troubleshootingSkill
- k8s-node-troubleshootingSkill
- k8s-opentelemetry-collector-troubleshootingSkill
- k8s-prometheus-troubleshootingSkill
- k8s-rbac-troubleshootingSkill
- k8s-storage-troubleshootingSkill
- k8s-triageSkill
- k8s-upgrade-readinessSkill
- k8s-velero-troubleshootingSkill
- k8s-workload-troubleshootingSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 22
Read first, ask before changing: Run the investigation commands freely.
- Very specific: line 51
Every create or update fails with failed calling webhook naming kyverno.
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Kyverno troubleshooting
Find out why Kyverno denies, changes or does not police a resource, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
Kyverno runs as admission webhooks plus background controllers. Two policy families exist side by side:
- ClusterPolicy and Policy (
kyverno.io/v1): rules withvalidate,mutate,generateorverifyImages. Deprecated in Kyverno 1.19 and planned for removal in 1.20; the API server prints a deprecation warning for them. - CEL policies (
policies.kyverno.io/v1): ValidatingPolicy, MutatingPolicy, GeneratingPolicy, DeletingPolicy, ImageValidatingPolicy, and theirNamespaced*variants.
A policy is a cluster-wide guardrail someone chose on purpose. The usual fix is to make the resource comply, not to weaken the policy; changing or excepting a policy is a decision for its owners.
The commands use placeholders: my-namespace, my-app, my-policy and my-rule.
Replace them with the real names. Never read Secrets (policies can reference them for
image verification or context; read their names only).
When to use
- Applying a manifest, a Helm install or a GitOps sync fails with a Kyverno denial.
- A resource is admitted although a policy should have blocked it.
- PolicyReports show
FAILresults, or a policy is not Ready. - A mutate or generate rule does not change or create what it should.
- Every create or update fails with
failed calling webhooknaming kyverno.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:
- The Kyverno pods crash or stay Pending:
k8s-workload-troubleshooting. - A different admission controller denies the request (Gatekeeper, a cloud policy
add-on):
k8s-gatekeeper-troubleshooting.
Investigate
Work through the steps in order. Stop when you have the cause and its evidence.
1. Read the denial the user got. It names the webhook, the policy and the rule:
validate.kyverno.svc-fail (ClusterPolicy and Policy) or vpol.validate.kyverno.svc-fail
(ValidatingPolicy). Rules that Kyverno generated for Deployments, StatefulSets, Jobs
and similar from a Pod rule are named autogen-<rule> (autogen-cronjob-<rule> for
CronJobs).
2. List the policies and their state.
kubectl get clusterpolicies,policies -A
kubectl get validatingpolicies,mutatingpolicies,generatingpolicies,imagevalidatingpolicies
kubectl get namespacedvalidatingpolicies -A3. Read the policy named in the denial: its match, its failure action, its rules.
kubectl get clusterpolicy my-policy -o yaml
kubectl describe clusterpolicy my-policy
kubectl get validatingpolicy my-policy -o jsonpath='{.spec.validationActions}{"\n"}{.spec.matchConstraints}{"\n"}{.spec.validations}{"\n"}{.status}{"\n"}'For ClusterPolicy and Policy, the failure action is validate.failureAction on each
rule (Enforce blocks, Audit only reports), or the older spec.validationFailureAction.
For a ValidatingPolicy it is spec.validationActions (Deny, Audit, Warn).
4. Read the policy reports for the resource: results of background scans and of admission in Audit mode.
kubectl get policyreports -n my-namespace
kubectl get clusterpolicyreports
kubectl get policyreports -n my-namespace -o jsonpath='{range .items[*].results[?(@.result=="fail")]}{.policy}{" "}{.rule}{" "}{.message}{"\n"}{end}'
kubectl events -n my-namespace | grep -i policy5. If Kyverno itself may be the problem, read its pods, its webhook configurations and its logs.
kubectl get deployments,pods -n kyverno
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations | grep -i kyverno
kubectl logs -n kyverno deployment/kyverno-admission-controller --tail=100
kubectl get configmap kyverno -n kyverno -o jsonpath='{.data.resourceFilters}{"\n"}'6. Open the matching reference and follow its steps:
| What you see | Reference |
A denial, was blocked due to the following policies, Policy ... failed, a policy that should block but only reports, PolicyReport failures | references/validation.md |
| Mutations that do not happen, generated resources missing, UpdateRequests, cleanup policies | references/mutate-and-generate.md |
failed calling webhook naming kyverno, a policy not Ready, namespaces Kyverno ignores, controllers down | references/webhooks-and-controllers.md |
Read the results
What the common findings mean (Kyverno 1.19):
admission webhook "validate.kyverno.svc-fail" denied the request: resource <Kind>/<ns>/<name> was blocked due to the following policies, followed by<policy>:and<rule>: <message>: a ClusterPolicy or Policy rule in Enforce mode failed. The message after the rule name is the policy's own text plus the path that failed (failed at path /spec/template/metadata/labels/...).admission webhook "vpol.validate.kyverno.svc-fail" denied the request: Policy <name> failed: <message>: a ValidatingPolicy withDenyfailed.- An
autogen-rule name: the policy was written for Pods; Kyverno applies it to the pod template of the controller you applied. - Admitted, and a
failresult in a PolicyReport: the rule runs inAudit(or a ValidatingPolicy withAudit): Kyverno reports and lets it through. - No report and no denial at all: the policy does not match the resource (kinds,
namespaces, selectors), the namespace is in the
resourceFiltersof thekyvernoConfigMap (by defaultkube-system,kube-public,kube-node-leaseandkyverno), or the policy is not Ready. failed calling webhook "validate.kyverno.svc-fail"(or another kyverno webhook): the API server cannot reach Kyverno, and the webhook fails closed. Nothing about the resource is wrong.
State the cause in one sentence, then the evidence you saw (the denial or report, the policy and rule, its failure action, its match), then the proposed fix.
Proposed fix
Show each fix to the user before running it: what it changes and how to undo it. Make the resource comply: change it in its source (manifest, Helm values, GitOps repository), so the next apply passes. Changing a policy's failure action or match, adding a PolicyException, and touching Kyverno's webhook configurations or ConfigMap change what is enforced for everyone: never run them yourself; hand them to the policy's owners as proposed fixes and say why. Write each undo with the values you found.
Bring back a Kyverno admission controller that was scaled to zero (write the undo with the replicas you found):
Ask first: the agent runs this only after the user approves it.
kubectl scale deployment/kyverno-admission-controller -n kyverno --replicas=1
# Undo: kubectl scale deployment/kyverno-admission-controller -n kyverno --replicas=0For a resource that must not comply (a real exception), the policy's owners add a
PolicyException. Exceptions are turned off by default in Kyverno and must be enabled
first (--enablePolicyException); see
references/validation.md.
Sources
Checked against the Kyverno 1.19 documentation and source (v1.19.2):
- Troubleshooting.
- Policy types and ValidatingPolicy.
- Validate rules, mutate rules and generate rules.
- Policy reports and policy exceptions.
- Upgrading to 1.19 (ClusterPolicy deprecated) and releases: 1.19 is tested with Kubernetes 1.33 to 1.35; these steps were also seen working on 1.37.
- Customization: container
flags and the
resourceFiltersConfigMap key. - Denial formats from the source: block.go, vpol/handler.go, autogen rule.go. The denials quoted here were also seen on a Kubernetes 1.37 kind cluster with Kyverno 1.19.2.
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

