Gateway API troubleshooting

Find out why a Gateway API object does not route, from the objects' status, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Gateway API splits routing across roles, and each object reports what its controller made of it in status.conditions:

• A GatewayClass (cluster-scoped) names a controllerName. The implementation whose controller claims that name accepts the class.
• A Gateway uses a class and has listeners (name, port, protocol, optional hostname, TLS, allowedRoutes). A ListenerSet adds listeners to a Gateway from another object, if the Gateway's allowedListeners admits it.
• Routes (HTTPRoute, GRPCRoute, TLSRoute, TCPRoute, UDPRoute) attach to a Gateway or one listener through parentRefs, and send traffic to backendRefs. A route reports its status per parent, under status.parents.
• A ReferenceGrant in a namespace lets objects in another namespace refer to its objects (a route to a backend Service, a listener to a certificate).
• A BackendTLSPolicy makes the Gateway use TLS to a backend Service.

This skill covers what the API says and works the same in every implementation. How the data plane behaves is the implementation's part.

The commands use placeholders: my-namespace, my-gateway, my-app (the route and its Service) and my-app.example.com. Replace them with the real names. The built-in view role cannot read Gateway API objects (the project ships no read role); if they are forbidden, say so and ask the user for them. Never read Secrets: not the TLS certificates that listeners refer to, not CA bundles in Secrets. A listener's status says whether its certificate resolves; that is enough.

When to use

• A Gateway has no address, or kubectl get gateways shows PROGRAMMED Unknown or False.
• A route exists but requests for its host or path get 404, or one listener (often https) does not answer while another works.
• A route in another namespace, or a ListenerSet, does not attach to a shared Gateway.
• Applying Gateway API objects or CRDs fails (no matches for kind, a version not served, the safe-upgrades admission policy).

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

• The class belongs to Envoy Gateway (gateway.envoyproxy.io/gatewayclass-controller), and the objects are Accepted but traffic still fails, or its own policies are involved: k8s-envoy-gateway-troubleshooting.
• The class belongs to Istio (istio.io/gateway-controller): k8s-istio-troubleshooting. To Cilium (io.cilium/gateway-controller): k8s-cilium-troubleshooting.
• Everything is Accepted and resolved, but the backend Service has no ready endpoints or its pods fail: k8s-network-troubleshooting, k8s-workload-troubleshooting.
• The listener's certificate comes from cert-manager and is not issued: k8s-cert-manager-troubleshooting.
• Applying a CRD or an object is refused by a webhook or an admission policy you did not expect: k8s-api-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. See the classes, the Gateways and the routes.

kubectl get gatewayclasses
kubectl get gateways -A
kubectl get httproutes,grpcroutes -A
kubectl get listenersets,referencegrants,backendtlspolicies -A

If a kind is unknown to the cluster, see step 6.

2. Read the GatewayClass: its controller and whether that controller accepted it.

kubectl get gatewayclass my-class -o jsonpath='{.spec.controllerName}{"\n"}{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'

3. Read the Gateway and each listener: Accepted and Programmed for the Gateway, then per listener Accepted, Programmed, ResolvedRefs, Conflicted and the number of attached routes.

kubectl get gateway my-gateway -n my-namespace -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}{range .status.listeners[*]}{.name}{" routes="}{.attachedRoutes}{"\n"}{range .conditions[*]}{"  "}{.type}={.status} {.reason}: {.message}{"\n"}{end}{end}'
kubectl get gateway my-gateway -n my-namespace -o jsonpath='{range .spec.listeners[*]}{.name}{" "}{.protocol}{" "}{.port}{" "}{.hostname}{" allowedRoutes="}{.allowedRoutes}{" tls="}{.tls}{"\n"}{end}'

4. Read the route's status for each parent: Accepted (did it attach?) and ResolvedRefs (are its backends usable?). An empty status means no controller wrote anything for it.

kubectl get httproute my-app -n my-namespace -o jsonpath='{range .status.parents[*]}{.parentRef}{"\n"}{range .conditions[*]}{"  "}{.type}={.status} {.reason}: {.message}{"\n"}{end}{end}'
kubectl get httproute my-app -n my-namespace -o jsonpath='{.spec.parentRefs}{"\n"}{.spec.hostnames}{"\n"}{range .spec.rules[*]}{.backendRefs}{"\n"}{end}'

5. Check what the route points at: the backend Services and their ports, and ReferenceGrants in the backend's namespace for a cross-namespace reference.

kubectl get services -n my-namespace
kubectl get service my-app -n my-namespace -o jsonpath='{range .spec.ports[*]}{.name}{" "}{.port}{" "}{.appProtocol}{"\n"}{end}'
kubectl get referencegrants -n my-namespace -o yaml

6. Check the CRDs: which Gateway API version and channel is installed, and which API versions are served.

kubectl api-resources --api-group=gateway.networking.k8s.io
kubectl get crd gateways.gateway.networking.k8s.io -o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}{" "}{.metadata.annotations.gateway\.networking\.k8s\.io/channel}{"\n"}'
kubectl get crd tlsroutes.gateway.networking.k8s.io -o jsonpath='{range .spec.versions[*]}{.name}{" served="}{.served}{" storage="}{.storage}{"\n"}{end}'
kubectl get validatingadmissionpolicies

7. Open the matching reference and follow its steps:

Read the results

Conditions and reasons are from the Gateway API 1.6 specification; the quoted messages are Envoy Gateway 1.9's. Other implementations use the same reasons with their own messages.

• GatewayClass Accepted Unknown, reason Pending (Waiting for controller): no running controller claims its controllerName. A typo in the name, an implementation that is not installed or not running, or a class meant for another cluster. Every Gateway of the class then shows Accepted and Programmed Unknown, Pending, and gets no address.
• GatewayClass Accepted False, InvalidParameters: its parametersRef points at an object that does not exist or is malformed.
• Gateway Accepted False, ListenersNotValid (No listeners are valid: ...): every listener is invalid; read their conditions. When only some are invalid, the Gateway stays Accepted and Programmed and only those listeners fail.
• Gateway Programmed False, AddressNotAssigned: the implementation could not give it an address, for example a LoadBalancer Service with no load balancer, or no valid listener to serve.
• Listener ResolvedRefs False, InvalidCertificateRef (No valid secrets exist: certificate refs 0: Secret my-namespace/my-app-tls does not exist.): the TLS certificate the listener names does not exist or is not a valid TLS Secret. The listener is Programmed False and nothing serves its port. Do not read the Secret; the condition is the evidence.
• Listener Conflicted True, ProtocolConflict or HostnameConflict: listeners on the same port disagree on protocol, or share a hostname.
• Listener ResolvedRefs False, InvalidRouteKinds: allowedRoutes.kinds names a route kind the implementation does not support on that protocol.
• Route Accepted False, NoMatchingParent (No listeners match this parent ref): the parent ref's sectionName or port matches no listener of the Gateway.
• Route Accepted False, NotAllowedByListeners (No listeners included by this parent ref allowed this attachment.): the listener's allowedRoutes does not admit the route's namespace or kind. The default, which the API server fills in, is namespaces.from: Same: only routes from the Gateway's own namespace attach.
• Route Accepted False, NoMatchingListenerHostname: the route's hostnames and the listener's hostname do not intersect.
• A route with no status.parents at all: no controller considers itself responsible. The parentRefs name a Gateway that does not exist (a typo, the wrong namespace), or one whose class nobody accepts.
• Route ResolvedRefs False, BackendNotFound or RefNotPermitted: the backend does not exist, or is in another namespace without a ReferenceGrant there.
• Route Accepted and resolved, but requests still fail: the API side is done; look at the backend's endpoints and at the implementation (hand-offs above).

State the cause in one sentence, then the evidence you saw (the object, the condition, its reason and message, the spec fields that disagree), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Fix the object that is wrong, and prefer fixing it in its source (Git, a Helm chart), since a controller or a GitOps tool may set it back.

A route belongs to the team that publishes the app, so correcting a route's parentRefs, hostnames or backend is reversible and goes under "Ask first". A Gateway, its listeners and allowedRoutes, a GatewayClass, a ReferenceGrant and the CRDs are shared: changing them changes who can route what for every team on the Gateway. Never run those yourself; hand them to the Gateway's owners as proposed fixes. A GatewayClass's controllerName cannot be changed after it is created. Write each undo with the values you found.

Attach a route to the listener that exists (use the index of the parent ref you found):

Ask first: the agent runs this only after the user approves it.

kubectl patch httproute my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/parentRefs/0/sectionName", "value": "http"}]'
# Undo: kubectl patch httproute my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/parentRefs/0/sectionName", "value": "web"}]'

Let a listener admit routes from a team's namespace as well as its own. A selector replaces Same, so it must also match the Gateway's namespace, or the routes already attached there are dropped; the kubernetes.io/metadata.name label is on every namespace. This changes who can attach routes to a shared Gateway, so the agent does not run it:

Proposed fix: review before running. The agent does not run this.

kubectl patch gateway my-gateway -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/listeners/0/allowedRoutes", "value": {"namespaces": {"from": "Selector", "selector": {"matchExpressions": [{"key": "kubernetes.io/metadata.name", "operator": "In", "values": ["my-namespace", "my-other-namespace"]}]}}}}]'
# Undo: kubectl patch gateway my-gateway -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/listeners/0/allowedRoutes", "value": {"namespaces": {"from": "Same"}}}]'

Move a Gateway to the class whose controller runs. Its implementation then takes the Gateway over, so the agent does not run it:

Proposed fix: review before running. The agent does not run this.

kubectl patch gateway my-gateway -n my-namespace --type=merge -p '{"spec": {"gatewayClassName": "my-working-class"}}'
# Undo: kubectl patch gateway my-gateway -n my-namespace --type=merge -p '{"spec": {"gatewayClassName": "my-class"}}'

A missing listener certificate is fixed by creating or issuing it (for example a cert-manager Certificate whose secretName is the Secret the listener names), never by reading or copying another Secret. Give the user the name and namespace from the listener's condition.

Sources

Checked against the Gateway API 1.6 specification and source (v1.6.3), and on a Kubernetes 1.37 kind cluster with Envoy Gateway 1.9.2 (Gateway API CRDs v1.6.1, experimental channel):

• API specification 1.6: the condition types and reasons.
• Their definitions in the source: shared_types.go (route conditions), gateway_types.go (Gateway and listener conditions) and gatewayclass_types.go (GatewayClass conditions, SupportedVersion).
• Gateway, HTTPRoute, ReferenceGrant, ListenerSet and BackendTLSPolicy.
• Cross-namespace routing and Security model: the roles, allowedRoutes and ReferenceGrants.
• Versioning and the v1.6.3 release: channels, the bundle-version and channel CRD annotations, the safe-upgrades.gateway.networking.k8s.io admission policy in the install files.
• The v1.7.0 release notes: HTTPRoute retries move to the Standard channel; nothing here changes.
• The messages quoted are Envoy Gateway 1.9.2's, seen on the kind cluster; its source has them too.