k8s-networking

Patterns and decisions for DNS, Ingress, TLS, network policies, and VPN access in self-hosted Kubernetes clusters.

DNS Record Quick Reference

Self-hosting setup: Create an A record pointing your domain apex (@) to your server's public IP. For www redirect to work with NGINX Ingress, also create an A record for www (or a wildcard *.example.com).

TTL guidance: Use 300s (5 min) when making changes; raise to 3600s (1 hr) once stable. Values over a few days are often ignored by resolvers.

Wildcard records: *.example.com matches all first-level subdomains. Does not match second-level (*.sub.example.com needs a separate record). Most DNS providers do not support wildcards for ALIAS records.

Flannel CNI (k3s default)

k3s ships with Flannel using the VXLAN backend by default. Flannel assigns a /24 subnet per node from the cluster CIDR (10.42.0.0/16 by default), supporting up to 255 nodes and ~254 pods per node.

Backend options:

k3s defaults to know:

• Cluster CIDR: 10.42.0.0/16 (pods)
• Service CIDR: 10.43.0.0/16 (services)
• Cluster DNS: 10.43.0.10 (CoreDNS)
• NodePort range: 30000–32767

Flannel itself does not enforce NetworkPolicies. k3s embeds the kube-router network policy controller internally (it does not run as a visible pod) to translate NetworkPolicy resources into iptables rules.

Ingress NGINX Controller

Why NGINX over Traefik: More mature, larger community, better HTTP/2 and compression support, lower memory footprint under load. k3s ships Traefik by default — disable it at install time with --disable=traefik.

Install

# Check https://github.com/kubernetes/ingress-nginx/releases for the current version
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.1/deploy/static/provider/cloud/deploy.yaml

Use the cloud static manifest for k3s (creates a LoadBalancer service; k3s's Klipper handles the actual load balancing). Use bare metal only if you need a NodePort service instead.

After install, verify:

kubectl get pods -n ingress-nginx
kubectl get svc -n ingress-nginx  # ingress-nginx-controller should show EXTERNAL-IP
curl http://<your-server-ip>      # expect 404 (no Ingress rules yet)

Basic Ingress (HTTP only)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  ingressClassName: nginx
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-svc
                port:
                  number: 80

Key rules:

• namespace must match the target Service's namespace — cross-namespace routing returns 404.
• ingressClassName: nginx is required even with a single controller.
• from-to-www-redirect: "true" redirects www.example.com → example.com (or vice versa, depending on which host is in rules). Requires a DNS record for www to exist.

Useful annotations

TLS with cert-manager + Let's Encrypt

Recommended approach: cert-manager + Let's Encrypt production issuer + HTTP-01 challenge. Free, automated, renews at 2/3 of validity (every ~60 days of a 90-day cert).

Staging vs production:

• Staging (acme-staging-v02.api.letsencrypt.org): Higher rate limits, untrusted root — use to test your setup before going live. Always test with staging first.
• Production (acme-v02.api.letsencrypt.org): Trusted by all browsers; rate-limited (50 new certs/domain/week, renewals don't count). Switch to production after staging confirms your setup works.

For the full TLS setup guide, see references/tls-setup.md.

Quick reference: Ingress with TLS

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-app
  namespace: production
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-production
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - example.com
        - www.example.com
      secretName: example-com-tls
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-app-svc
                port:
                  number: 80

Certificate issuance takes 10 seconds to a few minutes. During this window, NGINX will redirect HTTP → HTTPS but the cert isn't ready yet — brief TLS errors are expected. Monitor with:

kubectl get certificaterequests -A
kubectl describe certificate example-com-tls -n production

Network Policies

⚠ Flannel alone does not enforce NetworkPolicies. k3s embeds the kube-router network policy controller to handle enforcement via iptables — this runs internally, not as a visible pod. If you replaced the default CNI or disabled kube-router, install a policy-capable CNI (Calico, Cilium) before creating NetworkPolicy resources, or they will be silently ignored.

Default behavior: Without any NetworkPolicy, all pods can reach all other pods and the internet ("default allow").

When a NetworkPolicy is applied: It switches the selected pods to "deny all" for the specified direction (Ingress, Egress, or both). Rules then selectively allow traffic. Multiple rules within a policy are OR'd — any matching rule allows the traffic.

Recommended strategy: deny-all-then-allow. Define both Ingress and Egress in policyTypes, then explicitly allow only what's needed. This limits blast radius if a pod is compromised.

Selector types

Combine selectors within a single from/to entry for AND logic; use separate entries for OR logic.

Allow all traffic (escape hatch)

egress:
  - {}   # empty rule = allow everything

For the full NetworkPolicy patterns guide with examples, see references/network-policies.md.

Accessing the Cluster from Outside

Recommendation: Use Ingress for all HTTP/HTTPS traffic. Use WireGuard for developer/admin access to non-HTTP services and internal cluster DNS. Avoid NodePort for anything beyond temporary debugging.

For the WireGuard VPN setup guide, see references/wireguard.md.

Local DNS for Cluster Services

To resolve cluster-internal service names (e.g., my-svc.production.svc.cluster.local) from your local machine via WireGuard, configure per-domain DNS forwarding.

macOS — create /etc/resolver/cluster.local:

domain cluster.local
nameserver 10.43.0.10
timeout 5

sudo killall -HUP mDNSResponder

Ubuntu/Linux — create /etc/systemd/resolved.conf.d/cluster.conf:

[Resolve]
DNS=10.43.0.10
Domains=cluster.local

sudo systemctl restart systemd-resolved

Replace cluster.local with your custom cluster domain if you set --cluster-domain at k3s install time.