Service Mesh

A Service Mesh is a dedicated infrastructure layer for handling service-to-service communication in distributed microservices. It is typically implemented using lightweight network sidecar proxies (such as Envoy) deployed alongside application containers, managed by a centralized control plane.

When to Use

• Zero-Trust Microservice Security: Enforcing mutual TLS (mTLS) encryption and cryptographic workload identity (SPIFFE) across all pods.
• Fine-Grained Traffic Management: Implementing canary deployments, percentage-based splits, and circuit breakers at the network layer.
• Uniform Observability: Generating standardized golden signal metrics (latency, traffic, errors, saturation) across polyglot microservices.
• Fault Injection & Chaos Engineering: Simulating network latency, connection resets, and 500 errors to test system resilience.

Quick Start

# Canary deployment routing 90% traffic to v1, 10% to v2
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: user-service-route
spec:
  hosts:
    - user-service
  http:
    - route:
        - destination:
            host: user-service
            subset: v1
          weight: 90
        - destination:
            host: user-service
            subset: v2
          weight: 10

Core Concepts

Sidecar Proxy Architecture (Envoy)

A lightweight network proxy intercepts all inbound and outbound pod network traffic transparently:

[ Pod: Billing ]
  [ App Container: port 8080 ] ──localhost──→ [ Envoy Sidecar Proxy ]
                                                       │ (Encrypted mTLS)
                                                       ▼
[ Pod: Orders ]
  [ App Container: port 3000 ] ←──localhost── [ Envoy Sidecar Proxy ]

Mutual TLS (mTLS) & Workload Identity

Authenticates both client and server cryptographically without application-layer code changes:

# Istio PeerAuthentication resource
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: production
spec:
  mtls:
    mode: STRICT # Enforces mTLS across all microservice communication

Canary Traffic Splitting (VirtualService)

Directs traffic dynamically based on weights or request headers:

# Istio VirtualService Canary Routing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: payment-service-routing
spec:
  hosts:
    - payment-svc
  http:
    - route:
        - destination:
            host: payment-svc
            subset: v1
          weight: 90
        - destination:
            host: payment-svc
            subset: v2-canary
          weight: 10

Common Patterns

Zero-Trust mTLS Enforcement

Problem: Cleartext HTTP between internal pods exposes sensitive data to lateral attacks inside a Kubernetes cluster.

Solution: Enforce strict mutual TLS across the service mesh namespace:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod-services
spec:
  mtls:
    mode: STRICT

Best Practices

Do:

• Adopt Ambient / Sidecarless Meshes: Evaluate Istio Ambient or Cilium Service Mesh to reduce sidecar memory and CPU overhead.
• Enforce STRICT mTLS: Ensure plain unencrypted traffic cannot traverse internal Kubernetes node networks.
• Implement Network Egress Policies: Restrict which external internet domains pods are permitted to reach.
• Monitor Envoy Proxy Resource Usage: Size sidecar CPU and memory limits to prevent out-of-memory proxy crashes.

Don't:

• Deploy a service mesh for simple 3-service clusters: Service meshes add substantial operational complexity; adopt only when scale warrants it.
• Duplicate application-layer retries with mesh retries: Cascading retries cause rapid request storms and amplify outages.
• Ignore mesh control plane upgrades: Keep Istio/Linkerd control planes updated to prevent certificate rotation failures.

Troubleshooting

References

• Istio Documentation
• Linkerd Documentation
• Envoy Proxy Architecture