In the evolving landscape of service mesh architectures, understanding the network costs associated with different deployment modes is crucial for optimizing performance and resource efficiency. This blog post will compare the network costs of Istio’s sidecar mode and ambient mode, drawing insights from my recent article, Which Data Plane Should I Use—Sidecar, Ambient, Cilium, or gRPC?

Sidecar Mode

Istio’s sidecar mode intercepts traffic between services by deploying a sidecar proxy alongside each pod. This model introduces additional network hops, which may increase latency and resource usage. However, the mode includes important optimizations like locality-aware load balancing.

Figure 1 illustrates the traffic path when Application 1 accesses Application 2 located in a different availability zone (AZ) under the sidecar model.

Locality Awareness in Sidecar Mode

To better understand locality management, you can use the command istioctl proxy-config endpoint <pod-name[.namespace]> -o yaml to view an endpoint table containing locality information. This helps visualize how locality is managed in sidecar mode.Below is a sample output snippet showing endpoint information for the cluster outbound|9080||reviews.default.svc.cluster.local:

- addedViaApi: true
  circuitBreakers:
    thresholds:
    - maxConnections: 4294967295
      maxPendingRequests: 4294967295
      maxRequests: 4294967295
      maxRetries: 4294967295
    - maxConnections: 1024
      maxPendingRequests: 1024
      maxRequests: 1024
      maxRetries: 3
      priority: HIGH
  edsServiceName: outbound|9080||reviews.default.svc.cluster.local
  hostStatuses:
  - address:
      socketAddress:
        address: 10.244.0.98
        portValue: 9080
    healthStatus:
      edsHealthStatus: HEALTHY
    locality:
      region: us-central1
      zone: us-central1-c
    stats:
    - name: cx_connect_fail
    - name: cx_total
    - name: rq_error
    - name: rq_success
    - name: rq_timeout
    - name: rq_total
    - name: cx_active
      type: GAUGE
    - name: rq_active
      type: GAUGE
    weight: 1
  - address:
    # Additional addresses omitted
  - address:
    # Additional addresses omitted
  name: outbound|9080||reviews.default.svc.cluster.local
  observabilityName: outbound|9080||reviews.default.svc.cluster.local;

This shows how the sidecar proxy in each pod manages fine-grained traffic control, including circuit breaker configurations like maxConnections, maxRequests, and maxRetries, along with traffic metrics and health status indicators. This level of detail aids in maintaining healthy, stable, and latency-efficient traffic at the pod level.

In sidecar mode, each proxy prioritizes routing traffic to services within the same AZ or region, minimizing cross-AZ traffic and reducing latency and associated costs. This approach optimizes network paths, preventing cross-regional bottlenecks.

Although sidecar mode can be compute-intensive, its locality-aware functionality is instrumental in maintaining efficient traffic routing, particularly in multi-region cloud deployments, where it helps minimize cross-region traffic costs.

Ambient Mode

The following diagram shows the architecture of Istio’s ambient mode.

Istio ambient mode has two layers:

1. Ztunnel Layer (L3/L4 Traffic Management): In ambient mode, zTunnel manages traffic mainly at the network and transport layers, reducing overhead while ensuring basic connectivity and security requirements are met.
2. Waypoint Proxy Layer (L7 Traffic Management): This layer, which adds application-layer functionality like advanced routing, observability, and policy enforcement, requires optimal placement to avoid cross-AZ traffic. Waypoint proxies should be deployed within each AZ to ensure maximum performance.

Locality Awareness in Ambient Mode

In contrast to sidecar mode, ambient mode uses zTunnel and waypoint proxies for a different architectural approach. Like sidecar mode, zTunnel prioritizes routing traffic within the same AZ, limiting cross-AZ traffic and associated network costs.

Figure 3 illustrates the traffic path for Application 1 accessing Application 2 in a different AZ in ambient mode.

Note: In the figure, the waypoint proxy is shown separately for demonstration purposes; in practice, it is not bound to a specific node and can be colocated with zTunnel.

To gain further insights into locality in ambient mode, you can use istioctl ztunnel-config workload -o yaml, which provides detailed views of zTunnel configurations and traffic distribution.

Here’s an example snippet:

- applicationTunnel:
    protocol: ""
  canonicalName: productpage
  canonicalRevision: v1
  clusterId: Kubernetes
  hostname: ""
  locality:
    region: us-central1
    zone: us-central1-c
  name: productpage-v1-d5789fdfb-gmw5r
  namespace: default
  node: gke-cilium-default-pool-63a77182-f699
  protocol: HBONE
  serviceAccount: bookinfo-productpage
  status: Healthy
  trustDomain: cluster.local
  uid: Kubernetes//Pod/default/productpage-v1-d5789fdfb-gmw5r
  workloadIps:
    - 10.28.2.14
  workloadName: productpage-v1
  workloadType: deployment

This output demonstrates ztunnel’s locality-aware information, which centralizes traffic management for all pods on a node, including load balancing, health checks, and locality awareness.

Waypoint Proxies

The waypoint proxy, however, does not automatically have AZ awareness. To optimize for cost and performance, waypoint proxies must be deployed across AZs to enable local traffic handling. Otherwise, there is a risk of additional cross-AZ traffic and costs. Additionally, once traffic reaches a waypoint proxy, original locality information may be obscured, complicating route optimization.

For optimal cost and performance, waypoint proxies should be distributed within each AZ to process traffic locally. The proximity-aware communication design between ztunnel and waypoint proxies helps ensure traffic is routed to the nearest waypoint, further reducing cross-AZ costs and latency.

Visualizing with Kiali Dashboard

When comparing sidecar and ambient modes, visualizing locality and routing behavior with Kiali dashboard can help illustrate ambient mode’s complexity. Kiali provides a graphical view of traffic paths in different modes, aiding in understanding ambient mode’s operational intricacies.

Conclusion

In comparing the network costs of Istio’s sidecar and ambient modes, both architectures support locality-aware routing to minimize cross-AZ traffic. Sidecar mode offers more comprehensive locality management at each proxy, while ambient mode requires careful waypoint proxy deployment to avoid extra costs. Additionally, ambient mode’s sub-modes (with or without waypoint proxies) impact network cost and performance in distinct ways. For a deeper dive into the four primary service mesh data plane deployment modes, I encourage you to read the full article here.

This blog was initially published at tetrate.io.