In the early stages of the Istio project, configurations were pushed to Envoy proxies using the global State of the World (SotW) method. Whenever a service changed, the global configuration had to be pushed to all sidecars, causing significant network load and performance loss on the control plane. The Istio community began developing Incremental xDS a few years ago to address this issue and has supported Incremental xDS in recent Istio versions. In the recent Istio 1.22 release, Incremental xDS has become the default feature. This article will introduce you to xDS, Incremental xDS, and the method of configuration distribution in Istio.

What is xDS?

xDS (Extensible Discovery Service) is a communication protocol used for managing service discovery and dynamic configuration in a microservices architecture. This mechanism is widely used in Envoy proxies and Istio service meshes to manage various types of resource configurations, such as routing, service discovery, load balancing settings, etc.

What discovery services are included in xDS?

xDS includes the following main discovery services, each responsible for different types of network resource configurations:

1. LDS (Listener Discovery Service): Manages the configuration of Envoy listeners, which define how to receive and handle inbound connections.
2. RDS (Route Discovery Service): Provides routing information, defining how to route requests to different services based on specified rules.
3. CDS (Cluster Discovery Service): Manages cluster information, where a cluster represents a group of logically similar backend service instances.
4. EDS (Endpoint Discovery Service): Provides the network addresses of specific service instances that make up the clusters defined in CDS.
5. SDS (Secret Discovery Service): Manages security-related configurations, such as TLS certificates and private keys.
6. VHDS (Virtual Host Discovery Service): Provides virtual host configurations for RDS, allowing dynamic updates of virtual hosts without restarting connections.
7. SRDS (Scoped Route Discovery Service): Manages routing scopes, providing dynamic route selection based on different conditions (such as request headers).
8. RTDS (Runtime Discovery Service): Provides runtime configurations, which can be used for experimental features or fine-tuning system behavior.
9. ECDS (Extension Config Discovery Service): Supports dynamic configuration updates for a specific filter. Currently, ECDS is supported for network filters, HTTP filters and Listener filters.

These services collectively support the distribution and update of dynamic configurations, enabling Envoy-based application architectures to adapt in real-time to changes, enhancing scalability and flexibility. Each service can be implemented independently or managed collectively through an aggregated approach (such as ADS). CNCF has also established an xDS API working group to promote the xDS API as the de facto standard for L4/L7 data plane configuration, similar to the role OpenFlow plays in L2/L3/L4 in SDN.

Variants of the xDS Protocol

The xDS protocol primarily includes the following variants:

1. State of the World (SotW): A separate gRPC stream provides complete data for each type of resource, typically used during the initial startup of an Envoy proxy, and was the first type of xDS protocol used by Istio.
2. Incremental xDS (Delta xDS): Provides only the changed parts of the data for each type of resource, developed starting in 2021 and enabled by default in the Istio 1.22 version.
3. Aggregated Discovery Service (ADS): A single gRPC stream aggregates all types of resource data.
4. Incremental ADS (Delta ADS): A single gRPC stream aggregates all types of incremental data.

The table below summarizes the four variants of the xDS protocol, including explanations, usage scenarios, and a comparison of their advantages and disadvantages. These variants provide multiple options for different network environments and service needs, allowing you to choose the most suitable protocol variant to optimize service performance and resource use.

Service meshes using the xDS protocol can more flexibly manage communication and configurations between microservices, reducing the latency of configuration changes and improving system response speed and reliability.

In Istio, the DiscoveryServer acts as the implementation of Envoy’s xDS API, responsible for listening to the gRPC interface and dynamically pushing configurations according to Envoy’s needs. It can handle requests for various resource types and update Envoy configurations in real-time based on service changes. Additionally, it supports security features, such as verifying client certificates, ensuring only legitimate service instances can receive configuration data.

Configuration Examples for xDS Variants

Configuring xDS variants typically involves specifying the xDS server details in the configuration of the Envoy proxy or a similar service mesh. While the configuration details of different service meshes and proxy servers may vary, here are some common YAML configuration examples that illustrate how to specify xDS servers and how to use these protocol variants.

State of the World (SotW)

In Envoy’s configuration, you can use SotW either through static resources or by dynamically obtaining resources via API. Here’s a simple Envoy configuration example showing how to statically define clusters and listeners:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

static_resources:
  listeners:
    - address:
        socket_address: { address: 0.0.0.0, port_value: 80 }
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                stat_prefix: ingress_http
                codec_type: auto
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/" }
                          route: { cluster: local_service }
                    http_filters:
                      - name: envoy.router
                  clusters:
        - name: local_service
            connect_timeout: 0.25s
            type: STATIC
            lb_policy: ROUND_ROBIN
            hosts: [{ socket_address: { address: 127.0.0.1, port_value: 80 } }]

Incremental xDS

Incremental xDS configuration requires the xDS server to support the incremental protocol, and the client configuration must specify the use of Incremental xDS. The Envoy startup configuration needs to add an API version to enable Incremental xDS:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

dynamic_resources:
  cds_config:
    api_config_source:
      api_type: DELTA_GRPC
      grpc_services:
        envoy_grpc:
          cluster_name: xds_cluster
  lds_config:
    api_config_source:
      api_type: DELTA_GRPC
      grpc_services:
        envoy_grpc:
          cluster_name: xds_cluster

Aggregated Discovery Service (ADS)

When using ADS, the configuration of all resource types is aggregated through a single API endpoint. This is specified in the Envoy configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

dynamic_resources:
  cds_config:
    ads: {}
 lds_config:
    ads: {}
  ads_config:
    api_type: GRPC
    grpc_services:
      envoy_grpc:
        cluster_name: xds_cluster

Incremental ADS

Incremental ADS achieves more fine-grained updates by specifying the incremental API type in the ADS configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

dynamic_resources:
  cds_config:
    ads: {}
    resource_api_version: V3
  lds_config:
    ads: {}
    resource_api_version: V3
  ads_config:
    api_type: DELTA_GRPC
    grpc_services:
      envoy_grpc:
        cluster_name: xds_cluster

These configuration examples need to be adjusted according to your specific environment and requirements. For more details and advanced configurations, you can refer to the Envoy documentation.

How Does Istio Send Configurations to Envoy Sidecars?

Thanks to the xDS protocol, tools like Istio and Envoy Gateway can dynamically distribute configurations to Envoy proxies via API. The diagram below shows the configuration distribution process in Istio (Sidecar mode).

The main steps of the configuration distribution process in Istio are:

1. Declarative Configuration: Users define the service mesh’s configuration using YAML files or other configuration management tools. These configurations can include routing rules, security policies, telemetry settings, etc.
2. Kubernetes: Istio configuration files are submitted to the Kubernetes cluster, usually through the kubectl apply command or other CI/CD tools. Kubernetes receives the configuration files and stores them in the etcd database.
3. Istiod: Istiod is the control plane component of Istio, responsible for managing and distributing configurations. It listens for events coming from the Kubernetes API server, obtains relevant configuration changes, and processes them. Istiod parses the configuration files, generates corresponding routing rules and policies, and distributes these configurations to the data plane (Envoy proxies) via the xDS API.
4. xDS API: Istiod uses the xDS API to send configurations to each Envoy proxy.
5. Envoy Proxy: Envoy is the data plane component of Istio, running in a sidecar container alongside each service, intercepting and managing all inbound and outbound traffic. Envoy proxies receive configurations from Istiod via the xDS API and manage traffic, enforce policies, and collect telemetry data based on these configurations.
6. Pod: Each service instance runs in a Pod, which contains an application container and an Envoy proxy container. The Envoy proxy intercepts all network traffic to and from the application container and processes it according to the configurations.

This configuration distribution process ensures that Istio can dynamically manage and configure all service instances in the service mesh, providing consistent traffic management and policy enforcement.

The Evolution of xDS and the Implementation of Delta xDS in Istio

Initially, xDS adopted a “global state” (State of the World, abbreviated as SotW) design, which meant that any configuration change required sending the complete state of all configurations to Envoy. This approach created a huge burden on the network and control plane, especially in large-scale service deployments.

At EnvoyCon 2021, Aditya Prerepa and John Howard shared How Istio Implements Delta xDS, an incremental implementation of xDS. Compared to the traditional SotW xDS, Delta xDS sends only the changed configurations, significantly reducing the amount of configuration data that needs to be sent over the network, thus improving efficiency and performance. This method is particularly suitable for environments where configurations change frequently, as it updates only the changed parts rather than the entire configuration.

During the implementation of Delta xDS, the Istio

team faced several challenges, including ensuring the correctness of configuration updates and avoiding potential resource leaks. They addressed these challenges by adopting a Dry-run mode to run the SotW and Delta generators in parallel, gradually identifying and fixing flaws in the implementation. Additionally, they introduced new Envoy types, such as the Virtual Host Discovery Service, to support more fine-grained configuration distribution.

Delta xDS Incremental Configuration

The diagram below illustrates the process of Delta xDS incremental configuration.

The Delta xDS configuration process is as follows:

1. Initial Complete Configuration: The control plane sends the initial complete configuration to the proxy, using the StoW mode at this time.
2. Subscribe to Configuration Changes: The proxy subscribes to configuration changes from the control plane.
3. Check Configuration Changes: The control plane checks for configuration changes relative to the proxy’s known state.
4. Calculate Differences: The control plane calculates the differences (increments) between the current configuration and the previous configuration held by the proxy.
5. Send Only Differences: The control plane sends only the changed configuration (differences) to the proxy, which applies these incremental updates to its configuration.

This process ensures that only necessary changes are transmitted and applied, improving efficiency and reducing the load on network and proxy resources.

SotW vs Delta xDS

While Delta xDS solves the performance issues of configuration distribution in large-scale networks, the SotW mode still has its place, such as in the initial configuration delivery. The table below compares the two configuration distribution methods in Istio: SotW (State of the World) and Delta xDS.

This table compares SotW and Delta XDS from multiple perspectives, including data volume, efficiency, complexity, resource usage, timeliness, and applicability, helping you make the appropriate choice in different usage environments.

Conclusion

In this article, I shared the components of xDS and the process of configuration distribution in Istio, as well as the two modes of xDS, SotW and Delta xDS. With Delta xDS becoming the default configuration in Istio version 1.22, this will help users easily use Istio in large-scale network environments.

References

• xDS REST and gRPC protocol - envoyproxy.io

This blog was initially published at tetrate.io.