This article reviews the development of Istio open source in the past five years and looks forward to the future direction of Istio. The main points of view in this article are as follows:
Next, we start this article with the background of the birth of Istio.
Since 2013, with the explosion of the mobile Internet, enterprises have had higher requirements for the efficiency of application iteration. Application architecture has begun to shift from monolithic to microservices, and DevOps has also become popular. In the same year, with the open source of Docker, the problems of application encapsulation and isolation were solved, making it easier to schedule applications in the orchestration system. In 2014, Kubernetes and Spring Boot were open-sourced, and Spring framework development of microservice applications became popular. In the next few years, a large number of RPC middleware open source projects appeared, such as Google released gRPC 1.0 in 2016. The service framework is in full bloom. In order to save costs, increase development efficiency, and make applications more flexible, more and more enterprises are migrating to the cloud, but this is not just as simple as moving applications to the cloud. In order to use cloud computing more efficiently, a set of “cloud native” methods and concepts are also on the horizon.
Let’s briefly review the major events of Istio open source:
The emergence mentioned here refers to the birth of the concept of “service mesh”. After microservices and containerization, the increase in the use of heterogeneous languages, the surge in the number of services, and the shortened life cycle of containers are the fundamental reasons for the emergence of service meshes.
To make it possible for developers to manage traffic between services with minimal cost, Istio needs to solve three problems:
The above three conditions are indispensable for the Istio service mesh, and we can see from them that these requirements are basically the requirements for the sidecar proxy. The choice of this proxy will directly affect the direction and success of the project.
In order to solve the three problems above, Istio chose:
From the figure below, we can see the transition of the service deployment architecture from Kubernetes to Istio, with many changes and constants.
From Kubernetes to Istio, in a nutshell, the deployment architecture of the application has the following characteristics:
If you are using middleware such as gRPC to develop microservices, after integrating the SDK into the program, the interceptor in the SDK will automatically intercept the traffic for you, as shown in the following figure.
How to make the traffic in the Kubernetes pod go through the proxy? The answer is to inject a proxy into each application pod, share the network space with the application, and then modify the traffic path within the pod so that all traffic in and out of the pod goes through the sidecar. Its architecture is shown in the figure below.
From the figure, we can see that there is a very complex set of iptables traffic hijacking logic. The advantage of using iptables is that it is applicable to any Linux operating system. But this also has some side effects:
How to optimize the service mesh in response to these two problems?
This article will explain these details in the section on performance optimization later on.
Istio is built on top of Kubernetes, which can leverage Kubernetes’ container orchestration and lifecycle management to automatically inject sidecars into pods through admission controllers when Kubernetes creates pods.
In order to solve the resource consumption problem of Sidecar, some people have proposed four deployment modes for the service mesh, as shown in the following figure.
The following table compares these four deployment methods in detail. Each of them has advantages and disadvantages. The specific choice depends on the current situation.
Mode | Memory overhead | Security | Fault domain | Operation and maintenance |
---|---|---|---|---|
Sidecar proxy | The overhead is greatest because a proxy is injected per pod. | Since the sidecar must be deployed with the workload, it is possible for the workload to bypass the sidecar. | Pod-level isolation, if the proxy fails, only the workload in the Pod is affected. | A workload’s sidecar can be upgraded independently without affecting other workloads. |
Node sharing proxy | There is only one proxy on each node, shared by all workloads on that node, with low overhead. | There are security risks in the management of encrypted content and private keys. | Node-level isolation, if a version conflict, configuration conflict, or extension incompatibility occurs when a shared proxy is upgraded, it may affect all workloads on that node. | There is no need to worry about injecting sidecars. |
Service Account / Node Sharing Proxy | All workloads under the service account/identity use a shared proxy with little overhead. | Authentication and security of connections between workloads and proxy cannot be guaranteed. | The level isolation between nodes and service accounts, the fault is the same as “node sharing proxy”. | Same as “node sharing proxy” mode. |
Shared remote proxy with micro-proxy | Because injecting a micro-proxy for each pod, the overhead is relatively large. | The micro-proxy handles mTLS exclusively and is not responsible for L7 routing, which can ensure security. | When a Layer 7 policy needs to be applied, the traffic of the workload instance is redirected to the L7 proxy, and can be bypassed directly if it is not needed. The L7 proxy can run as a shared node proxy, a per-service account proxy, or a remote proxy | Same as “sidecar proxy” mode. |
Zhang Xiaohui of Flomesh explained the evolution of proxy software. I will quote some of his views below to illustrate the key role of programmable proxies in Istio.
The following figure shows the evolution process of the proxy software from configuration to programmable mode, and the representative proxy software in each stage.
The entire proxy evolution process is as the application moves from local and monolithic to large-scale and distributed. Below, I will briefly outline the evolution of the proxy software:
So, is it possible to build a service mesh with proxies? Now I will list the existing service mesh projects, and let’s explore the development law and nature of service mesh together. The table below compares the current popular open source “service mesh” projects.
Contrast | Istio | Linkerd | Consul Connect | Traefik Mesh | Kuma | Open Service Mesh (OSM) |
---|---|---|---|---|---|---|
Current version | 1.14 | 2.11 | 1.12 | 1.4 | 1.5 | 1.0l |
License | Apache License 2.0 | Apache License 2.0 | Mozilla License | Apache License 2.0 | Apache License 2.0 | Apache License 2.0 |
Initiator | Google, IBM, Lyft | Buoyant | HashiCorp | Traefik Labs | Kong | Microsoft |
Service proxy | Envoy, which supports proxyless mode for gRPC | Linkerd2-proxy | Default is Envoy , replaceable | Traefik Proxy | Envoy | Envoy |
Ingress controller | Envoy, custom Ingress, supports Kubernetes Gateway API | no built-in | Envoy, with support for the Kubernetes Gateway API | no built-in | Kong | Support Contour, Nginx, compatible with other |
Governance | Istio Community and Open Usage Commons, proposed to donate to CNCF | CNCF | View Contribution Guidelines | View Contribution Guidelines | CNCF | CNCF |
Comment | It is one of the most popular service mesh projects at present. | The earliest service mesh, the creator of the concept of “Service Mesh”, the first service mesh project to enter CNCF, using a lightweight proxy developed with Rust. | Consul service mesh, using Envoy as a sidecar proxy. | A service mesh project launched by Traefik, using Traefik Proxy as a sidecar and supporting SMI (mentioned below). | A service mesh project launched by Kong that uses Envoy as a sidecar proxy, using Kong’s own gateway as ingress. | An open source service mesh created by Microsoft, using Envoy as a sidecar, compatible with SMI (also proposed by Microsoft). |
The open source projects in the above table are summarized below, and several projects closely related to service mesh are added, as follows:
Looking at the so-called “service mesh” projects mentioned above, we can see that most service mesh project initiators start out as proxies, and then do the control plane. And Istio, Consul Connect, Open Service Mesh, and Kuma all use Envoy as a sidecar proxy. Only Linkerd and Traefik Mesh have launched their own proxies. And all service mesh projects support the sidecar pattern. Apart from Istio, Linkerd, and Consul Connect, which have been used in production, other service mesh projects are currently only “toys”.
After Istio 1.5 had a stable architecture, the community’s main focus was on optimizing Istio’s performance. Below, I will give you a detailed introduction to performance optimization methods in Istio, including:
Proxyless mode is an experimental feature proposed by Istio in version 1.11-a service mesh without sidecar proxy based on gRPC and Istio. Using this pattern allows you to add gRPC services directly to Istio without injecting an Envoy proxy into the Pod. The figure below shows a comparison of sidecar mode and proxyless mode.
As we can see from the above figure, although proxyless mode does not use a proxy for data plane communication, it still needs an agent for initialization and communication with the control plane. First, the agent generates a bootstrap file at startup, in the same way that it generates bootstrap files for Envoy. This tells the gRPC library how to connect to Istiod, where to find certificates for data plane communication, and what metadata to send to the control plane. Next, the agent acts as an xDS proxy, connecting and authenticating with Istiod
. Finally, the agent obtains and rotates the certificate used in the data plane communication.
The essence of a service mesh is not a sidecar model, nor a configuration center, or transparent traffic interception, but a standardized inter-service communication standard.
Some people say that the proxyless model has returned to the old way of developing microservices based on SDK, and the advantages of service meshes have been lost. Can it still be called service mesh? In fact, this is also a compromise on performance—if you mainly use gRPC to develop microservices, you only need to maintain gRPC versions in different languages; that is, you can manage microservices through the control plane.
Envoy xDS has become the de facto standard for communication between cloud-native application services.
In the section on transparent traffic hijacking, we can see the iptables rules and paths that an inter-service traffic passes through when it reaches the destination pod, which needs to go through multiple iptables rules, such as PREROUTING
, ISTIO_INBOUND
, ISTIO_IN_REDIRECT
, OUTPUT
, ISTIO_OUTPUT
, and so on. Suppose now that there is a service A that wants to call service B in another pod on a non-localhost through the network stack, as shown in the figure below.
From the figure, we can see that there are four iptables passes in the whole calling process. Among them, the outbound (iptables2) from Envoy in Pod A and the inbound (iptables3) from eth0 in Pod B are unavoidable. So can the remaining two, iptables1 and iptables4 be optimized? Wouldn’t it be possible to shorten the network path by letting the two sockets communicate directly? This requires programming through eBPF such that:
The transparent traffic interception network path using eBPF mode is shown in the following figure.
If service A and service B to be accessed are on the same node, the network path will be shorter.
Access between services in the same node completely bypasses the TCP/IP stack and becomes direct access between sockets.
We know that modifying the Linux kernel code is difficult, and it takes a long time for new features to be released into the kernel. eBPF is a framework that allows users to load and run custom programs within the operating system’s kernel. That is, with eBPF, you can extend and change the behavior of the kernel without directly modifying the kernel. I will briefly introduce eBPF to you below.
Since the eBPF program can directly monitor and operate the Linux kernel, it has a perspective on the lowest level of the system and can play a role in traffic management, observability, and security.
The open source project Merbridge uses eBPF to shorten the path of transparent traffic hijacking and optimize the performance of the service mesh. For some details on the Merbridge implementation, please refer to the Istio blog .
At first glance, eBPF seems to implement the functions of Istio at a lower level and has a greater tendency to replace sidecar. But eBPF also has many limitations that make it impossible to replace service meshes and sidecars in the foreseeable future. Removing the sidecar in favor of a proxy-per-host model would result in:
Moreover, eBPF is mainly responsible for Layer 3/4 traffic and can run together with CNI, but it is not suitable to use eBPF for Layer 7 traffic.
In the near future, eBPF technology will not be able to replace service meshes and sidecars.
The above two optimizations are carried out for the data plane. Let’s look at the performance optimization of the control plane. You can think of a service mesh as a show, where the control plane is the director and the data plane is all the actors. The director is not involved in the show but directs the actors. If the plot of the show is simple and the duration is very short, then each actor will be allocated very few scenes, and rehearsal will be very easy; if it is a large-scale show, the number of actors is large and the plot is very complicated. To rehearse the show well, one director may not be enough. He can’t direct so many actors, so we need multiple assistant directors (expanding the number of control plane instances); we also need to prepare lines and scripts for actors, if actors It is also possible to perform a series of lines and scenes in one shot (reduce the interruption of the data plane and push updates in batches), so is our rehearsal more efficient?
From the above analogy, you should be able to find the direction of control plane performance optimization, that is:
The easiest and most straightforward way to optimize control plane performance is to reduce the size of the configuration to be pushed. Assuming that there is workload A, if the proxy configuration related to A, that is, the configuration of the service that A needs to access, is pushed to A, instead of pushing the configuration of all services in the mesh to A, it can greatly reduce the number of services to be pushed. The size and application range of the configuration. The Sidecar resource in Istio can help us do this. The following is an example of a sidecar configuration:
apiVersion: networking.istio.io/v1alpha3
kind: Sidecar
metadata:
name: default
namespace: us-west-1
spec:
workloadSelector:
labels:
app: app-a
egress:
- hosts:
- "us-west-1/*"
We can use the workloadSelector
field to limit the scope of workloads that the sidecar configuration applies to, and the egress
field is used to determine the scope of services for the workload, so that the control plane can only push the configuration of its dependent services to service A, greatly reducing the need to push to the data plane. The configuration size of the service mesh reduces the memory and network consumption of the service mesh.
The process of pushing the proxy configuration from the control plane Istiod to the data plane is complex. The following figure shows the process.
After an administrator configures the Istio mesh, the process for pushing proxy configuration in Istiod is as follows:
DiscoveryServer
components will not push the configuration to the data plane immediately, but will add these events to the queue and continue to merge events within a period of time. This process is called debouncing, which is to prevent frequent updates to the data plane configuration.From the above process, we can see that the key to optimizing configuration push is the debounce period in step 2 and the current limit setting in step 4. There are several environmental variables that can help you set up control plane push:
PILOT_DEBOUNCE_AFTER
: The time after which the event will be added to the push queue.PILOT_DEBOUNCE_MAX
: This defines the maximum amount of time an event can debounce.PILOT_ENABLE_EDS_DEBOUNCE
: Specifies whether endpoint updates meet debounce rules or have priority and fall into the push queue immediately.PILOT_PUSH_THROTTLE
: Controls how many push requests are processed at once.Please refer to the Istio documentation for the default values and specific configuration of these environment variables .
How to set these values can follow the following principles:
As for how to set the optimal solution, you need to debug it in conjunction with your observable system.
If setting up debounce batch processing and sidecar cannot optimize the performance of the control plane, the final choice is to expand the scale of the control plane, including expanding the resources of a single Istiod instance and increasing the number of Istiod instances. Which expansion method is used depends on the situation:
Apache SkyWalking can serve as an observability tool for Istio and can also help us analyze the performance of services in dynamic debugging and troubleshooting. Its newly launched Apache SkyWalking Rover component can use eBPF technology to accurately locate Istio’s key performance issues.
On the data plane, we can increase Envoy’s throughput to optimize Istio’s performance by:
For data on the impact of the above optimizations on Envoy throughput, see Using eBPF to pinpoint key performance issues for service meshes.
We know that the service mesh is composed of the data plane and the control plane. From the above list of service mesh open source projects, we can see that most of the service mesh open source projects are based on Envoy, and then develop their own control plane. Remember when I likened a service mesh to a gig earlier in this article? In this service mesh performance, there is no way to think that Envoy is the leading actor. The xDS protocol, invented by Envoy, has basically become a general API for service meshes. Many open source projects for service meshes use Envoy as the data plane. Shown below is the architecture diagram of Envoy.
xDS is the core that differentiates Envoy from other proxies because its code and parsing process are very complex and it is difficult to expand. The following is a topology diagram of Istio components. From the figure, we can see that there is not only one envoy
process but a pilot-agent
.
The role of the pilot-agent
process is as follows:
the
parent process of, responsible for the lifecycle management of Envoy.From the above functions, we can see that the pilot-agent
process is mainly used to interact with Istiod and play a commanding and auxiliary role for Envoy. The core component of Istio is Envoy. So will Envoy “act and guide,” no longer cooperate with Istio, and build its own control plane?
In a Sidecar container, the
pilot-agent
is like Envoy’s “Sidecar”.
pilot-agent
function be built directly into Envoy, thus eliminating the pilot-agent
?
In Kubernetes, the first resource object used to expose services in a cluster, in addition to the service resource object, is Ingress. Using Ingress you simply open an external access point for the cluster, routing traffic to specific services via HTTP Hosts and path
. Compared with exposing services directly to service resources , it can reduce the network access point (PEP) of the cluster and reduce the risk of the cluster being attacked by the network. The following figure shows the process of using Ingress to access services in the cluster.
Before Kubernetes, API Gateway software was widely used as edge routing. When referring to Istio, Istio’s custom Gateway resources are added, which makes accessing resources in the Istio service mesh one more option, as shown in the following figure.
Now, to expose services in a single Istio mesh, how do I choose between NodePort
, LoadBalancer
, Istio Gateway, Kubernetes Ingress, and API Gateway software? If it is a multi-cluster service mesh, how do clients access services within the mesh? Our service mesh lead, Envoy, has done this well and is used in many forms:
These projects utilize Envoy to implement service meshes and API gateways, with a lot of functional overlap, proprietary features, or a lack of community diversity due to the fact that the Envoy community does not provide a control plane implementation. In order to change the status quo, the Envoy community initiated the Envoy Gateway project, which will not change the Envoy core proxy, xDS, and go-control-plane. The project aims to combine the experience of existing Envoy-based API Gateway related projects. Some Envoy-specific extensions to the Kubernetes Gateway API lower the barrier to entry for Envoy users to use gateways. Because the Envoy Gateway still issues configuration to the Envoy proxy through xDS, you can also use it to manage gateways that support xDS, such as the Istio Gateway.
The gateways we have seen now are basically used as ingress gateways in a single cluster and can do nothing for multi-cluster and multi-mesh. To cope with multiple clusters, we need to add another layer of gateways on top of Istio and a global control plane to route traffic between multiple clusters, as shown in the figure below.
Multi-cluster service mesh management is achieved by adding a layer of global control plane and APIs in addition to the Istio control plane. Deploying T1 gateways as a cluster prevents a single point of failure. To learn more about two-tier gateways, refer to Routing Service Mesh Traffic Through Two-Tier Gateway Designs .
The configuration of the T1 gateway is as follows:
apiVersion: gateway.tsb.tetrate.io/v2
kind: Tier1Gateway
metadata:
name: service1-tier1
group: demo-gw-group
organization: demo-org
tenant: demo-tenant
workspace: demo-ws
spec:
workloadSelector:
namespace: t1
labels:
app: gateway-t1
istio: ingressgateway
externalServers:
- name: service1
hostname: servicea.example.com
port: 80
tls: {}
clusters:
- name: cluster1
weight: 75
- name: cluster2
weight: 25
This configuration exposes servicea.example.com
service through the T1 gateway and forwards 75%
of the traffic accessing the service to cluster1 and 25%
of the traffic to cluster2. In addition, in order to deal with the traffic, services, and security configurations in multiple clusters, Tetrate’s flagship product, Tetrate Service Bridge, a series of group APIs have also been added to the TSB documentation for details.
Istio has been open source for more than five years, and many open source projects have emerged in the past two years, among which the more representative ones are:
Their presence makes Istio more intelligent and expands the scope of Istio.
Slime is an Istio-based smart mesh manager open-sourced by the NetEase Shufan microservices team. Slime has been implemented based on Kubernetes Operator and can be used as the CRD manager of Istio. It can define dynamic service governance policies without any customization of Istio, so as to achieve the purpose of automatically and conveniently using the high-level functions of Istio and Envoy.
In the previous control plane performance optimization, we mentioned optimizing the performance of Istio by “reducing the configuration that needs to be pushed”, but Istio cannot automatically identify and cannot rely on the proxy configuration that needs to be pushed to each sidecar to optimize. Slime provides a lazyload
controller, which can help us implement lazy loading of configuration. Users do not need to configure the SidecarScope
manually. Istio can load service configuration and service discovery information on demand.
The following figure shows the flow chart of updating the data plane configuration with Slime as the management plane of Istio.
The specific steps for updating the data plane configuration are as follows:
Because the first invocation of all services in the data plane passes through the Global Proxy, the Proxy can record the invocation and dependency information of all services, and update the configuration of the Sidecar resource in Istio according to the dependency information. When the routing information is redefined, the original record of the Global Proxy becomes invalid, and a new data structure is required to maintain the calling relationship of the service. Slime created a ServiceFence
to maintain the service invocation relationship and solve the problem of missing service information.
Aeraki Mesh is a service mesh project open sourced by Tencent Cloud in March 2021. It expands support for seven-layer protocols based on Istio and focuses on solving the service governance of non-HTTP protocols in Istio. It entered the CNCF sandbox in June 2022.
The following figure shows the architectural diagram of Aeraki.
The process of using Aeraki to serve non-HTTP into an Istio mesh is as follows:
tcp-metaprotocol-dubbo
) through the port command, then generates the MetaProtocol Proxy Filter (compatible with EnvoyFilter) configuration, and at the same time modifies the RDS address to point it to Aeraki.The key to the whole process of accessing non-HTTP services in Istio is the MetaProtocol Proxy. Istio supports HTTP/HTTP2, TCP and gRPC protocols by default, and experimentally supports Mongo, MySQL and Redis protocols. To use Istio to route traffic for other protocols not only requires a lot of work to modify the Istio control plane and extend Envoy, but also a lot of duplication because different protocols share common control logic. The Envoy MetaProtocol Proxy is a general seven-layer protocol proxy implemented based on Envoy. The MetaProtocol Proxy is an extension based on the Envoy code. It implements basic capabilities such as service discovery, load balancing, RDS dynamic routing, traffic mirroring, fault injection, local/global traffic limiting, etc. for the seven-layer protocol, which greatly reduces the difficulty of third-party protocol development for Envoy.
The following figure shows the architecture diagram of MetaProtocol Proxy.
When we want to extend Istio to support other seven-layer protocols such as Kafka, Dubbo, and Thrift, we only need to implement the codec interfaces (Decode and Encode) in the above figure, and then we can quickly develop a third-party protocol plug-in based on MetaProtocol. Because MetaProtocol Proxy is an extension of Envoy, you can still develop filters for it in different languages and use EnvoyFilter
resources to deliver configuration to the data plane.
WasmPlugin is an API introduced in Istio 1.12. As a proxy extension mechanism, we can use it to add custom and third-party Wasm modules to the data plane. The diagram below shows how a user can use the WasmPlugin in Istio.
Specific steps are as follows:
WasmPlugin
configuration and applies it to Istio.WasmPlugin
configuration for the workload in the configuration is selected by the Istio control plane, and the Wasm module is injected into the specified Pod.pilot-agent
in the sidecar loads the Wasm modules from remote or local files and run them in Envoy.Well, having said that, what does this have to do with you? Istio’s relationship with you depends on your role.
The following diagram shows the adoption path for service meshes.
Whether to adopt a service mesh depends on your technology development stage, whether the application implements containerization and microservices, the need for multi-language, whether mTLS is required, and the acceptance of performance loss.
The development of technology is changing with each passing day. In the past two years, some new technologies have appeared, which seem to challenge the status of the service mesh. Some people claim that it can directly replace the existing service mesh of the classic sidecar model. We should not be confused by the noise of the outside world, correcting the positioning of service mesh in the cloud native technology stack.
Blindly promoting a technology and ignoring its applicable scenarios is hooliganism.
The diagram below shows the cloud-native technology stack.
We can see that the “cloud infrastructure”, “middleware”, and “application” layers in the cloud native technology stack diagram all enumerate some iconic open source projects that build standards in their fields:
The diagram below shows how Istio is positioned for seven-tier mesh management in a cloud-native deployment.
Similarities between Istio and Dapr:
Differences between Istio and Dapr:
In the above article, I introduced the development context and open source ecosystem of Istio. Next, I will introduce the future trends of Istio service mesh:
The future of service meshes lies in being the infrastructure for zero-trust networks and hybrid clouds.
This is also the direction of Tetrate, the enterprise-level service mesh provider of the author’s company. We are committed to building an application-aware network suitable for any environment and any load and providing a zero-trust hybrid cloud platform. Shown below is the architecture diagram of Tetrate’s flagship product, Tetrate Service Bridge.
Tetrate was founded by the founders of the Istio project, and TSB is based on open source Istio, Envoy, and Apache SkyWalking. We also actively contributed to the upstream community and participated in the creation of the Envoy Gateway project to simplify the use of Envoy gateways (XCP in the figure above is a gateway built with Envoy).
Zero Trust is an important topic at IstioCon 2022. Istio is becoming an important part of zero trust, the most important of which is identity-oriented control rather than network-oriented control.
Istio 1.14 adds support for SPIRE. SPIRE (SPIFFE Runtime Environment, CNCF Incubation Project) is an implementation of SPIFFE (Secure Production Identity Framework For Everyone, CNCF Incubation Project). In Kubernetes, we use ServiceAccount to provide identity information for workloads in Pods, and its core is based on Token (using Secret resource storage) to represent workload identity. A token is a resource in a Kubernetes cluster. How to unify the identities of multiple clusters and workloads running in non-Kubernetes environments (such as virtual machines)? That’s what SPIFFE is trying to solve.
The purpose of SPIFFE is to establish an open and unified workload identity standard based on the concept of zero trust, which helps to establish a fully identifiable data center network with zero trust. The core of SPIFFE is to define a short-lived encrypted identity document—SVID (SPFFE Verifiable Identity Document)—through a simple API, which is used as an identity document (based on an X.509 certificate or JWT token) for workload authentication. SPIRE can automatically rotate SVID certificates and keys according to administrator-defined policies, dynamically provide workload identities, and Istio can dynamically consume these workload identities through SPIRE.
The Kubernetes-based SPIRE architecture diagram is shown below.
Istio originally used the Citadel service in Istiod to be responsible for certificate management in the service mesh, and issued the certificate to the data plane through the xDS (to be precise, SDS API) protocol. With SPIRE, the work of certificate management is handed over to SPIRE Server. SPIRE also supports the Envoy SDS API. After we enable SPIRE in Istio, the traffic entering the workload pod will be authenticated once after being transparently intercepted into the sidecar. The purpose of authentication is to compare the identity of the workload with the environment information it runs on (node, Pod’s ServiceAccount and Namespace, etc.) to prevent identity forgery. Please refer to How to Integrate SPIRE in Istio to learn how to use SPIRE for authentication in Istio.
We can deploy SPIRE in Kubernetes using the Kubernetes Workload Registrar, which automatically registers the workload in Kubernetes for us and generates an SVID. The registration machine is a Server-Agent architecture, which deploys a SPIRE Agent on each node, and the Agent communicates with the workload through a shared UNIX Domain Socket. The following diagram shows the process of using SPIRE for authentication in Istio.
The steps to using SPIRE for workload authentication in Istio are as follows:
Please refer to the SPIRE documentation for the detailed process of registering and authenticating workloads.
When each workload has an accurate identity, how can the permissions of these identities be restricted? RBAC is used by default in Kubernetes for access control. As the name suggests, this access control is based on roles. Although it is relatively simple to use, there is a role explosion problem for large-scale clusters—that is, there are too many roles, and the types are not static, making it difficult to track and audit role permission models. In addition, the access rights of roles in RBAC are fixed, and there is no provision for short-term use rights, nor does it take into account attributes such as location, time, or equipment. Enterprises using RBAC have difficulty meeting complex access control requirements to meet the regulatory requirements that other organizations demand.
NGAC, or Next Generation Access Control, takes the approach of modeling access decision data as a graph. NGAC enables a systematic, policy-consistent approach to access control, granting or denying user management capabilities with a high level of granularity. NGAC was developed by NIST (National Institute of Standards and Technology) and is currently used for rights management in Tetrate Service Bridge. For more information on why you should choose NGAC over ABAC and RBAC, please refer to the blog post Why you should choose NGAC as your permission control model.
In practical applications, we may deploy multiple Kubernetes clusters in various environments for reasons such as load balancing; isolation of development and production environments; decoupling of data processing and data storage; cross-cloud backup and disaster recovery; and avoiding vendor lock-in. The Kubernetes community provides a “cluster federation” function that can help us create a multi-cluster architecture, such as the common Kubernetes multi-cluster architecture shown in the figure below, in which the host cluster serves as the control plane and has two member clusters, namely West and East.
Cluster federation requires that the networks between the host cluster and member clusters can communicate with each other but does not require network connectivity between member clusters. The host cluster serves as the API entry, and all resource requests from the outside world to the host cluster will be forwarded to the member clusters. The control plane of the cluster federation is deployed in the host cluster, and the “Push Reconciler” in it will propagate the identities, roles, and role bindings in the federation to all member clusters. Cluster federation simply “connects” multiple clusters together, replicating workloads among multiple clusters, and the traffic between member clusters cannot be scheduled, nor can true multi-tenancy be achieved.
Cluster federation is not enough to realize hybrid clouds. In order to realize hybrid clouds in the true sense, it is necessary to achieve interconnection between clusters and realize multi-tenancy at the same time. TSB builds a general control plane for multi-cluster management on top of Istio and then adds a management plane to manage multi-clusters, providing functions such as multi-tenancy, management configuration, and observability. Below is a diagram of the multi-tenancy and API of the Istio management plane.
In order to manage the hybrid cloud, TSB built a management plane based on Istio, created tenant and workspace resources, and applied the gateway group, traffic group, and security group to the workloads in the corresponding cluster through selectors. For the detailed architecture of TSB, please refer to the TSB documentation.
Thanks for reading.
Last updated on Nov 27, 2024