In the previous blog, I discussed how certificates are managed in Istio. This article will guide you on how to use an external CA by integrating SPIRE and cert-manager to achieve fine-grained certificate management and automatic certificate rotation.

If you are not familiar with what SPIRE is and why we use SPIRE, I recommend you read the following:

• Why does Istio use SPIRE for authentication?
• How to integrate SPIRE in Istio?

Overview of Certificate Issuance and Management Process

The diagram below shows the certificate trust chain used in this article based on cert-manager and SPIRE:

From the diagram, you can see:

• cert-manager acts as the Root CA, issuing certificates to istiod and SPIRE. We use a self-signed Issuer, but you can also configure it to use built-in Issuers like Let’s Encrypt, Vault, Venafi, or other external Issuers. Additionally, you can choose other UpstreamAuthorities such as Vault or SPIRE federations;
• SPIRE issues SVID certificates to Istio mesh workloads and Ingress Gateway, Egress Gateway for inter-service mTLS;
• Certificates used by the mesh-external access to the Ingress Gateway and those used by the Egress Gateway to access external services of the mesh need additional configuration;

The diagram below shows the certificate issuance and renewal process after integrating SPIRE and cert-manager in Istio.

1. The SPIRE Controller Manager in the SPIRE Server automatically registers workloads in Kubernetes, generating SPIFFE-standard identities for all workloads;
2. cert-manager issues and manages CA certificates for istiod;
3. The Envoy proxy in the workload sends a CSR request to the SPIRE Agent on the same node via a UNIX Domain Socket (UDS) through the SDS API;
4. The SPIRE Agent sends the CSR to the SPIRE Server;
5. The SPIRE Server returns the signed certificate to the SPIRE Agent;
6. The SPIRE Agent returns the signed certificate to the workload;
7. SPIRE manages and updates the certificates for the workloads;

After understanding the general process, we will install the components in sequence. Versions of the components are as follows:

• cert-manager v1.15.1
• SPIRE v1.2.0
• Istio v1.21.1

Installing cert-manager

Run the following command to install cert-manager, which we will use to achieve automatic certificate rotation:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.15.1/cert-manager.yaml

The Root CA uses a self-signed certificate, run the following command to configure the Root CA:

cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: cert-manager
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-ca
  namespace: cert-manager
spec:
  isCA: true
  duration: 21600h
  secretName: selfsigned-ca
  commonName: certmanager-ca
  subject:
    organizations:
      - cert-manager
  issuerRef:
    name: selfsigned
    kind: Issuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-ca
spec:
  ca:
    secretName: selfsigned-ca
EOF

Then configure a certificate for istiod:

kubectl create namespace istio-system
cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cacerts
  namespace: istio-system
spec:
  secretName: cacerts
  duration: 1440h
  renewBefore: 360h
  commonName: istiod.istio-system.svc
  isCA: true
  usages:
    - digital signature
    - key encipherment
    - cert sign
  dnsNames:
    - istiod.istio-system.svc
  issuerRef:
    name: selfsigned-ca
    kind: ClusterIssuer
    group: cert-manager.io
EOF

Now that we have installed cert-manager and created a clusterIssuer named selfsigned-ca, next, we will install SPIRE and set cert-manager as SPIRE’s UpstreamAuthority.

Installing SPIRE

Run the following command to quickly install SPIRE:

kubectl apply -f https://jimmysong.io/blog/cert-manager-spire-istio/manifests/spire-with-cert-manager-upstream-authority-quick-start.yaml

This YAML file includes adaptations for cert-manager compared to the samples/security/spire/spire-quickstart.yaml file in the Istio installation package, such as:

• Adding permissions for the cert-manager.io API group to the spire-server-cluster-role ClusterRole;
• Adding the UpstreamAuthority "cert-manager" configuration in the SPIRE Server settings;

This command installs the SPIRE Controller Manager, which automatically registers workloads in Kubernetes. All workloads are registered with SPIFFE standard service identity format spiffe://<trust-domain>/ns/<namespace>/sa/<service-account> based on their service accounts.

If you want to adjust the TTL of SPIRE CA and SVID certificates, you can modify ca_ttl (default 24h) and default_svid_ttl (default 1h) in the SPIRE Server configuration, detailed in SPIRE Server Configuration.

Installing Istio

The Envoy Agent in Pod shares the Unix Domain Socket of the local SPIRE Agent and call the SPIRE Server through the Workload API to obtain the certificate, as shown in the following figure.

Run the following command to install Istio and enable automatic CA certificate rotation:

istioctl install --skip-confirmation -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
spec:
  profile: default
  meshConfig:
    # Trust domain should be the same as configured in the SPIRE Server
    trustDomain: example.org
  values:
    global:
    # Custom sidecar template
    sidecarInjectorWebhook:
      templates:
        spire: |
          spec:
            containers:
            - name: istio-proxy
              volumeMounts:
              - name: workload-socket
                mountPath: /run/secrets/workload-spiffe-uds
                readOnly: true
            volumes:
              - name: workload-socket
                csi:
                  driver: "csi.spiffe.io"
                  readOnly: true          
  components:
    pilot:
      k8s:
        env:
          # If enabled, if users introduce a new intermediate plugin CA, users do not need to restart istiod to obtain certificates. Istiod will retrieve the newly added intermediate plugin CA's certificates and update them accordingly. Inserting a new Root-CA is not supported.
          - name: AUTO_RELOAD_PLUGIN_CERTS
            value: "true" 
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        label:
          istio: ingressgateway
        k8s:
          overlays:
            - apiVersion: apps/v1
              kind: Deployment
              name: istio-ingressgateway
              patches:
                - path: spec.template.spec.volumes.[name:workload-socket]
                  value:
                    name: workload

-socket
                    csi:
                      driver: "csi.spiffe.io"
                      readOnly: true
                - path: spec.template.spec.containers.[name:istio-proxy].volumeMounts.[name:workload-socket]
                  value:
                    name: workload-socket
                    mountPath: "/run/secrets/workload-spiffe-uds"
                    readOnly: true
EOF

Since we are using the spire template declared in the Istio Operator to deploy workloads, we run the following command to deploy the Bookinfo application:

istioctl kube-inject -f bookinfo-with-spire-template.yaml | kubectl apply -n default -f -

Note: The bookinfo-with-spire-template.yaml file used in the above command can be found here, and the only difference from the samples/bookinfo/platform/kube/bookinfo.yaml file in the Istio installation package is that each Deployment’s template has the following annotation added:

annotations:
  inject.istio.io/templates: "sidecar,spire"

Verifying SPIRE

We will verify that SPIRE is effective by checking the identity and certificate configuration of the productpage service.

Use the following command to check whether SPIRE has issued identity proofs to the workloads:

kubectl exec -i -t spire-server-0 -n spire -c spire-server -- /bin/sh -c "bin/spire-server entry show -socketPath /run/spire/sockets/server.sock -spiffeID spiffe://example.org/ns/default/sa/bookinfo-productpage"

You can see the identity information of the productpage service in the output result:

Found 1 entry
Entry ID         : 69fbf896-a296-4c3c-8179-44bf4e49e474
SPIFFE ID        : spiffe://example.org/ns/default/sa/bookinfo-productpage
Parent ID        : spiffe://example.org/k8s-workload-registrar/demo-cluster/node/gke-cluster-1-default-pool-18d66649-z1lm
Revision         : 1
TTL              : default
Selector         : k8s:node-name:gke-cluster-1-default-pool-18d66649-z1lm
Selector         : k8s:ns:default
Selector         : k8s:pod-uid:73347537-a3e5-4e43-b8c5-bd315c7385b7
DNS name         : productpage-v1-7f444fc4dd-rq47m
DNS name         : productpage.default.svc

View the productpage pod’s certificate trust chain:

istioctl -n default proxy-config secret deployment/productpage-v1 -o json | jq -r \
'.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode > chain.pem

View the root certificate:

istioctl -n default proxy-config secret deployment/productpage-v1 -o json | jq -r \
'.dynamicActiveSecrets[1].secret.validationContext.trustedCa.inlineBytes' | base64 --decode > root.pem

The chain.pem file is the certificate trust chain, containing two certificates. Save them to two files:

split -p "-----BEGIN CERTIFICATE-----" chain.pem cert-

Then use OpenSSL to view all certificates:

openssl x509 -noout -text -in cert-aa
openssl x509 -noout -text -in cert-ab
openssl x509 -noout -text -in root.pem

You will see the following certificate trust chain root.pem -> cert-aa -> cert-ab, as shown in the diagram:

View the certificate of Istiod:

istioctl -n istio-system proxy-config secret deployment/istiod -o json | jq -r \
'.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode > chain.pem

From the certificate trust chain, we can see:

• cert-manager acts as the PKI root node, issuing certificates to istiod;
• SPIRE acts as an intermediate CA, issuing certificates to various workloads;
• The X509 v3 subject alternative names of workloads in the Istio service mesh follow the SPIFF identity specification;
• The identity and certificates of all workloads in the Istio service mesh are managed by SPIRE;

Setting Automatic Certificate Rotation

If you want to modify the rotation period of istiod certificates from 60 days (1440 hours) to 30 days (720 hours), run the following command:

cat << EOF | kubectl apply -f -
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cacerts
  namespace: istio-system
spec:
  secretName: cacerts
  duration: 720h 
  renewBefore: 360h
  commonName: istiod.istio-system.svc
  isCA: true
  usages:
    - digital signature
    - key encipherment
    - cert sign
  dnsNames:
    - istiod.istio-system.svc
  issuerRef:
    name: selfsigned-ca
    kind: ClusterIssuer
    group: cert-manager.io
EOF

Run the following command to view the istiod logs:

kubectl logs -l app=istiod -n istio-system -f

After two minutes, you will see certificate change records similar to the following:

2022-12-23T03:48:42.697360Z	info	Update Istiod cacerts
2022-12-23T03:48:42.697503Z	info	Using kubernetes.io/tls secret type for signing ca files
2022-12-23T03:48:42.778241Z	info	Istiod has detected the newly added intermediate CA and updated its key and certs accordingly
2022-12-23T03:48:42.779459Z	info	x509 cert - Issuer: "CN=istiod.istio-system.svc", Subject: "", SN: d7acac2301045f741e5e30cff380deaf, NotBefore: "2022-12-23T03:46:42Z", NotAfter: "2032-12-20T03:48:42Z"
2022-12-23T03:48:42.779561Z	info	x509 cert - Issuer: "CN=certmanager-ca,O=cert-manager", Subject: "CN=istiod.istio-system.svc", SN: 164bf045670a1716ed3f0f1c89b56122, NotBefore: "2022-12-23T03:48:14Z", NotAfter: "2023-01-22T03:48:14Z"
2022-12-23T03:48:42.779642Z	info	x509 cert - Issuer: "CN=certmanager-ca,O=cert-manager", Subject: "CN=certmanager-ca,O=cert-manager", SN: 8533dbfe0b84ed1fc4e3c76be7ef612f, NotBefore: "2022-12-20T07:50:12Z", NotAfter: "2025-06-07T07:50:12Z"
2022-12-23T03:48:42.779657Z	info	Istiod certificates are reloaded

To modify the automatic rotation period for workload certificates, you can set the environment variable SECRET_TTL of the pilot-agent command, which defaults to 24h0m0s.

Summary

In this article, we used cert-manager as the PKI, integrated SPIRE into our certificate chain of trust, and created identities and certificates for workloads in the Istio mesh. Using cert-manager means you don’t have to worry about istiod certificate expiration, and you can renew certificates as needed. You can also integrate cert-manager with certificate providers, such as Let’s Encrypt, HashiCorp Vault, Venafi, etc. You can also use istio-csr to let the cert-manager manage certificates in Istio directly. Or use Vault to store certificates.

This blog was originally published at tetrate.io.