In March 2024, during KubeCon EU, the Cloud Native Computing Foundation (CNCF) released its first detailed whitepaper on Cloud Native Artificial Intelligence (CNAI). This report thoroughly explores the current state, challenges, and future directions of integrating Cloud Native technologies with artificial intelligence. This article delves into the core content of this whitepaper.

What is Cloud Native AI?

Cloud Native AI refers to the approach of building and deploying artificial intelligence applications and workloads using Cloud Native technology principles. This includes leveraging microservices, containerization, declarative APIs, and continuous integration/continuous deployment (CI/CD) to enhance the scalability, reusability, and operability of AI applications.

The diagram below illustrates the architecture of Cloud Native AI, redrawn based on the whitepaper.

Relationship Between Cloud Native AI and Cloud Native Technologies

Cloud Native technologies provide a flexible, scalable platform that makes the development and operation of AI applications more efficient. Through containerization and microservices architecture, developers can iterate and deploy AI models rapidly while ensuring high availability and scalability of systems. Kubernetes and other Cloud Native tools provide essential support such as resource scheduling, automatic scaling, and service discovery.

The whitepaper provides two examples illustrating the relationship between Cloud Native AI and Cloud Native technologies, namely running AI on Cloud Native infrastructure:

• Hugging Face Collaborates with Microsoft to launch Hugging Face Model Catalog on Azure
• OpenAI Scaling Kubernetes to 7,500 nodes

Challenges of Cloud Native AI

Despite providing a solid foundation for AI applications, Cloud Native technologies still face challenges when integrating AI workloads with Cloud Native platforms. These challenges include the complexity of data preparation, resource requirements for model training, and maintaining the security and isolation of models in multi-tenant environments. Additionally, resource management and scheduling in Cloud Native environments are crucial, especially for large-scale AI applications, and further optimization is needed to support efficient model training and inference.

Development Path of Cloud Native AI

The whitepaper proposes several development paths for Cloud Native AI, including improving resource scheduling algorithms to better support AI workloads, developing new service mesh technologies to enhance the performance and security of AI applications, and driving innovation and standardization of Cloud Native AI technology through open-source projects and community collaboration.

Cloud Native AI Technology Landscape

Cloud Native AI involves a variety of technologies, from containers and microservices to service meshes and serverless computing. Kubernetes is a key platform for deploying and managing AI applications, while service mesh technologies like Istio and Envoy provide powerful traffic management and security features. Additionally, monitoring tools like Prometheus and Grafana are essential for maintaining the performance and reliability of AI applications.

Below is the Cloud Native AI landscape provided in the whitepaper.

General Orchestration

• Kubernetes
• Volcano
• Armada
• Kuberay
• Nvidia NeMo
• Yunikorn
• Kueue
• Flame

Distributed Training

• Kubeflow Training Operator
• Pytorch DDP
• TensorFlow Distributed
• Open MPI
• DeepSpeed
• Megatron
• Horovod
• Apla
• …

ML Serving

• Kserve
• Seldon
• VLLM
• TGT
• Skypilot
• …

CI/CD - Delivery

• Kubeflow Pipelines
• Mlflow
• TFX
• BentoML
• MLRun
• …

Data Science

• Jupyter
• Kubeflow Notebooks
• PyTorch
• TensorFlow
• Apache Zeppelin
• …

Workload Observability

• Prometheus
• Influxdb
• Grafana
• Weights and Biases (wandb)
• OpenTelemetry
• …

AutoML

• Hyperopt
• Optuna
• Kubeflow Katib
• NNI
• …

Governance & Policy

• Kyverno
• Kyverno-JSON
• OPA/Gatekeeper
• StackRox Minder
• …

Data Architecture

• ClickHouse
• Apache Pinot
• Apache Druid
• Cassandra
• ScyllaDB
• Hadoop HDFS
• Apache HBase
• Presto
• Trino
• Apache Spark
• Apache Flink
• Kafka
• Pulsar
• Fluid
• Memcached
• Redis
• Alluxio
• Apache Superset
• …

Vector Databases

• Milvus
• Chroma
• Weaviate
• Quadrant
• Pinecone
• Extensions
  • Redis
  • Postgres SQL
  • ElasticSearch
• …

Model/LLM Observability

• Trulens
• Langfuse
• Deepchecks
• OpenLLMetry
• …

Summary

Finally, let me summarizes the following key points:

• Role of the Open Source Community: The whitepaper clearly points out the role of the open-source community in advancing Cloud Native AI, including accelerating innovation and reducing costs through open-source projects and extensive collaboration.

• Importance of Cloud Native Technologies: Cloud Native AI is built and deployed according to Cloud Native principles, highlighting the importance of repeatability and scalability. Cloud Native technologies provide an efficient development and runtime environment for AI applications, especially in terms of resource scheduling and service scalability.

• Challenges Exist: Despite the many advantages brought by Cloud Native AI, there are still challenges in data preparation, model training resource requirements, and model security and isolation.

• Future Development Directions: The whitepaper proposes development paths including optimizing resource scheduling algorithms to support AI workloads, developing new service mesh technologies to enhance performance and security, and leveraging open-source projects and community collaboration to further promote technological innovation and standardization.

• Key Technological Components: Key technologies involved in Cloud Native AI include containers, microservices, service meshes, and serverless computing. Kubernetes plays a central role in deploying and managing AI applications, while service mesh technologies such as Istio and Envoy provide necessary traffic management and security.

For more details, please download the Cloud Native AI Whitepaper.