From Kubernetes and GPUs to MCP and agent runtimes, rethinking infrastructure and the open-source ecosystem for the AI era.
While following the progress of the Agentic AI Foundation (AAIF), I noticed something interesting: many friends I have known from the cloud native community over the past decade are moving into AI.
Some former colleagues have joined OpenAI; others are building AI gateways, MCP servers, and agent frameworks. Developers who used to work on Kubernetes, DevOps, and platform engineering are shifting toward GPUs, model inference, and agent runtimes.
I have gone through the same transition myself. From Kubernetes, CNCF, and the Chinese cloud native community, to five years at Tetrate, and now a focus on HAMi, GPU resource governance, and AI native infrastructure, I have kept asking the same question: how do we make complex systems run more efficiently and reliably?
But faced with the endless stream of new concepts like AI Gateway, MCP Gateway, Agent Runtime, and Sandbox, I keep coming back to one question:
Do the cloud native foundations we spent more than a decade building really need to be reinvented for the AI era?
AI Native Is Not a Clean-Slate Rewrite of Cloud Native
Looking back at how cloud native evolved, Docker, Kubernetes, and the open-source ecosystem around CNCF solved a long list of problems: application delivery, resource orchestration, elastic scaling, failure recovery, security isolation, and observability.
None of these capabilities become obsolete in the AI era.
Model inference services still need scheduling and autoscaling; AI platforms still need identity, security, and multi-tenant management; agent applications still need reliability and observability.
What actually changes is the characteristics of the workloads.
Traditional web services care mostly about request throughput and latency, while LLM inference also has to account for GPU memory, KV cache, time to first token (TTFT), and inter-token latency (ITL). Traditional applications follow relatively deterministic execution paths, while agents can plan tasks dynamically, call tools, and even generate code.
AI native needs new infrastructure abstractions, but that does not mean a decade of engineering experience should be thrown away.
In my view, AI native is not a replacement for cloud native. It is an infrastructure refactor centered on AI workloads.
Kubernetes and GPUs: Extend the Old Architecture, or Start Over?
Whether Kubernetes fits AI workloads has long been debated.
Supporters point to its mature declarative APIs, controllers, and resource orchestration; critics point out that it was never designed for GPUs, distributed training, or large-scale inference.
I think the real question is not whether to use Kubernetes, but which capabilities can be reused and which abstractions need to change.
Take HAMi, a project I am currently involved in. Traditional Kubernetes GPU management mostly allocates whole cards, which easily wastes resources for small inference and development tasks.
HAMi extends Kubernetes device management and scheduling to support GPU sharing, resource allocation, and isolation, rather than reimplementing a container orchestration system.
But GPU sharing is not only about utilization. You also have to distinguish resource isolation, fault isolation, and security isolation. A working memory quota does not guarantee that one failing task will not affect the others.
That is exactly the new challenge AI workloads pose to traditional resource management.
And higher GPU utilization alone does not mean lower business cost. If overcommitment drives up inference latency, fails agent tasks, or forces repeated retries, the final bill can be even higher.
The optimization target for AI infrastructure should move beyond GPU utilization toward cost, reliability, and service quality per successful task.
From Tetrate to AI Gateways: Familiar Technology Enters a New Arena
In September 2026, Agent Router, driven by Tetrate, joined AAIF. Before that, agentgateway from Solo.io had also joined the foundation.
Seeing this news hit home for me. Cloud native infrastructure companies I knew well are meeting again in the AI native space.
That is no surprise.
Model routing, load balancing, authentication, rate limiting, security policy, and observability have always been problems that API gateways and cloud native infrastructure are good at solving.
But agents bring new challenges.
A model call can be billed by token; an MCP tool call may involve user authorization; an agent may dynamically choose different tools mid-flight and even perform side-effecting operations.
So an AI gateway should reuse proven traffic-handling machinery like Envoy, but it must also understand the new semantics of models, tools, and agents.
The question is: does an enterprise need one unified AI gateway, or separate model gateways, MCP gateways, and agent collaboration gateways?
That is not a product naming question. It is a question of where infrastructure responsibilities should be drawn.
MCP and Agent Runtimes: Which Wheels Actually Need Redesigning?
The evolution of MCP reminds me of the path cloud native infrastructure once took.
MCP lowers the cost of connecting AI applications to tools and data sources through a unified protocol. But once MCP servers enter production, problems such as scalability, authorization, load balancing, failure recovery, and auditing appear.
MCP moving toward a stateless protocol core in 2026 proves the point: the protocol can depend less on connection sessions, but business state, authorization context, and task lifecycle still need explicit management.
These are all familiar distributed systems problems.
Agent runtimes, however, do need some genuinely new abstractions.
A traditional Kubernetes Pod can host an agent process, but one agent task may span multiple model calls, execution environments, and tools, and may need to run for a long time, pause, and resume.
The lifecycle of a Pod is not the lifecycle of an agent task.
Likewise, containers and sandboxes provide execution isolation, but they do not by themselves solve delegated permissions, tool authorization, or approval for sensitive operations.
So my view is:
An agent runtime does not need to reinvent the container, but it does need a new task, state, and security model built on top of existing infrastructure.
At the same time, AI native security design cannot focus only on prompt injection. It also needs explicit resource isolation, fault isolation, and security isolation, so that even when an agent makes a mistake, the blast radius stays bounded.
Will AAIF Become the CNCF of the AI Native Era?
Beyond the technology, the open-source ecosystem deserves even more attention.
Over the past decade, the value of CNCF was not just hosting Kubernetes. It pushed vendors to collaborate around open standards and gradually built a complete cloud native ecosystem.
Today, AAIF is trying to build a similar foundation for agentic AI. Projects around MCP, A2A (Agent2Agent), AI gateways, and developer tooling are exploring protocol interoperability and neutral governance.
But I do not think we can conclude yet whether AAIF will become the next CNCF.
It has to answer a deeper question first: with CNCF, LF AI & Data, and other foundations already in place, why does agentic AI need a new foundation? And how do we avoid duplicating work across foundations, protocols, and projects?
Membership and project counts are not sufficient proof of success. What matters is cross-vendor interoperability, production adoption, open governance, and whether developers can keep participating in technical decisions.
As AI makes code generation easier and easier, one shift is worth noting:
The scarcest value in open-source communities may be shifting from code itself to consensus, trust, and long-term collaboration.
That is why I am paying close attention to AAIF.
From Cloud Native to AI Native Is Also a Generational Shift for Developers
Over the past decade, I have met many developers through CNCF, the Chinese cloud native community, and Tetrate.
Some have joined AI companies like OpenAI; some are building agents, MCP, and AI gateways; and many former Kubernetes, platform engineering, and infrastructure developers are now working on inference services, GPUs, and AI platforms.
This is not just people chasing the next hot thing.
What matters more is that AI is generating a large number of engineering problems that look a lot like cloud native’s, yet with new characteristics.
People who know distributed systems need to learn how models and agents actually run, while AI developers need to learn reliability, security isolation, resource management, and production operations.
The most competitive engineers of the next era will probably not be those who know only cloud native or only AI, but those who understand where the two meet and can combine them.
I also hope this cross-domain exchange happens in open-source communities.
Summary
Back to the original question: do we really need to reinvent the wheel?
We do not need to reinvent infrastructure that has already been proven, but we must redefine the new abstractions that AI workloads demand.
Kubernetes does not need to be abandoned, but GPU and inference scheduling need to evolve. Gateway technology like Envoy is worth reusing, but MCP and agents need new governance. Containers and sandboxes still matter, but agent runtimes need their own task lifecycle and permission model. Traditional observability remains the foundation, but AI platforms need to track task success rate, end-to-end cost, and service quality.
AI native is neither a simple upgrade of cloud native nor a wholesale rebuild.
Ten years ago we built the cloud native ecosystem around Kubernetes and CNCF; ten years later we are debating a new technical architecture around AI native.
But I believe what actually moves technology forward is still open standards, engineering practice, and sustained collaboration among developers.
Do you see AI native as the natural evolution of cloud native, or a new paradigm that requires rebuilding infrastructure?
If you are building agents, MCP, AI gateways, model inference, or GPU platforms, I would love to hear about the problems you are hitting, especially the ones existing cloud native technology does not solve well.
References
- HAMi - GitHub
- Agentic AI Foundation - aaif.io
- Model Context Protocol - modelcontextprotocol.io
- Agent Router - GitHub
- agentgateway - GitHub
These views are my own.
