Jimmy Song – Blog

After HAMi Became a CNCF Incubating Project: Open Source Is Moving from Code to Consensus

Jimmy Song — Wed, 08 Jul 2026 13:51:44 +0000

After HAMi became a CNCF Incubating project, I want to talk about something overlooked: AI is shifting the scarce resource of open source communities from code to consensus.

On July 2, 2026, HAMi officially became a CNCF Incubating project (see the announcement). For an open source project, this means more than recognition of its technical capability; it means that community governance, ecosystem building, and real-world adoption have all entered a new phase.

But if you only read HAMi’s growth as “a GPU virtualization project succeeded,” you might miss the more important shift.

My time building the HAMi community has left me with one increasingly strong feeling: AI is changing how open source communities produce. As AI coding drives the cost of producing code lower and lower, the core of competition in open source will no longer be who wrote the most code, but who can build stronger technical consensus, attract more contributors, and form a sustainable ecosystem network.

This article is my attempt to lay out that argument clearly.

Figure 9: Congratulations to HAMi for becoming a CNCF Incubating project

HAMi’s Growth Shows That a Project’s Real Asset Is Its Community

Let’s start with the data. Here is where HAMi stands today:

Metric	Value
GitHub Stars	3,700+
Contributors	Nearly 500, from 27 countries
Participating organizations	Multiple, and growing
Release cadence	Once every three months

Table 3: HAMi community key metrics (as of July 2026)

Figure 10: HAMi community contributors map, contributors from 27 countries

I’m not listing these numbers to show off “growth metrics.” I’m making a different point: an open source project is shifting from “software maintained by a team” into “a technical community of people gathered around a shared goal.”

That distinction matters. Software can be forked, rewritten, or generated overnight by AI. But a community with a shared goal, trust, and rhythm cannot be forked. That is the irreplaceable asset of an open source project.

From a governance-maturity perspective, HAMi has passed through three milestones:

Figure 11: Evolution of HAMi’s governance maturity

Note the middle stretch: from entering Sandbox in August 2024 to reaching Incubating in July 2026, roughly two years. During those two years the code certainly grew, but what actually convinced the CNCF Technical Oversight Committee (TOC) was the diversification of the community, the formalization of governance, and real production adoption. None of that is something you produce by writing code.

When HAMi Open-Sourced in 2021, There Was No AI Coding

HAMi was first open-sourced in 2021. Back then, most developers did not see AI coding the way we do today. Whether an open source project survived depended on developers genuinely investing their time, discussing problems in issues, submitting code through pull requests, and building trust through code review.

Today, the environment has changed.

In a recent HAMi community livestream (Mastering HAMi DRA, Yang Shouren, HAMi Community Livestream Episode 2), someone asked the maintainers a question: “How much of HAMi’s code now comes from AI assistance?”

The answer from Yang Shouren stuck with me: about half of the code in the HAMi community today is already AI-assisted.

Half. And that share is still rising.

This immediately raises a sharp question: if AI can write more and more code, what is the value of an open source community? Could one person plus a few AI agents just fork a “new HAMi”?

My answer is no, because what AI lowers is the cost of producing code, not the cost of building technical consensus.

In the AI Era, Code Is No Longer Scarce; Consensus Is

Let me sharpen that point.

An AI agent can already do a lot today: write code, fix bugs, add tests, generate docs, produce migration scripts. These capabilities are getting stronger fast. But there are a few things in a community that AI cannot replace today, and in my judgment will not replace soon.

First, deciding which problems are worth solving.

Take HAMi. Why does GPU sharing matter? Not because “slicing cards finely” is a cool technique. It matters because once AI infrastructure scales, reality looks like this: GPU costs are enormous, heterogeneous hardware keeps multiplying, and Kubernetes’ native resource model is no longer enough.

The community has to first agree that “this problem is worth investing in” before anyone writes any code. That agreement is a human-to-human matter, supported by real scenarios, real costs, and real pain. AI can solve a problem you have already defined, but “which problem is worth defining” is decided by community consensus.

Second, choosing a technical path.

In GPU virtualization there are many paths: MIG, MPS, time-slicing, vGPU, DRA. Each has trade-offs, and choosing wrong can cost you two years of detours.

Code can be generated, but architectural choice is fundamentally a value judgment. HAMi’s decision on Ascend 910C to move from hardware SR-IOV to userspace HAMi-core was not about someone writing a better piece of code; it was about the maintainers holding to a judgment that “hardware partitioning is too coarse, software partitioning is more flexible.” That kind of judgment is ground out through repeated discussion, failure, and validation in the community, not prompted out.

Third, trust.

Users don’t choose HAMi because of “how much AI-generated code is in this repo.” They care about: who maintains it? Who reviews it? Are there real production cases? Does the community respond when something breaks?

Each of these is a relationship between people, a product of community organization, not a product of code quality.

Put these three together, and the production model of open source communities in the AI era is shifting:

Figure 12: How the open source production model is changing in the AI era

In the past, code came first and the community sedimented out of the code; in the future, consensus comes first, AI rapidly turns consensus into code, and code flows back to test the consensus. The center of gravity of the scarce resource moves from “code” on the left to “consensus” on the right.

In the AI Era, Open Source Governance Itself Has to Level Up

Since AI has become a new category of contributor, a community’s governance rules have to keep up.

My advice is: don’t treat AI merely as a tool, treat it as a new type of contributor. It used to be “developers write code, humans review”; in the future it will be “humans set intent, AI generates code, the community reviews, and shared knowledge is distilled.” There is an extra layer in the middle, and an extra layer of governance complexity.

HAMi is already responding to this. Its CONTRIBUTING.md is explicit:

If you are using any kind of AI assistance to contribute to HAMi, it must be disclosed in the pull request.

In other words, if you use AI to help with a contribution, you must declare it in the PR. But the community also knows that a norm without a gate is not enough (see the discussion in Issue #1998), and there is already ongoing discussion about how to give that norm real enforcement.

This is actually a problem every AI-era open source project will run into. I’d break it into a few questions:

Must AI-generated code always be declared?
Which model, and what context, did the contributor use?
How do you ensure the security of AI code, avoiding injection and licensing risks?
What process should maintainers use to review an AI diff they may not be able to fully trace themselves?

Whoever figures out and operationalizes these rules first will keep contribution quality stable in the AI era. HAMi’s exploration here is worth a look for every open source project.

What CNCF Incubating Really Means

Back to the promotion itself.

I’d lean against treating it as an “honor.” What CNCF Incubating really validates is not code quality, but whether a project has the capacity to become infrastructure. It examines a whole package: technical maturity, community governance, production adoption, and ecosystem building.

HAMi’s case, in one sentence, is not “a Chinese team built a GPU project.” It is this:

An AI Infrastructure community, jointly shaped by developers from around the world, is taking shape.

The first is a product story; the second is an ecosystem story. The Incubating recognition from the CNCF is recognizing the latter, because competition over infrastructure is never competition between individual products; it is competition between ecosystem networks.

Summary

The open source competition of the next decade will not be just a competition of code, but a competition of communities.

Once AI gives everyone near-infinite capacity to produce code, the truly scarce capabilities will be three: finding the right problem, building technical consensus, and organizing developers worldwide to solve a problem together.

HAMi’s path to CNCF Incubating is just one snapshot of how open source communities are evolving in the AI era. Code will keep getting cheaper, and consensus will keep getting more expensive. Whoever understands this inversion will be the one who can build open source communities with real depth in the AI era.

Join the HAMi Community

Add me on WeChat (jimmysong) or follow HAMi on GitHub to join the community focused on GPU virtualization and heterogeneous compute scheduling. Let’s talk about open source governance in the AI era.

Olares and HAMi: A New Inflection Point for Desktop AI Workstations

Jimmy Song — Wed, 24 Jun 2026 22:30:00 +0800

HAMi used to save cards in the cluster. Now it decides how good a desktop AI workstation feels.

Figure 1: Olares and HAMi: a new inflection point for desktop AI workstations

An Old Friend Mentions a Name

A few days ago an old friend came by to chat. We used to run the cloud-native community scene together back home, so we go way back. She recently joined a company called Olares, and somewhere in the conversation she dropped this: their project had integrated HAMi.

I run the HAMi community day to day, so whenever I hear someone using it for something, I want to take a closer look. I went and dug through the Olares repo and website, and my first reaction was, huh, this is actually interesting.

Isn’t this exactly the kind of local AI workstation I’d been eyeing forever but never pulled the trigger on? I wrote in My Personal AI Stack that for someone like me who mainly works with my head, subscribing to models beats maintaining a high-end GPU. But how far can “one machine running the full AI stack” really go, and who has actually built it, I’ve wanted to see with my own eyes.

What Is This Thing, Really

Olares isn’t a “NAS with a GPU bolted on,” and it isn’t a “mini PC with Ollama installed.” It’s more like a personal cloud built on Kubernetes: local models, apps, identity, remote access, dev environment, storage, even GPU governance, all packed into one machine as a desktop cloud OS.

HAMi isn’t filler here. It’s the layer that turns “one card” into “a resource pool you can share, isolate, and schedule.”

How It Differs from the AI Mini-PC Crowd

There’s no shortage of things calling themselves AI workstations, but most are just beefier mini PCs. Olares is different. It’s genuinely designed as a cloud.

The company positions this machine outright as a 24/7 personal AI cloud, not a PC you sit in front of. That single framing explains almost everything that follows: why it has to use Kubernetes, why it cares so much about network traversal and remote access, why GPU governance suddenly matters here.

Two other details tell you a lot: it supports Thunderbolt 5 external eGPUs, and two machines can cluster up. In other words, it was never a sealed box. It starts as a single node and grows upward.

Stuffing a Whole Cloud Runtime into One Box

Architecturally, what Olares does is compress an entire cloud runtime into a single machine: auth and authorization, app lifecycle, tunnels and traversal, secrets, observability middleware, not a layer skipped.

This isn’t “a few AI apps preinstalled.” It’s a complete cloud runtime stuffed into a desktop device.

The diagram below is my own re-drawn layering based on its public docs, with the HAMi layer pulled out explicitly because it’s the point of this whole piece.

Figure 2: Olares system layered architecture, with HAMi as the GPU resource plane

One Detail That Caught My Eye

One thing jumped out while I was reading: Olares’s docs haven’t kept up with its own product.

Its 2025 architecture page still says nvshare, noting that GPUs are limited to one card per node. But flip through the release notes and 1.12 already integrates the HAMi scheduler, with exclusive, time-slicing, and memory-slicing modes all there; 1.12.2 adds multi-GPU; 1.12.5 supports DGX Spark outright and rolls automatic scheduling across all three modes.

The product is outrunning its docs. That alone tells you something: it’s shifting from a “personal cloud OS” toward a “local AI cloud OS.” That’s also why I wanted to write a whole piece on it.

What HAMi Actually Does in There

HAMi is a CNCF Sandbox project, positioned as a heterogeneous AI compute virtualization middleware. On the path there’s a Webhook, a scheduler, a Device Plugin, and HAMi-core handling in-container resource control. I summed it up in one line in Kubernetes Is Becoming the GPU Control Plane of the AI Era: it turns GPU slicing from “a hardware capability” into “a control-plane capability.”

On Olares, that’s the real thing behind the GPU mode toggle in settings. On the surface it’s a UI option; underneath, it’s a policy switch on the resource plane.

The most critical piece is almost certainly HAMi-core, which in one sentence: intercepts CUDA calls inside the container and does memory virtualization, compute throttling, and utilization monitoring. It doesn’t slice with hardware. It manages with software.

The diagram below puts that injection path and the three GPU modes side by side.

Figure 3: HAMi-core injection path and Olares’s three GPU modes

The evidence lines up too. In Olares’s GitHub issues you can see HAMi’s libvgpu.so crashing under WSL2, the discussion mentions it getting injected into every process via /etc/ld.so.preload, and the Olares team itself says this implementation “drew heavy inspiration” from the HAMi project.

One thing I should be clear about, though: whether Olares uses upstream HAMi-core directly or maintains a forked version, there’s no single official answer in public. I won’t fill that in for them.

The three modes make sense in order. Full-card exclusive is for heavy loads; time-slicing lets lightweight services take turns, and Olares even swaps inactive models into memory first, with roughly 5% switching overhead; memory-slicing cuts the memory into fixed quotas so multiple apps run together. On something like DGX Spark, where CPU and GPU share memory, it defaults to memory-slicing because there’s no traditional memory paging in and out to begin with.

Why This Matters for HAMi

HAMi used to live mostly in big clusters: multi-tenant, inference serving, mixed training-and-inference, heterogeneous cards. NIO running it for sharing across 80 nodes and 600 cards is the textbook example. That line is well-trodden.

What feels new about Olares is that it moves the same problem onto a single machine.

Figure 4: HAMi’s value narrative shifts from cluster efficiency to edge product experience

What you actually run at once on Olares is never just one Ollama: a local model, a chat UI, a research agent, an image or video pipeline, plus a pile of OCR, speech-to-text, and automation tools. In a scenario like that, without a GPU scheduling layer, the GPU always degenerates into “whoever starts first grabs it.”

So for HAMi this matters. It goes from “a tool that saves cards for platform engineers” to “something that directly decides whether an ordinary user has a good time.” In the cluster it manages utilization and cost; on Olares it manages concurrent experience, model switching, and whether this machine is actually pleasant to use. HAMi doesn’t only talk to platform engineers anymore.

Where Edge AI Goes from Here

My sense is, edge AI won’t settle at “a stronger local model box.” It’ll grow into a full stack of “control plane plus model plane plus application plane.”

NVIDIA pushing DGX Spark and RTX Spark onto the desktop has already moved the story from “run models locally” to “run agents locally.” Olares’s CUDA-plus-x86 is one path, DGX Spark’s unified memory is another, and below that there’s the DIY mini-PC-plus-eGPU route you assemble yourself. Interestingly, Olares lists eGPU and dual-node as supported paths itself, so the line between appliance and DIY isn’t that sharp.

But my guess is, the one that breaks out won’t be the one with the most ferocious specs. It’ll be the one that fuses resource governance, dev experience, app ecosystem, security, and remote access into one closed loop. If desktop AI workstations really become a category, what people compete on isn’t GPU and memory, it’s the control plane.

I went ahead and added Olares to the AI Native Landscape I maintain, so I can keep watching how it grows.

Conclusion

In one line: Olares is a desktop AI OS with Kubernetes at its core, and HAMi is the layer that turns its single-machine GPU into a shareable, isolatable, schedulable resource pool.

From cluster to desktop, HAMi’s story is expanding from “saving cards” to “making edge AI usable.” If desktop AI workstations become a real category, what people compete on is the control plane, not single-card performance.

References

Every Nation Begins with Textiles

Jimmy Song — Sat, 20 Jun 2026 05:43:26 +0000

In a bamboo-weaving workshop in Anji, holding a strip of bamboo split hair-thin, I realized for the first time: a bolt of cloth, a machine, a country, all begin with this single thread in the hand.

Figure 1: Bamboo strip

Introduction

This is something I’d been thinking about for a long time, without ever quite sorting it out.

In February this year, our company offsite took us to Anji, in Zhejiang province. It’s famous bamboo country in China, the mountains covered in moso bamboo, and the locals have been weaving with it for generations. Bamboo weaving is a national intangible cultural heritage. The offsite included a hands-on session for us to try it ourselves.

I hadn’t expected much from this kind of “experiential activity.” But once I actually sat down, picked up a bamboo strip, and listened to the old master explain how to lift, press, and raise the threads, my mind started to wander. Up and down, lift and press, the warp and weft interlacing in his hands until a pattern slowly emerged. The motion was deeply repetitive, a fixed rhythm, a clear logic.

And in that moment an almost absurd thought popped into my head: this is just 0 and 1.

A warp thread raised is 1, pressed down is 0. Unfold a piece of cloth and it’s a two-dimensional structure made of 0s and 1s. And the thing the old master kept referring to as the “flower program” (花本), the pre-designed sequence of warp-lifting, is essentially a program: follow it, and you weave exactly the pattern you intended.

Figure 2: A woven bamboo horse I finished in Anji, mounted for framing

That thought flashed by and I didn’t make much of it. But then in March I went to the Netherlands and spent a week in Amsterdam. The two experiences sat together, and some feelings that had been vague began to sharpen: how could a single strip of bamboo, a single motion, connect to the looms of two centuries ago, to today’s computers, even to a country’s industrialization?

I looked it up later and found I wasn’t the first to make the connection. Many historians of technology argue that weaving is one of the earliest forms of large-scale information encoding. Now, in June, I’m sitting down to try to lay out this thread. This isn’t a technical piece. It’s a personal, cultural reflection.

A Cultural Contrast

Let me start with the Netherlands.

What struck me most about that country wasn’t the windmills or the tulips, but a quality that the whole nation seems to exude: restraint, order, and an almost engineering-like way of being. The canals are dug, the land reclaimed from the sea is called a “polder,” and the windmills were never there to look pretty. They were built to pump water and drain the land. A country that sits below sea level literally engineered itself out of the sea. Though the windmills aren’t only about drainage. The wind in the Netherlands is genuinely strong. In March I got battered by it. It’s a wind that pushes at you constantly. So a below-sea-level country hauled itself out of the sea with engineering, and along the way put that fierce wind to work too.

Figure 3: A canal and church in central Amsterdam

Figure 4: Windmills at Zaanse Schans

This quality seeps into every corner of daily life. Shops close around six in the evening, the streets slowly quiet down, and you almost never hear a car horn. Cars stop well ahead of time to let you cross. One evening I wandered a little too close to the bike lane, and a passing car actually rolled down its window to remind me to use the sidewalk. It surprised me at first, and then I understood: the rules there are clear, and everyone assumes you’ll follow them too.

This temperament even shows up in how people get around. In the city center, not many people drive. Bicycles are the default. For one thing, the Netherlands is flat, so cycling takes no effort. For another, dedicated bike lanes are everywhere, and the streets are narrow, which makes them ill-suited to cars. Add that commutes are generally short, and cycling is just right. What’s interesting is that even in the southern suburbs, where the roads are wide and every household has a car, you still see a lot of people on bikes. For them, a bicycle isn’t exercise. It’s the default way to commute.

Figure 5: A street in the southern suburbs of Amsterdam: wide roads, cars in every driveway, and still plenty of cyclists

I set that against the noise and speed of China, the loud, garish signboards back home. These two temperaments are really two paths of modernization. The Netherlands is an early, slow-and-steady kind of country, from the Age of Discovery to the world’s first stock exchange, from water engineering to today’s ASML and Philips. It has always moved forward in a “long-term, restrained, systematic” way. China is a late developer chasing from behind, relying on speed, scale, and a generation’s sheer exertion to cover in a few decades what took others centuries.

Neither path is higher or lower than the other, but they settle into completely different urban textures and national characters. The ease of “closing at six” that the Dutch have is something China won’t reach for many years; and the Chinese speed of “just do it,” of laying a high-speed rail line overnight, is something the Netherlands probably couldn’t match either.

But here’s the interesting part. If you trace both civilizations back through history, these two seemingly different cultures share a common starting point. Textiles.

Why Textiles Were the Starting Point of Industrialization

Our generation is so familiar with the phrase “Industrial Revolution” that it’s almost gone numb. But have you ever asked: where did the first Industrial Revolution actually break out? Not in steel, not in coal, not in the railways. In textiles.

The British Industrial Revolution is practically a history of textile machinery. The flying shuttle in 1733 made weaving faster. The spinning jenny in 1764 let one person spin many threads at once. The water frame in 1769 began replacing human power with water power. The power loom in 1785 turned weaving into an automated process. Every one of these inventions happened inside the textile industry.

Why textiles of all things? At first it feels counterintuitive, since textiles seem too “light,” lacking the heft of making steel or guns. But think about it for a moment and it becomes obvious.

Textiles are a basic need. Everyone has to wear clothes, clothes wear out, and they have to be replaced constantly. That’s an enormous market that already existed back in the agrarian age. Once the demand is there, any gain in efficiency turns immediately into profit, into capital that can be reinvested.

More importantly, the processes of textile production are especially easy to mechanize. Spinning and weaving are deeply repetitive motions with a fixed rhythm and a clear logic, so machines can directly replace the human hand. Steelmaking, by contrast, has a far higher technical barrier, the science of chemistry wasn’t yet mature, and machine manufacturing itself needed an existing industrial base before it could even start.

And textile production has a moderate investment threshold. Building a textile mill was far cheaper than building a steel plant, so merchant capital and money from colonial trade flowed in easily. A large share of early British industrial capital came from the cotton trade and colonial trade, and that money went first into textile mills.

The most crucial point is that textiles don’t exist in isolation. They pull an entire supply chain along with them. To spin and weave, you first have to grow and transport cotton. To run machines, you have to build textile machinery. To power the machines, you need steam engines. To fuel steam engines, you have to mine coal. To move cotton and cloth, you have to build railways. So the steam engine was first deployed at scale in textile mills, the earliest railways carried cotton and cloth, and the earliest machine manufacturing served textile equipment. A supposedly “light” industry ended up dragging the entire mechanical and energy industries into being.

So textiles became the progenitor of industry not because they were the most complex, but because they were the first to satisfy four conditions at once: large demand, mechanizable, low threshold, and able to pull others along. Textiles were humanity’s first trial plot on the way from handicraft into an industrial system.

Late Developers All Come Up This Same Road

Here’s a pattern I’d never noticed before: the industrialization of late developers, almost without exception, begins with textiles.

Take Japan. Today when you think of Japanese industry, you think of Toyota cars. But how did Toyota begin? Its founder, Sakichi Toyoda, didn’t start by making cars. He invented automatic looms. In 1924 he invented the Type G automatic loom, and in 1926 he founded Toyoda Automatic Loom Works, the origin of the entire Toyota Group. It was his son Kiichiro who later moved the business into automobiles.

So Toyota cars quite literally “grew out of a loom.” Its whole system of lean production, kanban management, total quality control, carries in its bones the obsession with precision, process, and zero defects that came from making looms.

Then look at South Korea. Samsung today is a global electronics and semiconductor giant, but when it was founded it exported dried fish, vegetables, and fruit, only later moving into textiles and sugar. As South Korea industrialized after the war, textiles and garments were among its earliest foreign-exchange earners, and Samsung built its first fortune on trade and textiles in that period before pivoting to electronics in the 1970s. The Hyundai Group took a similar path, starting in engineering and heavy industry before expanding into automobiles, shipbuilding, and heavy industry.

Britain was first; Japan and Korea are the late-developer cases. You’ll notice the sequence is strikingly consistent across all three: textiles first, then light industry, then machinery, then heavy industry, and finally high-tech. Nobody designed this path. It was forced by cost and by the market. Because textiles are the lowest-threshold, largest-market form of manufacturing, every nation that wants to turn from an agrarian country into an industrial one has to start from the easiest, most necessary step.

Behind this lies a very plain truth: modernization doesn’t happen in one leap. It needs a starting point that can earn money, accumulate experience, and train the first generation of industrial workers. And textiles happen to be exactly that kind of starting point.

Where China Sits on This Road

Writing this far, I can’t help but turn and look back at China itself.

China’s modern industrialization also began with textiles. The most famous Chinese-owned enterprises of the late Qing and early Republic were almost all cotton mills. Zhang Jian founded the Dasheng Cotton Mill in Nantong. The Rong brothers, Zongjing and Desheng, founded the Shenxin Cotton Mills in Wuxi. They were China’s first generation of national industrial capitalists, rising out of textiles and holding up half of modern Chinese industry. The thinking of Zhang Jian’s generation was plain and direct: foreigners were using machines to weave cloth and taking our money, so we had to set up our own mills, weave our own cloth, and keep that money at home.

That was China’s first stretch. Back then, China was walking the very road that Britain, Japan, and Korea had all walked.

Then the road broke. War, turmoil, the planned economy. Chinese industrialization took many detours. It wasn’t until reform and opening up that it reconnected. In the 1980s the coast was blanketed with processing factories for garments, shoes, and toys. In essence it was still that same old road of “starting with textiles and light industry,” only this time China relaunched it through exports, cheap labor, and the role of factory to the world.

Further on, through the 1990s and 2000s, China pushed into heavy industry: steel, cement, shipbuilding, chemicals, catching up to the world’s front ranks one after another. Then electronics, home appliances, mobile phones. Then the internet, high-speed rail, new-energy vehicles, solar, semiconductors, and the AI that everyone talks about today.

String this line together and you realize China is actually completing the same road that every late developer has walked, only faster, fiercer, and at a far greater scale. From Zhang Jian’s cotton mills to today’s new-energy vehicles and large AI models is barely over a hundred years. In just over a century we’ve run the entire course of industrialization that took Britain more than two hundred and Japan more than a hundred, and we’re still going.

This often leaves me with a complicated feeling. On one hand, admiration: generation after generation, from Zhang Jian to today’s engineers, really did turn an agrarian country into the factory of the world and then into a country that now leads in quite a few high-tech fields. On the other hand, a faint unease: this road was run so fast that a lot of things got left behind.

A Thread That Never Broke

Inside this long thread of industrialization, there’s one detail I never forgot: that moment in the bamboo-weaving workshop in Anji back in February.

The “flower program” the old master described actually has a very old tradition in China. The drawloom of the Han dynasty used a pre-designed system of cords to control which warp threads rose and fell, so a weaver following it could reproduce complex patterns. Then in the early nineteenth century, the Frenchman Joseph Marie Jacquard pushed the idea a big step forward, using punched cards to control the weaving pattern: where the card had a hole, a hook passed through and lifted the warp; where there was no hole, the hook was blocked and the warp stayed down. Hole or no hole, that’s 1 and 0.

Jacquard’s punched cards were later borrowed by the computing pioneer Charles Babbage for his Analytical Engine, and Ada Lovelace used them to write the first computer program in history. Further on, IBM rose on punched-card machines, and if you trace today’s entire computing industry back to its source, it leads, improbably, to a loom.

And that’s not the end of it. The Transformer, matrix operations, the weights of neural networks in today’s AI, at bottom are all about computing relationships across vast fields of “warp and weft.” A loom decides which warp threads to lift and when; a neural network decides which parameters relate to which others. The logic is the same.

So the word “weaving” is more than the starting point of an industry. It’s a metaphor: for thousands of years, human beings have been learning how to encode complex things into structure, from a bolt of cloth, to a program, to a neural network. Every nation does this. They’re just at different stages.

I plan to write the technical bloodline of this story separately, in another piece. Here I’ll only point to it, to make one thing clear: the thread that began with that strip of bamboo has never broken.

Summary

From a bamboo-weaving session in Anji in February, to the canals and windmills of Amsterdam in March, and on to the industrialization of Britain, Japan, Korea, and China, what I want to say really comes down to one thing.

Every nation’s modernization is a road from a single thread woven into a net. Textiles are the common starting point of that road, not because textiles are so lofty, but because they are the most necessary, the easiest, the quickest to earn that first money and train that first generation of workers. From that starting point, some walk with restraint, like the Netherlands; some move with ferocity, like China. Some take two hundred years, some a hundred, some only a few decades. But the starting point is the same, and so is the direction: from that thread in the hand, step by step, woven into the whole of industrial civilization as we know it today.

The Dutch ease of closing at six, of no honking, of swans gliding through the canals, is a place China hasn’t reached yet. The Chinese speed of just doing it, of rolling something out overnight, is something the Netherlands couldn’t pull off either. Between these two temperaments of modernization there is no higher or lower, only trade-offs. And behind the trade-offs lies the question of where a country sits in its industrialization, and what rhythm it’s willing to pay for that net.

That day in Anji, holding a strip of bamboo, I clumsily followed the old master, lifting and pressing, weaving a small, lopsided patch of pattern. I wasn’t thinking about any of this then. But looking back, from that single lift and press, you can trace upward to the Han dynasty drawloom, outward to a country’s industrialization, and forward, faintly, to today’s computers and AI. A single strip of bamboo, connected to so much.

Perhaps the evolution of civilization, in the end, is just humanity learning, over and over, how to weave one thing into another.

Starting from a single thread.

Why GPUs Became the Foundation of AI: A GPU Primer for K8s Veterans

Jimmy Song — Wed, 17 Jun 2026 14:04:42 +0000

Note

This post builds on Ameen Alam’s three-part GPU Architecture series and draws on TrendForce, NVIDIA (Slinky/Slurm, the GPUDirect data path) and Mirantis material on GPU infrastructure and agentic AI. It’s written for Kubernetes veterans who have never touched a GPU and never trained or served a model.

Figure 1: GPU: the foundation of AI

Why a cloud-native veteran is writing about GPUs

For the past decade my home turf has been containers and Kubernetes. From service mesh to the whole cloud-native ecosystem, I’ve spent almost every day on scheduling, networking, storage and observability, but always on the CPU side of cloud native. Last year I formally moved into AI infrastructure (AI Infra).

Once I dove in, I found the concept density absurd. Token, Transformer, Tensor Core, HBM, KV Cache come at you one after another, and almost every doc and article assumes you already know them, which is deeply unfriendly to engineers who have never trained a model or run inference.

I quickly hit on a trick: don’t learn from scratch, use what you already know by heart as a translator.

The mental model a Kubernetes veteran already carries, scheduling, Jobs, Deployments, controllers, caches, utilization, multi-tenancy, and then microservices, service mesh, distributed systems, event-driven, high-availability, maps onto AI Infra almost one to one. Once you build that mapping, the intimidating concepts become “oh, it’s just the GPU version of X”. This “translate AI through cloud-native eyes” approach got me up to speed fast, and I’m writing it down to help friends with the same background skip the detour.

This is the first post in that translation series, tackling the most fundamental question: why does AI absolutely need GPUs? Later posts will cover GPU resource management, scheduling and observability, topics a K8s veteran knows well. If you’re also crossing over from cloud native, I hope this saves you a few days of digging.

You type a sentence and the AI replies word by word. For someone who has used K8s, the most natural mental model is: AI is a “workload” that runs on GPUs, and a GPU is a special kind of “node” built for it.

Translating the jargon into K8s

The AI world throws around terms that read like scripture to anyone who hasn’t done training or inference. Here’s a cheat sheet; I’ll re-explain each term in plain words below.

AI concept	In plain words	K8s veteran’s analogy
Token	A small unit text is chopped into; AI generates them one at a time	A log line, a text chunk
Model	A “scoring program” holding billions of numbers (weights)	A giant image whose “weights” aren’t code
Training	Feed it huge data, tune params repeatedly, build the model	Run a Job to build an image; offline, batch, care about throughput
Inference	Model is done, serve user requests and emit answers	Run a Deployment taking traffic; online, care about latency
Transformer	The architecture shared by nearly all LLMs (GPT, Claude, LLaMA)	A controller design pattern
Tensor Core	A GPU unit that only does “matrix multiply” but insanely fast	A sidecar worker that only does batched multiply
HBM	The GPU’s on-board high-bandwidth memory; model and cache live here	Node-local RAM, but with bandwidth that crushes normal RAM
KV Cache	Attention state stored at inference time, grows with the conversation	A pod-local session notebook that gets thicker the longer you talk

Table 1: AI jargon → K8s veteran cheat sheet

Memorize this table; the rest follows.

The question most people never ask

GPUs are everywhere in AI now, so accepted that most people skip past it. We care about which card to rent, which framework to use, which model to deploy, but rarely stop to ask the deeper question:

Why GPUs?

Why not the more general CPU? Why did a chip originally built to render game graphics become the foundation of the most important technology shift in a generation?

The answer isn’t just “it’s fast”, it’s about how computation itself is organized, and why AI workloads demand an architecture fundamentally different from fifty years of software.

Two philosophies of computing: the craftsman and the factory

You know CPUs well. K8s’s control plane, etcd, the scheduler all run on CPU; it excels at executing complex instructions one after another, with few cores (8 to 128) but each highly capable. A CPU optimizes for latency: how fast can I finish one complex task? Like a master craftsman doing one intricate job at a time.

A GPU takes the opposite path: execute simple instructions across massive amounts of data at once. A modern data-center GPU has thousands of small cores, and it optimizes for throughput: how many simple tasks can I finish at the same moment? Like a factory floor of thousands of workers, each doing one identical step simultaneously.

Figure 2: The CPU craftsman vs the GPU factory: latency-first vs throughput-first computing

For any single complex job, the craftsman (CPU) is faster; but as long as the work is uniform and parallelizable, the factory (GPU) crushes the craftsman in total output per second. For decades CPU dominated because most software (web servers, databases, operating systems) is inherently serial. Then deep learning arrived.

Why AI broke the CPU

The core of a neural network is, essentially, a giant pile of matrix multiplications (an operation that batch-multiplies-and-adds two sets of numbers). When a model processes a token, it runs thousands or tens of thousands of these multiplies, and they don’t depend on each other.

The key insight in one line: these operations are naturally parallel.

A 128-core CPU looks at this workload and sweeps through it sequentially; a many-thousand-core GPU chops the matrix into blocks and hands them to its small cores to run at once. Same work, orders of magnitude faster on GPU, not because a single GPU core is faster (it’s slower), but because the problem is parallel and the GPU is built for exactly that shape of computation.

That’s the real reason AI runs on GPUs: not marketing, not legacy, architectural fit.

The three things inside a GPU, in K8s terms

1. Tensor Core: the sidecar that only does batched multiply

A GPU has two kinds of cores. Regular CUDA cores are general-purpose workers that can compute anything; a Tensor Core is a dedicated unit that does exactly one thing: multiply two small matrices in a single shot. But that one thing it does insanely fast, finishing in one cycle what would take a regular core thousands.

AI’s core operation is matrix multiplication, so Tensor Cores are tailor-made for it. In K8s terms: regular CUDA cores are like general pods in a Deployment that do everything; a Tensor Core is like a highly specialized sidecar that only does “batched multiply”, single-function but with crushing throughput.

2. HBM: the node’s high-speed local memory

Computing fast isn’t enough; you have to get the data. A GPU’s memory is layered just like a CPU node’s, and if you understand a K8s node’s L1/L2 cache, RAM and local disk, you understand the GPU’s:

Figure 3: GPU memory hierarchy: faster and smaller up top, slower and bigger below; HBM bandwidth is the lifeblood of AI

The bottom layer, HBM (High Bandwidth Memory), is the GPU’s “main memory”; model weights, intermediate results and KV cache all live here. It’s characterized by large capacity (hundreds of GB per card) and extremely high bandwidth.

Why build HBM at all? Because traditional VRAM (GDDR) lies flat on the circuit board, you run out of routing space and hit a bandwidth wall. HBM’s answer is to stack memory chips vertically (connected by Through-Silicon Vias, TSVs), packed right against the GPU die with an interface thousands of lines wide running in parallel. By analogy: normal memory is like a warehouse spread across a parking lot where the movers can’t keep up; HBM is like building that warehouse into a dozens-story tower right next to the workshop, with TSV elevators running up and down at full speed.

Here’s a counter-intuitive fact: modern AI is memory-bound more than compute-bound. Tensor Cores do matrix multiplication faster than HBM can feed them data, so the GPU is often waiting. That’s why each new GPU generation (H200 → Blackwell → Vera Rubin) sees its most important upgrade in HBM bandwidth, not compute (4.8 → 8 → 22 TB/s, source).

Since data movement is the bottleneck, NVIDIA’s other move is to let data bypass the CPU and reach the GPU directly, the three “expressways” collectively called GPUDirect:

GPUDirect Storage: data goes straight from NVMe to GPU memory, without detouring through host memory and the CPU.
GPUDirect RDMA: GPU memory talks directly to the NIC (InfiniBand), so cross-node gradient exchange no longer hops through the CPU.
NVLink: GPUs inside one machine connect directly and share memory.

In one line: the CPU is no longer the traffic cop for every data movement. In K8s terms, it’s like sending data over a direct data plane instead of routing every packet through the apiserver. NVIDIA’s edge isn’t just fast compute; it’s that the entire data highway from storage to GPU, GPU to GPU, and GPU to network has been straightened out.

3. Transformer + Token: the program that emits answers word by word

A Transformer isn’t hardware, it’s a model architecture (a program structure). GPT, Claude, LLaMA and other large models are all built on it. What it does is actually plain:

Read a piece of text, predict the next token.

A token is a small unit text is chopped into (a character, half a word). A Transformer feeds the input token sequence in, emits “the most likely next token”, appends it, feeds the new sequence back in, predicts the next-next, and so on, generating the whole answer one piece at a time.

Figure 4: Token and Transformer: AI plays relay, reading the tokens so far at each step and predicting the next

In K8s terms: a Transformer is a controller whose reconcile logic is “look at the current state (tokens so far) → emit the next action (the next token)”, looping continuously. That’s where the “word by word” effect of AI assistants comes from.

Training vs Inference: writing the recipe vs serving the dish

This is the easiest pair to confuse when starting out, yet the most critical. Training and inference are two different things with completely different hardware needs.

Training: feed huge data, repeatedly tune those billions of parameters, “teach” the model into existence. It runs for weeks to months as an offline batch job, cares about throughput, not per-request latency. In K8s terms, it’s a long-running Job whose goal is to build a model image.
Inference: once the model is trained and online, it takes each user’s input, computes an answer and emits it back. The user is waiting, so it cares about latency; it must serve thousands of users at once, so it cares about concurrency. It’s like a Deployment taking traffic.

Figure 5: Training vs Inference: training is a Job writing the recipe (offline, throughput), inference is a Deployment serving the dish (online, latency)

This directly affects which GPU metrics you should care about. The headline numbers (Tensor Core FLOPS, NVLink bandwidth, cluster size) are mostly training metrics; what actually determines whether your AI assistant feels snappy (HBM capacity, memory bandwidth, KV cache management) are inference metrics, and they get little airtime. Yet inference is where most production AI actually runs.

Multi-GPU collaboration: an AI cluster is just a distributed system

With single-card covered, back to reality: the biggest models don’t fit on one card, and training routinely needs hundreds or thousands of cards working together. At that point the GPU cluster is essentially a distributed system, and almost everything you know from cloud native applies.

Many GPUs = a microservice cluster. A card is like a service instance; the model is sharded across cards, each computes a slice and the results are stitched back. That’s exactly the microservices playbook of splitting by responsibility and sharing load.

Inter-card communication = the service-mesh data plane. Cards constantly exchange data (gradients especially during training). Within a rack it’s NVLink (several TB/s), across racks it’s InfiniBand. It’s just the east-west traffic between microservices: in-node pod-to-pod direct is fastest (NVLink), cross-node cross-cluster goes over the network (IB). NVIDIA packaging GPUs, switches, DPUs and RDMA into a whole rack (Vera Rubin NVL72) is exactly a service mesh unifying the data plane, control plane and observability.

Distributed training’s all-reduce = distributed consensus. Each training step synchronizes and averages the gradients across all cards, a step called all-reduce. Anyone who has done etcd or Raft gets it instantly: it’s distributed-system consensus and consistency, except here you’re syncing gradients, not a state-machine log.

Continuous batching = event-driven. During inference, new requests are dropped into the running batch as they arrive, no waiting for the whole batch to finish. Requests are events, the batcher is a consumer, batch-then-process: that’s the event-driven / message-queue mindset, all to keep the GPU busy.

Disaggregated inference = microservices. Split prefill (process input, compute-heavy) and decode (emit tokens, memory-bandwidth-heavy) into separate GPU pools that scale independently, with KV cache passed between them over NVLink/RDMA. That’s entirely the microservices pattern of splitting by responsibility and scaling each part; KV cache is the session context passed between services.

See the pattern? An AI cluster isn’t a new species, it’s your familiar distributed-systems, microservices and service-mesh toolkit replayed on GPU hardware. The instincts you built tuning traffic on Istio and Envoy, or consensus on etcd, are worth exactly as much here.

Running GPUs on K8s: Slurm for training, K8s for inference

By now, as a K8s veteran, you’re bound to ask: so what actually schedules a GPU cluster? The interesting answer: training and inference often run on two different schedulers.

On the training side, the HPC world’s heavyweight is Slurm. It manages 65% of the world’s TOP500 supercomputers (NVIDIA), and large AI training teams have years invested in Slurm scripts, fair-share policies and accounting. Training is a long-running batch job that wants topology awareness (place chatty GPUs close together), long exclusive holds and high throughput, areas Slurm has refined for over a decade.

On the inference side, the home field is Kubernetes. Inference is an online service that wants fast autoscaling, per-request scheduling and integration with service mesh and observability, exactly K8s’s strengths.

So what if a team needs both, maintain two environments? NVIDIA open-sourced the Slinky project to solve exactly this, in a very K8s-native way:

slurm-operator: turns each Slurm daemon (slurmctld for scheduling, slurmd for compute, slurmdbd for accounting) into a K8s CRD and Pod, with the control plane made highly available through Pod regeneration instead of Slurm’s native HA. Config changes sync automatically via ConfigMap/Secret, workers autoscale with HPA, scale-in drains running jobs first, and upgrades use PodDisruptionBudget to protect in-flight work.
GPU Operator + DCGM Exporter: auto-installs drivers and the device plugin, and can label metrics by Slurm job ID, giving you per-job GPU metrics (scraped by Prometheus).
ComputeDomains + DRA: for cross-node-NVLink machines like GB200 NVL72, K8s uses DRA (Dynamic Resource Allocation) to dynamically manage the cross-node GPU interconnect domain, so distributed training hits full NVLink bandwidth across nodes.

NVIDIA itself runs Slinky in production up to 8,000+ GPUs, with NCCL all-reduce/all-gather performance matching bare Slurm, the K8s layer adding almost no overhead (NVIDIA production data). K8s is becoming the substrate for GPU computing, with Slurm as the scheduling layer on top.

Training and inference have fundamentally different GPU-scheduling needs, so they can’t be managed the same way. That’s also why GPU resource management (covered later) has to be scenario-specific, and why solutions like HAMi aim to be a unified GPU resource-management layer across Slurm + K8s hybrid environments.

The real trap of inference: KV Cache

If there’s one concept that separates “GPU theory” from “AI inference reality”, it’s KV Cache.

When a Transformer generates each token, it has to reference the relationships among all previous tokens (this is “attention”). To avoid recomputing from scratch every time, it stores each token’s attention state, and that store is the KV Cache.

The problem is it grows without bound:

A user sends a 10,000-token conversation; the model stores a state entry for each.
To generate the next token, it reads the entire KV Cache.
A 70B model at 32K context can produce a KV Cache of 8 to 16 GB per request, larger than the model itself.
50 concurrent users land, and KV Cache eats hundreds of GB of memory.

Figure 6: KV Cache is a session notebook that thickens the longer you talk: every new token forces a full read of the whole notebook

In K8s terms: KV Cache is like a session notebook local to a Pod. The longer the conversation, the thicker the notebook, and every new word forces a full read from cover to cover. So inference memory is often eaten not by the model but by this “notebook”.

That’s also why we got paged attention (managing the notebook like virtual-memory pages to cut fragmentation, popularized by vLLM) and NVIDIA Dynamo’s multi-tier cache (hot pages in HBM, warm offloaded to CPU memory, cold spilled to NVMe). Half an inference engineer’s job is managing this notebook.

GPU utilization lies

K8s folks know “utilization lies” best: a pod reporting Running isn’t necessarily doing work, it might be in CPU steal or waiting on IO. GPUs are worse.

That “GPU utilization %” in monitoring usually only measures “is the GPU executing any kernel”, not how efficiently. A card can report 90% utilization while its Tensor Cores are actually busy only 30% of the time, with the rest spent on memory ops, kernel-launch overhead, or just waiting for data.

The more honest metric is SM Efficiency (SM activity rate): it looks at how many SMs are doing useful work each clock cycle. A card showing 100% utilization in nvidia-smi may have an SM Efficiency of only 20-30%. Many companies think their “GPUs are maxed out” when in fact huge amounts of compute are spinning idle. So to judge whether a GPU is truly working, don’t look at utilization, look at SM Efficiency.

The metrics that actually matter (mapping to the QPS, P99 latency and resource levels you watch in K8s):

Token throughput: tokens generated per second per card, how many users you can serve (like QPS).
TTFT (Time To First Token): time from receiving a request to emitting the first token, sets the “responsiveness feel” (like cold-start time).
TPOT (Time Per Output Token, also called ITL / inter-token latency): average time to produce each token, sets streaming smoothness (like P99). Serving frameworks like vLLM and TGI generally use TPOT; NVIDIA more often calls it ITL, same thing.
Memory breakdown: how much is weights vs KV cache vs transient activations, tells you whether you’re memory-bound (like breaking down a pod’s memory usage).

Why NVIDIA is so hard to dislodge

You can’t talk GPUs without NVIDIA’s dominance. The hardware is excellent, but hardware alone can’t explain why AMD, Intel and a crowd of startups have failed to gain ground. The answer is ecosystem depth.

CUDA isn’t just a parallel-computing platform, it’s a programming model, compiler, runtime and a stack of libraries, refined for 18 years. Nearly every AI framework (PyTorch, TensorFlow, JAX) grew up on CUDA first and was ported elsewhere as an afterthought. In K8s terms: CUDA is to GPUs roughly what the Linux kernel + containerd + the whole CNCF toolchain are to the container ecosystem, not something you replace by swapping a kernel.

Switching to another card means re-validating every layer of your inference stack (kernels, libraries, frameworks, serving, monitoring, ops). The switching cost isn’t buying different hardware, it’s rebuilding an entire software ecosystem. That’s the “CUDA moat”.

The future: AI factories

NVIDIA no longer describes its business in terms of “GPUs” or even “data centers”, but as “AI factories”: facilities that continuously convert electricity, silicon and data into intelligence. Beneath the language is a real architectural shift:

Power is now the primary constraint: a single Vera Rubin rack draws over 100 kW (source), and a mid-size training cluster needs 10 to 50 MW, comparable to a small town. The GPU is no longer the hard part; securing reliable power is.
Cooling moves off air: liquid cooling is now standard for high-density GPUs.
Networking decides GPU placement: AI cluster design now starts with network topology and works backward to where GPUs go, the reverse of traditional data centers.
Memory bandwidth remains the scaling frontier: each generation adds compute, but what actually moves the experience is HBM bandwidth.
It’s fundamentally HA + datacenter ops: power redundancy, cooling failover, latency determined by network topology, single-point failure and DR are all old problems for anyone who has done distributed-systems HA. The AI factory isn’t new magic; it’s your HA-architecture skillset moved into a data center an order of magnitude denser in power and compute.

Don’t retire the CPU just yet: agentic AI is rebalancing the CPU:GPU ratio

After all this GPU praise, you might think the CPU is sidelined in the AI era. The opposite is true: agentic AI is putting the CPU back at center stage.

In traditional LLM inference the CPU mostly just compresses and routes data for the GPU, so AI data centers ran CPU:GPU ratios as low as 1:4 or even 1:8. Agents are different: they plan tasks autonomously, call tools, route data between sub-agents and decide whether a task is complete, and all of that orchestration logic lands squarely on the CPU. Add that agents are often trained with reinforcement learning, where every action has to be evaluated by the CPU, and the CPU load gets heavier still.

The signal for K8s veterans is clear: the future AI node is a mixed-workload node where CPU and GPU are billed together, and scheduling and resource management must handle both, not just stare at the GPU.

What this means for me (a K8s veteran)

After all this hardware, it lands on what I work on: only by first understanding why the GPU is the foundation of AI can you understand why “GPU resource management” is a real problem.

When a card costs hundreds of thousands of yuan, a whole rack draws over a hundred kilowatts, and production GPUs still run below capacity because memory bandwidth, KV cache or batching aren’t tuned, just “handing out cards” is nowhere near enough. How to run multiple tenants safely on one card (like K8s scheduling many pods onto one node), how to partition resources between the very different workloads of training and inference, how to push utilization from “looks full” to “actually full”, that’s exactly what GPU virtualization and sharing (the work I do) solves.

Here’s a sobering number: per the ClearML AI infrastructure survey, only about 7% of enterprises hit over 85% GPU utilization at peak, more than half sit at 51-70%, and 15% are below 50%. In other words, a big chunk of the GPUs companies pay dearly for are spinning idle. That’s usually not a hardware shortage, it’s scheduling and management not keeping up.

My own take is that GPU resource management is moving through three stages: Allocation → Utilization → Efficiency. Stage one only answers “who gets this card” (device plugin, exclusive or time-slicing); stage two asks “is this card full” (dynamic batching, elastic autoscaling, where most enterprises are stuck); stage three asks “is this card’s compute producing maximum value” (SM Efficiency, tokens per watt, bandwidth utilization). The real challenge is stage three, and the future GPU scheduler shouldn’t be a resource allocator but a fine-grained orchestrator of the GPU’s internals, reading how many SMs are active, the Tensor Core utilization, how much memory KV cache holds, and only then deciding whether one more request fits.

Plainly put, the AI era is replaying the cloud-native story: from “single-machine exclusive” toward “multi-tenant sharing + scheduling + observability.” And the GPU is the main stage of that play.

This is only the first half: who else is at the table besides NVIDIA

By now you’ve probably noticed this post barely mentions anyone but NVIDIA. Unavoidable, it’s the absolute protagonist today. But if you think the AI accelerator world begins and ends with NVIDIA, you’re very wrong.

In fact, an “anti-NVIDIA alliance” is gathering from all sides:

Cloud-vendor silicon: Google’s TPU is on its sixth generation and backs almost all of its own AI; AWS’s Trainium (training) and Inferentia (inference) keep spreading; Meta and Microsoft aren’t sitting still.
Traditional chip giants: AMD presses hard with the Instinct line and ROCm, Intel holds ground with Gaudi and oneAPI.
China’s heterogeneous accelerator ecosystem: Huawei Ascend, Hygon DCU, Cambricon, Moore Threads, Enflame, Kunlunxin, Metax, Biren…, sprinting through the domestic-substitution window, with their software stacks climbing from “works” toward “works well”.
The open ecosystem fights back: the UXL Alliance, OpenAI Triton, and PyTorch’s native AMD/TPU backends are all gnawing at the walls of the CUDA moat.

What does that mean for a K8s veteran? It means the future AI cluster is almost certainly heterogeneous: a rack might hold NVIDIA, AMD, TPU and domestic cards side by side, and one schedule has to manage several completely different kinds of hardware.

So the really interesting questions follow:

How do you mix multiple accelerator types in one cluster and still schedule and observe them uniformly?
Where do K8s device plugins and DRA (Dynamic Resource Allocation) evolve to, so they can elegantly describe this menagerie of hardware?
Will the CUDA moat be breached by the open ecosystem, or will NVIDIA rule long-term like x86 did?
What pieces are still missing before domestic heterogeneous accelerators are truly production-ready?

I’ll dig into those in later posts. This one only nails “why GPUs”; the next will tackle the big chess game of how AI infrastructure should schedule things once “GPUs aren’t just one kind”.

References

Ameen Alam, Why GPUs Became the Foundation of Modern AI (Part 1 of the trilogy)
Ameen Alam, The Hidden Technology Behind Modern AI GPUs (Part 2)
Ameen Alam, Real AI Infrastructure and the Future of GPUs (Part 3)
TrendForce, The Great Rebalance: How Agentic AI Is Reshaping the CPU/GPU Ratio
Anton Polyakov (NVIDIA), Running Large-Scale GPU Workloads on Kubernetes with Slurm (Slinky / Slurm on K8s)
Kawonise, Why NVIDIA GPU Architecture Is Perfect for AI: GPU Data Path for a Single Node (GPUDirect data path)
Mirantis, GPU Infrastructure Automation and Strategy (GPU infra automation and utilization)

GPU Utilization Is Breaking: AI Infrastructure Needs a New Definition of Efficiency

Jimmy Song — Wed, 17 Jun 2026 06:22:24 +0000

Every GPU should not just be used. It should create value.

We have all been chasing GPU utilization

For the past few years, whether it is Kubernetes GPU scheduling, vGPU, MIG, or HAMi, everyone has really been doing the same thing: pushing one number up.

GPU Utilization

It makes sense. GPUs are expensive. An H100 can run anywhere from a few dollars to over ten dollars per GPU-hour, and nobody can afford to let a GPU sit idle. So the entire AI Infra community’s narrative for the past few years has boiled down to one line:

Maximize GPU utilization.

I have been involved in the HAMi community for a long time, and a few posts I have written, like Kubernetes as the GPU Control Plane for AI and From GPU to Token: An Eight-Layer Observability Stack for AI Infrastructure, are really about the same thing: how to slice GPUs finer, share them more thoroughly, and schedule them more sensibly.

But recently I read Arjun Kaarat’s piece in Towards Data Science, When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI, and it made me rethink a question:

Is GPU utilization really the metric we should be optimizing for?

There is one line in the article that made me pause for a few seconds the first time I read it:

“GPUs can be busy without being productive.”

Arjun Kaarat

It punctures a layer of illusion. The number we have spent so much effort pushing up may have been answering the wrong question from the start.

GPU Busy is not GPU Productive

Kaarat tells a representative story in the article.

At 2 AM, an infrastructure team gets paged: inference latency just spiked 60%. They open the monitoring dashboard, and GPU utilization looks perfectly normal:

GPU: 79%
GPU: 82%
GPU: 84%

Looks healthy. So the usual playbook kicks in: trigger autoscaling, add nodes, add GPUs. The cloud bill climbs, but latency barely improves.

An hour later, they find the root cause: three nodes had quietly entered a RAID rebuild state, storage throughput was severely dragged down, and the inference tasks around them were starving. The scheduler kept treating these nodes as “still healthy enough” because the GPU and memory metrics looked fine, but the underlying disk performance had collapsed.

What strikes me most about this story is that it is not a rare edge case. It is a failure mode that is becoming common.

Many teams look at their monitoring and see:

GPU: 82%
GPU: 84%
GPU: 79%

and think “our cluster is busy and healthy.” But at the same time:

Latency ↑
Queue ↑
Throughput ↓
Cost ↑

The real problem is that the GPU is waiting.

Waiting for what?

The retrieval pipeline to feed embeddings over;
The SSD to read context out of storage;
The CPU to prepare the data pipeline;
The KV cache to have room for new requests;
Storage I/O to not be squeezed out by background tasks.

The GPU end is full, but the data path feeding it is empty, blocked, or collapsing. From the dashboard the GPU reads 84%, but the actual output may be less than half. Kaarat describes this state precisely in the original piece:

A GPU that appears active may still spend meaningful time waiting for the system around it.

That is the overlooked gap between “busy” and “productive”.

The “illusion” of GPU utilization

The RAID story above is still just a “point failure”. The more compelling part of Kaarat’s article describes a systemic phenomenon: Fragmentation.

Consider a cluster with three nodes, after running a mixed wave of GenAI workloads:

Node	GPU compute	HBM	Storage bandwidth	I/O CPU
A	available	nearly full	available	available
B	available	available	saturated	available
C	limited	available	available	saturated

Table 1: Residual resources on three nodes after a GenAI wave

Now a new inference job arrives, with an ordinary footprint: a little GPU, a little VRAM, decent storage bandwidth, decent I/O capacity.

In total, the cluster still has plenty of resources. A has GPU and bandwidth, B has VRAM, C has bandwidth and CPU. But no single node can take this job on its own.

That is fragmentation. I drew it out, roughly like this:

Figure 1: The cluster is not short on resources. It is short on resources of the right shape.

The cluster is not empty. It has just been carved into “leftovers” that can no longer be used productively. Kaarat sums up the phenomenon in one line, which I think is the most memorable sentence in the whole piece:

The cluster is not empty. It is fragmented into leftovers that are difficult to use productively.

Put another way:

The cluster does not lack resources. It lacks resources of the right shape.

This judgment matters a lot for a project like HAMi, which builds a GPU resource control plane. We used to think that “slicing the card finely and letting more people share it” was the answer to fragmentation. But Kaarat points to a deeper problem: fragmentation is not just a GPU-layer concern. It spans GPU, HBM, storage bandwidth, and I/O CPU. You can slice the GPU as finely as you like, but if the storage dimension is choked, that node is still unavailable for the next genuinely useful task.

What HAMi solves

Let me directly answer a question: what role does HAMi play in this chain?

HAMi solves a very specific, and very foundational, problem:

Can the GPU be used by more people.

What it does can be summarized in one line: reduce fragmentation at the GPU layer. The concrete forms include:

GPU Sharing: letting multiple Pods share one card instead of one card per Pod;
vGPU / HAMi-core: doing memory isolation and compute throttling in userspace, slicing one card into MB-level virtual devices;
MIG integration: managing NVIDIA MIG hardware partitions in software;
Heterogeneous GPU abstraction: abstracting more than a dozen device families, including NVIDIA, Ascend, Cambricon, Hygon, and Vastai, into semantics the scheduler can consume uniformly;
DRA compatibility: keeping pace with the evolution of the Kubernetes resource model.

This is the first layer of efficiency. It answers the question “can the GPU be put to use”.

At this layer, HAMi’s value is already clear: in real environments with mixed domestic and foreign GPUs, mixed training and inference, and multi-tenant sharing, HAMi lets one card serve more workloads, so the cluster is no longer wasted by a coarse-grained “one card per Pod” model.

But note: this is only the first chapter of the efficiency story.

What comes after HAMi

If I step back and look at it from a higher vantage point, “improving GPU utilization” is actually solved across three distinct layers.

Layer one: can the GPU be sliced, shared, and allocated?

This is what a project like HAMi solves. It corresponds to the GPU resource control plane.

Layer two: who gets to use the GPU? Who runs first, who queues, who has priority?

This is what schedulers like Volcano, Kueue, and KAI Scheduler solve. It corresponds to job queuing, fair share, priority, and gang scheduling.

Layer three: can the GPU actually run? Are the data path, storage I/O, and KV cache keeping up?

This is exactly where Kaarat’s article sounds the alarm. When a node’s RAID is rebuilding, its SSD queue is exploding, and its I/O CPU is eaten by background tasks, no matter how much GPU you allocate to it and no matter how elegantly the scheduler queues its tasks, it still cannot produce effective compute.

Draw these three layers together, and you get what next-generation AI infrastructure should actually look like:

Figure 2: From GPU utilization to productive GPU-hours

I think this diagram is the single most important one for understanding the whole picture.

For the past few years, almost all of the industry’s attention has been on the bottom two layers: Kubernetes and HAMi. Those two layers have essentially solved “can the GPU be put to use”. Volcano, Kueue, and KAI are also mature at layer two, solving queuing and priority.

But layer three, storage / I/O-aware scheduling, is currently almost a blank. And that is precisely the layer Kaarat’s article keeps emphasizing, the one that is becoming increasingly valuable in modern GenAI systems. Because for workloads like RAG, long context, and multimodal, the bottleneck has long since shifted from “is the GPU enough” to “is the data path feeding the GPU clear”.

To put it bluntly: HAMi slices the GPU finely, and Volcano queues the jobs well, but if a node assigned a task cannot feed its GPU, then all that upstream effort is just pumping blood into an idle endpoint.

What should next-gen AI infrastructure optimize for

Based on the layering above, I want to make one clear point: we should upgrade the optimization target from “GPU utilization” to “Productive GPU-Hours”.

This is not wordplay. It is a shift that translates directly into money.

Kaarat does the math in the article. A 1000-H100 cluster, at a blended cost of about $3 per GPU-hour, runs around $26 million a year. If fragmentation and I/O stall quietly waste 10% of the effective GPU time, that is roughly $2.6 million a year of wasted spend. Not because the GPUs are missing, but because the system failed to use them efficiently.

That math can be translated into a simple contrast.

The past target:

Maximize GPU Utilization

Today’s target:

Maximize Productive GPU-Hours

The future target:

Maximize Productive Compute
Across Heterogeneous AI Clusters

This evolution maps exactly onto the three-layer structure in the diagram above. That final line, “Across Heterogeneous AI Clusters”, is the heterogeneous narrative HAMi has been pushing all along: in the future you will not optimize just one kind of GPU. You will maximize effective compute uniformly across completely different cards from NVIDIA, Ascend, Cambricon, and Hygon.

In other words, HAMi’s long-term value should not be boxed into the old “improve GPU utilization” narrative. Its real direction is: a resource control plane that lets Productive GPU-Hours be maximized across heterogeneous AI clusters.

From utilization to productive GPU-hours

If I had to summarize this whole line of thinking in one sentence, I would put it like this:

HAMi solves “how to let more GPUs be used”, and next-generation AI infrastructure has to solve “how to let every GPU actually create value”.

That is also the deepest line Kaarat’s article left me with:

The real question is no longer “Are the GPUs busy?”

It is: “Are they productively busy?”

It also makes me rethink what the HAMi community’s tagline for the next phase should be. We used to say “let GPUs be shared by more people”. Next, we should probably move toward:

Turn GPU utilization into productive GPU-hours.

Let every GPU not just be used, but genuinely create value.

GPU utilization was never the endpoint. It was only the first chapter of the efficiency story. Only when we start talking about Productive GPU-Hours, and start caring about storage, I/O, the data path, and the aggregate output of heterogeneous clusters, does AI infrastructure truly enter its second chapter.

Acknowledgments and references

Parts of this post were inspired by Arjun Kaarat’s piece in Towards Data Science, When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI, and quote its published paper and article. My thanks to the author. Both diagrams in this post (resource fragmentation, the efficiency layering) were redrawn by the author based on the ideas in the original, not copied from it.

Arjun Kaarat. When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI. Towards Data Science, 2026.
Kaarat, A., Batthula, V. J. R., & Segall, R. Fitting the Void: Residual-Aware Geometric Packing for GenAI Workloads. IEEE, 2025.

When an Agent Becomes a Distributed State Machine: Agentic AI Infrastructure Reliability

Jimmy Song — Tue, 16 Jun 2026 12:45:16 +0000

Statement

This is my reading and critique of the paper “AI Infrastructure Reliability Features and Architecture for Agentic AI” (June 2026), shared by Hesham ElBakoury in the Open Compute Project (OCP) community. It blends my personal engineering perspective from the Kubernetes / AI Infra space, and is not a translation of the original paper.

An Agent is not a single inference request but a long-running distributed state machine; therefore, Agent reliability is fundamentally a distributed-systems problem.

Why I wanted to write about this paper

After years in cloud native, I have a stubborn instinct: whether a new technology is mature is not judged by how powerful its model is or how flashy the demo looks, but by whether it has a systematic language for reliability. Kubernetes won not on the container runtime, but on liveness/readiness probes, the reconciler loop, PDs/PDBs, and the whole SRE vocabulary that made “long-running distributed state” legible.

So when I came across the paper Hesham ElBakoury shared in the Open Compute Project (OCP) community, it caught my eye. It proposes no new algorithm or framework; the entire paper does one thing: systematically translate the traditional SRE reliability vocabulary onto Agentic AI. This is exactly what I have been circling around for the past six months in Agentic Runtime Realism and Ark Agentic Runtime, Analyzed, without a canonical reference to align against. So I decided to write a dedicated post: unpack its framework, then give it a critical evaluation from the AI Infra practitioner’s point of view.

One-line positioning: it reads more like an SRE white paper for Agentic AI than a systems paper. Its value is not in novelty but in “building consensus”.

One-line summary

Traditional AI cares about model accuracy, while Agentic AI must care about “reliability during long-running operation”, so fault tolerance, recovery, monitoring, security, and state management must be elevated to first-class citizens of architectural design.

The fundamental split between Traditional AI and Agentic AI

The paper argues the fundamental difference between Traditional AI and Agentic AI lies in the execution model. Traditional AI is one-shot request-response, focused on accuracy, latency, and throughput; Agentic AI is a continuous loop, where a single error is no longer just a wrong answer but a wrong decision that may change every subsequent action of the Agent.

Figure 1: Traditional AI’s request-response vs Agentic AI’s perceive→think→act→observe loop

The comparison table below states this most clearly. I suggest you focus on the last row, “failure impact”, because that is the thesis of the whole paper:

Aspect	Traditional AI	Agentic AI
Execution model	Request-response	Continuous loop
State management	Stateless	Stateful
Trigger	Human-triggered	Self-initiated
Time span	Single interaction	Long-running session
Failure impact	Single wrong output	Cascading behavior changes
Resource usage	Bursty	Continuous

Table 1: Execution model: Traditional AI vs Agentic AI

The point of this table is not the technical details but the shift in mental model: when the impact of failure escalates from “one wrong answer” to “behavior-level cascading errors”, the weight of reliability must be reassigned from the very bottom of the architecture. Engineering fault tolerance for a stateless API and for a self-directing, continuously running state machine are simply not the same problem.

The core contribution: a five-dimension Agent reliability framework

This is the most memorable part of the paper. The author decomposes Agent reliability into five dimensions, forming a complete evaluation coordinate system. I drew it out: “Agent Reliability” sits in the center, with the five dimensions fanning out like petals:

Figure 2: The five-dimension Agent reliability framework: the first four describe the Agent’s own behavior, the fifth describes the platform that runs it

One by one:

Functional Reliability: can the Agent complete the task? Concerns correctness, accuracy, consistency, completeness. Plainly: can this Agent get the job done?
Temporal Reliability: can the Agent stay stable over time? Concerns timeliness, responsiveness, stability, durability. Plainly: can it keep getting the job done?
Environmental Reliability: is it still effective when the environment changes? Concerns adaptability, robustness, portability. Plainly: can it still work in a different environment?
Social Reliability: this is an Agent-unique dimension. Concerns safety, trustworthiness, collaborativity, explainability. Plainly: can it collaborate safely with humans and other Agents?
Systemic Reliability: the infrastructure layer. Concerns availability, fault tolerance, scalability, security. Plainly: is the platform underneath the Agent solid?

Why this framework matters

The real value of these five dimensions is that they turn “Agent reliability” from a vague concept into a decomposable, measurable, separately-owned engineering problem. The first four describe behavioral attributes of the Agent itself; the fifth describes the platform that carries it, and that fifth dimension is exactly the landing point those of us doing AI Infra / Kubernetes should pick up.

Treat the Agent as a “stateful service”

The paper’s second core judgment is one I strongly agree with: the biggest infrastructure challenge for an Agent is not inference, it is state.

An Agent simultaneously holds Memory, Goal, Plan, Context, and tool state. Treating it as HTTP API + Model to operate is the root cause of why most demos today cannot survive production. Its true shape is closer to a composite of database + workflow engine + LLM + distributed system:

Figure 3: Wrong ops mindset (left) vs right ops mindset (right): an Agent is fundamentally a stateful distributed system

This is entirely consistent with where the industry is heading: LangGraph, Temporal, and the OpenAI Responses API are all promoting “stateful, long-running logic” to a first-class citizen. When an Agent runs for hours or even days, its state is more critical than the model parameters themselves, and far easier to lose for good on a restart. A model can be reloaded, but a three-hour planning context, once lost, is usually lost.

Fault tolerance: the “golden trio” of Agent systems

The paper provides a catalog of Agent fault-tolerance patterns:

Mechanism	Effect	Complexity
Redundancy	Standby Agent takes over at any time	High
Checkpointing	Save state for recovery	Medium
Heartbeat	Liveness detection	Low
Circuit Breaker	Isolate abnormal Agents, prevent cascading	Low
Rollback	Revert a wrong decision	Medium
Quorum	Multiple Agents vote to reach consensus	High
Self-Healing	Automatically detect and correct errors	High

Table 2: Agent fault-tolerance pattern catalog: mechanism, effect, and implementation complexity

The author values the Checkpointing + Redundancy + Heartbeat trio most, calling it the “golden trio” of Agent systems. I drew a closed loop to show how they relate; none of the three can be missing, and drop any one link and the recovery chain is broken:

Figure 4: The Agent fault-tolerance golden trio: heartbeat detects the issue → checkpointing saves the scene → redundancy switches the instance

The closed-loop logic of the golden trio

Heartbeat detects problems fast, checkpointing saves recoverable state, and redundancy takes over seamlessly on failure. Together they form the closed loop of “detect the problem → save the scene → switch the instance”. This is also why these mechanisms all sound like old friends from distributed systems: because Agent reliability is, by nature, a distributed-systems problem.

Recovery: porting traditional DR thinking onto the Agent

This section essentially ports the RTO/RPO thinking of traditional DR (disaster recovery) onto the Agent scenario. The paper compares several recovery approaches:

Recovery approach	RTO	RPO	Complexity
Cold Start	minutes to hours	High (total loss)	Low
Warm Start	seconds to minutes	Medium	Medium
Hot Start	seconds	Low (minimal loss)	High
Checkpoint Recovery	seconds to minutes	Medium (since last checkpoint)	Medium
State Reconstruction	minutes to hours	Low (fully recoverable)	High

Table 3: Agent recovery approaches compared: RTO, RPO, and implementation complexity

The author’s conclusion: the Agent fits Checkpoint Recovery best. The reason was foreshadowed above: an Agent’s state is far more important than its model parameters. Checkpoint Recovery strikes the most balanced trade-off among RTO, RPO, and implementation complexity. That is also why checkpointing sits at the center of the “golden trio” earlier.

Agent observability = infrastructure metrics + behavior analysis

This section resonates with me the most. The paper argues that traditional monitoring is far from enough: traditional monitoring watches CPU, memory, latency, and QPS, while an Agent must additionally monitor the behavior itself.

Agent Health: heartbeat, resource utilization, error rate, decision latency.
Behavioral: action frequency, decision patterns, state transitions, goal progress.
System: Agent count, communication volume, infrastructure health.

The “behavior” layer is unique to Agents.

The easiest trap to fall into

An Agent’s CPU may be low and its latency normal, but if its “action frequency” suddenly spikes or its “decision pattern” deviates from baseline, that is often the truly dangerous signal. In other words: Agent Observability = Infra Metrics + Behavior Analytics. Watching only infrastructure metrics means you will perfectly miss the Agent’s “behavioral runaway”.

This lines up completely with the thinking I discussed in GPU to Token Observability: in the Agent era, behavioral signals must become first-class citizens of observability, rather than staying stuck at the hardware / inference-metric layer.

The recommended hybrid architecture

The paper compares four architectures:

Architecture	Complexity	Scalability	Fault isolation	Suitability for Agents
Monolithic	Low	Limited	Poor	Poor
Layered	Medium	Medium	Medium	Good
Microservices	High	High	Excellent	Excellent
Event-driven	High	High	Good	Excellent

Table 4: Four architectures compared for Agent systems

The final recommendation: production-grade Agent systems adopt a hybrid architecture, the combination of “layered + microservices + event-driven”:

Figure 5: The hybrid architecture for production-grade Agent systems: layered + microservices + event-driven

Layering provides clear separation of concerns, microservices provide independent scaling and fault isolation, and event-driven provides loosely-coupled async collaboration. No single architecture can satisfy an Agent system’s demands for clarity, elasticity, and collaboration at once, so hybrid is the inevitable conclusion. For K8s veterans this layered stack should look very familiar: it is the cloud-native layering of governance, ported verbatim onto the Agent.

The biggest value of this paper, from the AI Infra perspective

From the Kubernetes / AI Infra standpoint, what this paper really conveys is a paradigm shift: Agent infrastructure will move from “Serving” to “Runtime”.

Figure 6: Paradigm shift: from model Serving to Agent Runtime, the focus shifts fully rightward

The past cared about GPU utilization, throughput, and latency; the future cares about state recovery, long-task continuity, Agent isolation, Agent scheduling, Agent observability, and Agent security governance.

My read on the trend

The next phase of AI Infra is not better model Serving, but a more reliable Agent Runtime. This is exactly the direction I have kept emphasizing in Ark Agentic Runtime, Analyzed and Agentic Runtime Realism: the Agent is moving from “a class you import” to “a workload you must govern”.

My evaluation

Strengths:

Proposes a fairly complete Agent reliability framework (the five-dimension model), turning a vague concept into a measurable engineering problem.
Systematically ports traditional SRE thinking onto the Agent scenario, with clear mappings for fault tolerance, recovery, and monitoring.
Offers strong engineering guidance on architecture selection (layered / microservices / event-driven / hybrid).

Weaknesses:

Lacks real production cases: the three cases in the paper (autonomous vehicle fleets, supply-chain optimization, customer-service bots) read more as illustrative scenarios than verifiable engineering practice.
The math models (series/parallel reliability, Markov chains, MTBF/MTTR) are essentially standard reliability-engineering textbook material, not innovation.
It never touches real Agent Runtime implementations like Kubernetes, Ray, Temporal, or LangGraph, so it feels light on engineering grounding.
Its discussion of GPU and inference systems, the core AI Infra resource layer, is shallow, barely staying at the “compute/storage/network” level of abstraction.

My score

By academic novelty: 6.5 / 10. By “giving AI Infra practitioners a mental framework for Agent reliability”: 8 / 10.

Its biggest insight is not some technical detail but that opening line: an Agent is not a single inference request but a long-running distributed state machine.

Summary

The real contribution of this paper is that it re-categorizes the reliability problem of Agentic AI. It is no longer a model problem, nor a prompt-engineering problem, but a distributed-systems problem, and specifically a distributed-systems problem that is stateful, long-running, and self-directing.

For AI Infra practitioners, this means two things. First, the traditional SRE toolbox (checkpointing, redundancy, heartbeat, circuit breaker, RTO/RPO) can be ported over directly, but it must be extended to the “behavior layer”. Second, the focus of infrastructure will shift from “how to serve models faster” to “how to run Agents more reliably”, which is the Agent Runtime.

The AI platforms of the future must not only run fast, they must run stable.

References

From GPU to Token: The 8-Layer Observability Stack for AI Infrastructure

Jimmy Song — Tue, 09 Jun 2026 03:31:16 +0000

GPU utilization is not the destination. Token cost is the true North Star metric for AI infrastructure.

For the past few years, one of the hottest topics in AI infrastructure has been GPU scheduling. Whether it’s Kubernetes, Volcano, Kueue, or HAMi, they all fundamentally solve the same problem — how to make expensive and scarce GPUs more efficiently utilized.

But as more enterprises begin running production-grade Large Language Model (LLM) services, a new phenomenon has emerged: GPU utilization is high, but users still complain about slow response times; GPU clusters are near full capacity, but business throughput hasn’t grown proportionally; VRAM and compute still have headroom, yet TTFT (Time To First Token) continues to degrade. This reveals a fundamental truth — GPU utilization alone is no longer sufficient to describe the real operational state of modern AI systems.

For traditional cloud-native applications, we focus on CPU, memory, network, and disk. For AI systems, however, we also need to care about: whether GPUs are truly performing useful computation, whether NCCL (NVIDIA Collective Communications Library) communication has become a bottleneck, whether Kubernetes has correctly allocated resources, whether KV Cache is exhausted, whether Token latency meets user experience requirements, and whether the cost per Token is reasonable. In other words, the observability target for AI infrastructure has expanded from the GPU to the entire inference chain.

Recently, while participating in the development of an industry standard — “Technical Capability Requirements for Computing Power Efficiency Enhancement: Heterogeneous Computing Services” organized by the China Academy of Information and Communications Technology (CAICT) — a core question came up repeatedly in discussions with experts: How do GPU and Token relate? How do we measure GPU output through Tokens? The standard introduces the concept of “Token as a Service,” shifting the unit of measurement for computing services from traditional GPU hours to Tokens, yet a mature practical framework for building a complete observability chain from hardware to Token was still missing.

Around the same time, I came across a technical article on GPU and LLM observability (GPU & LLM Inference Observability — Layer-by-Layer Coverage) that proposed a constructive approach: decomposing the AI system into eight observability layers from GPU hardware to business cost, which neatly fills the observability gap between GPU and Token. This article reorganizes and interprets the original content, removes product-specific implementation details and vendor promotion, and extends the model by incorporating Kubernetes, HAMi, and modern LLM inference architectures.

This article aims to answer one question:

After we’ve solved GPU scheduling, what should we observe next?

8-Layer Observability Architecture Overview

The diagram below shows the eight observability layers from GPU hardware to business cost, each corresponding to different observability targets and areas of responsibility:

Figure 1: 8-Layer AI Infrastructure Observability Architecture

Original image source GitHub.

Each layer collects data through the OTel Collector and routes it to metrics, tracing, and logging backends, forming a complete observability loop. Let’s examine each layer in detail.

L1 GPU Hardware Layer

This layer focuses on the GPU itself. The core question is: Is the GPU running healthy?

The table below lists the key observability metrics for the GPU hardware layer:

Category	Key Metrics
Compute	GPU Utilization, SM Occupancy, Tensor Core Activity
Memory	VRAM Usage, HBM Bandwidth
Interconnect	NVLink Throughput, PCIe Throughput
Reliability	ECC Error, XID Error
Thermal	Temperature, Power Draw, Throttle Reason

Table 1: L1 GPU Hardware Layer Key Metrics

Many teams only track GPU Utilization, but there’s a critical distinction — GPU Utilization ≠ GPU Efficiency. Two GPUs might both show 90% Utilization: one executing Tensor Core operations, the other waiting for memory access. The actual performance can be vastly different. Therefore, SM (Streaming Multiprocessor) Occupancy and Tensor Core Activity are often more valuable than Utilization alone.

L2 CUDA Runtime and Communication Layer

For distributed training, GPU computation is usually not the bottleneck — communication is. The core question at this layer is: Is the GPU computing, or waiting for other GPUs?

The table below lists the key metrics for the CUDA runtime and communication layer:

Category	Metrics
NCCL	AllReduce, AllGather, ReduceScatter
Communication	Duration, Bandwidth
Kernel	Execution Time, P99 Duration
Straggler	Rank Skew

Table 2: L2 CUDA Runtime and Communication Layer Key Metrics

In large-scale training scenarios, a single slow node can drag down the entire training job. By monitoring NCCL communication Duration and Bandwidth, as well as Skew between Ranks, you can quickly pinpoint communication bottlenecks.

L3 Host / OS Layer

Many GPU problems are ultimately not GPU problems. When GPU utilization is abnormal, the root cause may lie in the host’s CPU, memory, disk, or network.

The table below lists the key metrics at the host level:

Category	Metrics
CPU	Utilization, IO Wait
Memory	Usage, Swap
Disk	Throughput, Latency
Network	Retransmit, Bandwidth
Process	RSS, Thread Count

Table 3: L3 Host / OS Layer Key Metrics

A common misconception: when GPU utilization is low, the first reaction is that the GPU isn’t fast enough. In reality, the GPU is often waiting for data — a slow DataLoader, insufficient CPU, network congestion, or inadequate storage performance can all leave the GPU idle.

L4 Kubernetes and Scheduling Layer

This layer is the core of cloud-native AI infrastructure. The question to answer is: Who owns the GPU? You need to know which Pod is using the GPU, which Namespace consumes the most resources, which team has the highest cost, and which model occupies the most VRAM.

The table below lists the key observability dimensions at the scheduling layer:

Category	Key Attributes
Ownership	Pod, Container
Workload	Deployment, Job
Organization	Namespace, Team
Scheduling	GPU Sharing, GPU Partition
Topology	Node, AZ, Region

Table 4: L4 Kubernetes and Scheduling Layer Key Dimensions

Projects like HAMi, Volcano, and Kueue primarily operate at this layer. HAMi solves core problems like GPU Sharing, GPU Partitioning, and heterogeneous GPU scheduling, while observability answers the question: Has scheduling actually improved resource utilization? The observability data at this layer is the foundation for resource auditing and cost allocation.

L5 Training Runtime Layer

The training phase requires monitoring the model’s runtime state. The table below lists the key metrics for the training runtime:

Category	Metrics
Efficiency	MFU, TFLOPS, Step Time
Gradient	Norm, NaN, Inf
Loss	Training Loss
Data	DataLoader Wait
Checkpoint	Save Time, Restore Time

Table 5: L5 Training Runtime Key Metrics

The most important metric here is MFU (Model FLOPs Utilization). Many training jobs show high GPU utilization but only 30% MFU, meaning a significant amount of GPU time is not being converted into actual training progress. MFU is the golden metric for measuring training efficiency — it directly reflects the ratio of hardware compute converted into effective training computation.

L6 Inference Engine Layer

This is the most critical layer in production environments and the one seeing the fastest growth in industry attention. The table below lists the core metrics for inference engines:

Metric	Description
TTFT	Time To First Token
ITL	Inter Token Latency
Queue Wait	Request queuing time
Throughput	Token throughput
Batch Size	Batch processing efficiency
KV Cache Usage	KV Cache utilization

Table 6: L6 Inference Engine Key Metrics

Many teams still focus solely on GPU Utilization, but the true capacity metric for inference systems is often KV Cache (Key-Value Cache) Utilization.

Why KV Cache Matters More Than GPU Utilization

In modern LLM inference systems, GPU compute usually has headroom, but KV Cache often runs out first. A typical scenario: GPU utilization is only 60%, KV Cache is at 95%, new requests start queuing, and TTFT spikes rapidly.

For inference systems, KV Cache is more akin to a database’s Buffer Pool — it often determines the capacity ceiling of the entire system. When KV Cache approaches saturation, the system is forced to perform Eviction, leading to context loss, request retries, and ultimately latency spikes and throughput drops.

L7 GenAI API Layer

With the development of OpenTelemetry GenAI Semantic Conventions, the industry is converging on unified AI observability standards. The table below lists the key metrics at the API layer:

Metric	Description
Input Tokens	Input token count
Output Tokens	Output token count
Request Duration	Request latency
TTFT	First Token latency
Token Throughput	Token throughput

Table 7: L7 GenAI API Layer Key Metrics

This layer marks the shift from infrastructure-centric to application-centric observability. API-layer metrics directly face end users and business systems, serving as the core data source for SLA/SLO quality assessment.

L8 Business and Cost Layer

Ultimately, enterprises don’t care about GPU utilization — they care about: Is the GPU creating value? The table below lists the core metrics for the business and cost layer:

Metric	Description
Cost per GPU Hour	GPU hourly cost
Cost per Token	Token cost
Cost per Request	Request cost
Idle GPU Cost	Idle resource cost
Tokens per Watt	Inference efficiency

Table 8: L8 Business and Cost Layer Key Metrics

The core competitive metric for future AI infrastructure may not be GPU Utilization, but rather Cost per Useful Token — the comprehensive cost of producing one useful Token. This metric unifies hardware cost, energy consumption, inference efficiency, and business value into a single measurement framework.

Cross-Layer Troubleshooting

Real production issues often span multiple layers. The table below lists common cross-layer failure symptoms and their potential causes:

Symptom	Possible Causes
TTFT Increase	CPU IO Wait, Queue Depth, KV Cache Pressure
Throughput Drop	NCCL, Batch Size, Network
Latency Spike	KV Cache Eviction
OOM	Insufficient VRAM, Oversized Batch
Training Stall	NCCL Straggler

Table 9: Cross-Layer Troubleshooting Reference

Modern AI operations are evolving from single-point monitoring to cross-layer observability. The root cause of a single-layer metric anomaly often lies in another layer. Only by building cross-layer correlation capabilities can teams achieve rapid root-cause identification and precise troubleshooting.

From GPU Control Plane to AI Observability Plane

Over the past few years, the industry has focused primarily on projects like Kubernetes Scheduler, Volcano, Kueue, and HAMi, solving the problem of how to allocate GPUs. In the coming years, the industry will start asking whether GPUs are truly generating value, leading to the formation of a new AI infrastructure technology stack:

GPU Hardware
↓
Kubernetes Control Plane
↓
Inference Runtime
↓
Observability Plane
↓
Optimization Plane

GPU scheduling determines how resources are allocated. Observability determines whether resources are being used correctly. Together, they form the next-generation AI Infrastructure Stack. The evolution from GPU Control Plane to AI Observability Plane marks a new era where AI infrastructure transitions from “resource management” to “value management.”

Summary

AI infrastructure observability is undergoing a fundamental paradigm shift. We used to look only at GPU utilization; now we need to build a complete observability system across eight layers — from GPU hardware, CUDA runtime, host OS, Kubernetes scheduling, training runtime, inference engine, GenAI API, to business cost.

Key takeaways:

L1-L3 focus on hardware and systems: GPU health, CUDA communication efficiency, and host resource adequacy
L4 focuses on resource allocation: Kubernetes scheduling and GPU sharing, with tools like HAMi solving allocation problems
L5-L6 focus on model runtime: Training efficiency (MFU) and inference capacity (KV Cache) are the core metrics
L7-L8 focus on business value: From Token throughput to cost per Token, ultimately measuring whether GPUs are creating value

GPU scheduling is the starting point for AI infrastructure, but not the end goal. The 8-layer observability stack from GPU to Token is the critical closed loop that ensures AI infrastructure truly delivers business value.

References

GPU & LLM Inference Observability — Layer-by-Layer Coverage - GitHub

My Personal AI Stack: Building a Continuously Running Personal AI Infrastructure for ~$100/Month

Jimmy Song — Sun, 07 Jun 2026 16:58:18 +0800

A personal AI infrastructure is not a single tool — it’s a system of long-term synergy.

Figure 17: My Personal AI Stack

Many people talk about AI Agents, Second Brain, Personal Knowledge Management (PKM), and digital avatars.

But over the past year, I’ve come to realize that what I’m actually building is not some AI assistant — it’s a continuously running Personal AI Infrastructure.

It’s not a single product, nor a single model. It’s a set of systems that work together over the long term.

This system helps me think, research, write, code, manage knowledge, process emails, maintain my website, and accumulate long-term memory — every single day.

If I had to summarize it in one sentence:

ChatGPT handles thinking, OpenClaw handles execution, Obsidian handles memory, GitHub handles publishing.

From Tools to Infrastructure

Most AI workflow articles follow a similar structure: start with the model, then the plugins, then the editor.

But I increasingly feel that tools are not the point.

What matters is how these tools work together.

My work spans:

AI infrastructure research
Open source community operations
Technical writing
Developer Relations
Product and ecosystem building

Every day produces a massive amount of information:

ChatGPT conversations
Technical research
GitHub activity
Community discussions
Email newsletters
Hacker News
Discord
WeChat and Lark messages

The problem is never a lack of information — it’s how to organize it.

So I gradually built a Personal AI Stack around my own work.

Workflow Perspective

This is not a single tool, not a single model. It emphasizes continuous synergy across four layers: thinking, memory, execution, and publishing.

Overall Architecture

The architecture diagram below shows how my Personal AI Stack works in layers.

Figure 18: Personal AI Stack Architecture Layers

The core components for each layer are listed below.

Layer	Components
Interface Layer	ChatGPT, Telegram, Discord, Lark, WeChat
Reasoning Layer	ChatGPT, GLM-5.1, Claude Code, Codex
Memory Layer	Obsidian, Markdown, iCloud
Execution Layer	OpenClaw, Gmail, Calendar, Lark CLI
Publishing Layer	GitHub, Hugo, Cloudflare Pages

Table 5: Personal AI Stack Layers and Components

ChatGPT: My Thinking System

Although many workflows revolve around Agents, the tool I use most frequently is actually ChatGPT.

I primarily use it for:

Deep Research
Technical analysis
Architecture discussions
Content planning
Writing assistance
Career decisions

Rather than calling it an assistant, it’s more like:

Research Partner
Technical Advisor
Thinking Companion

Years of accumulated conversations have helped it gradually understand my background, projects, and long-term goals.

Many articles, talks, and technical judgments actually originate from these ongoing conversations.

OpenClaw: My Execution System

OpenClaw is the OpenClaw Agent I deployed on my Mac mini M4 at home.

I mainly interact with OpenClaw through Telegram.

To avoid context mixing, I use separate Telegram groups for different topics:

HAMi
AI Handbook
Personal
Work
Research
Blog

This naturally creates Context Isolation.

OpenClaw handles:

Gmail email management
Apple Calendar scheduling
Scheduled tasks
Obsidian operations
GitHub operations
Website maintenance
Lark knowledge base operations
Automated workflows

It’s more like a Chief of Staff than a chatbot.

OpenClaw’s Positioning

OpenClaw’s value lies in unifying scattered execution entry points into structured workflows, rather than being a simple chat interface.

OpenClaw’s Workflow

OpenClaw’s main path unfolds through the Telegram entry point for multi-platform execution.

Figure 19: OpenClaw Execution Path

Lark: Company Workflow Entry Point

I had barely used Lark before.

After joining my current company, which heavily promotes Lark adoption, all daily workflows, knowledge bases, and collaborative communication happen in Lark.

At first, I just treated it as an enterprise IM tool.

But looking at it now, Lark is more of a company-level workflow entry point:

Group chats
Documents
Knowledge bases
Approvals
Tasks
Automation

For me, what truly changed the experience was Lark CLI.

Through Lark CLI, I can more easily manage the company’s knowledge bases, documents, and some process-driven information.

This turns Lark from merely a chat tool into something that OpenClaw can incorporate into its automation system.

From the perspective of a Personal AI Stack, Lark serves as:

The organizational workflow layer.

It connects information, knowledge, and tasks in a company context.

Obsidian: Working Memory

Obsidian is my most frequently used knowledge tool.

But I don’t consider it my final knowledge base.

For me:

Obsidian
=
Working Memory

This is where I store:

Daily Notes
Weekly Reports
Research Notes
Inbox
Drafts
Fleeting thoughts

Starting three years ago, I developed a habit of consistently writing weekly reports.

These reports record:

Work progress
Learning content
Community activities
Project evolution
Personal reflections

In a sense, weekly reports constitute my daily memory.

And Obsidian is the carrier of these memories.

GitHub and jimmysong.io: Long-term Memory

Many people think of a blog as a content publishing platform.

But for me, jimmysong.io is closer to a long-term memory system.

This website has been continuously maintained for nearly ten years.

What’s recorded here is not just technical articles, but more importantly:

My viewpoints
My judgments
My experiences
My growth trajectory

Unlike Obsidian, content that makes it to the website typically goes through:

Research
↓
Thinking
↓
Validation
↓
Writing
↓
Revision
↓
Publishing

Therefore:

Obsidian
=
Working Memory

jimmysong.io
=
Long-term Memory

The following flow shows the basic path from Obsidian to website publishing.

Figure 20: Long-term Memory Publishing Flow

Information Flow

There are clear boundaries between content sources, curation methods, and output channels. This diagram reveals my information flow logic.

Figure 21: Personal AI Stack Information Flow

GitHub: Execution and Publishing Layer

For me, GitHub is no longer just a code repository.

It also handles:

Blog content
Website source code
Documentation system
AI Handbook
AI Native Landscape

All content eventually makes its way into GitHub.

GitHub Actions automatically handles:

Building
Testing
Publishing

Finally, everything is served through Cloudflare Pages.

The diagram below shows the publishing path from Markdown to static site.

Figure 22: GitHub Publishing Pipeline

My Development Toolchain

I currently use three main AI development tools.

Claude Code

My daily development workhorse.

Despite the name Claude Code, I primarily use Zhipu’s GLM-5.1 model.

It handles:

Coding
Refactoring
Debugging
Documentation maintenance

Codex

Mainly used for:

Project initialization
Large-scale code generation
Automated execution of complex tasks

OpenClaw

Handles automation work beyond development.

The three form a clear division of labor.

The diagram below shows how my toolchain collaborates.

Figure 23: Development Toolchain Collaboration

Why Not Claude?

This is one of the questions I get asked most often.

Objectively speaking, Claude is indeed very strong in code generation and code understanding.

I seriously considered using Claude as my primary model.

But ultimately, I chose not to.

The reason is not about model capability — it’s about overall return on investment.

First, there’s the account issue.

I’ve registered Claude accounts multiple times in the past, and each time I ran into restrictions and bans.

Second, there’s the cost issue.

Because I pay for all my AI tools out of pocket.

So I care more about:

Capability / Cost

Rather than:

Absolute Capability

For enterprise users, Claude Max might be a very reasonable choice.

But for individual paying users, the conclusion may be different.

This article discusses a Personal AI Stack that is entirely self-funded.

If the company reimburses expenses, or if you have an enterprise budget, many choices would change.

Why Not Self-host Large Models?

This is another frequently asked question.

Many people believe:

Buy a GPU + Open-source Model = Free AI

In reality, this is often not the case.

If the goal is simply to get a stable, powerful AI assistant, I lean toward:

Subscription > Self-hosting

The reasons include:

Time Cost

Maintaining a model is work in itself.

Including:

CUDA
Drivers
Inference frameworks
Model upgrades
Networking issues

All of these require time.

And I’d rather spend my time on:

Writing
Community
Product
Technical research

Cost Issue

Currently:

ChatGPT Plus
GLM Coding Plan Max

Total monthly cost is under 600 CNY.

While a high-end GPU:

RTX 5090D
RTX PRO
Enterprise GPU

Typically costs tens of thousands of CNY.

Plus:

Electricity
Depreciation
Maintenance

For my use case, it’s simply not worth it.

Model Upgrade Speed

Cloud models upgrade every month.

Local models require manual follow-up.

For knowledge workers:

Using the latest model is more important than owning a model.

Subscription vs. Self-hosting

For individual users, subscription services typically offer advantages in time cost, upgrade speed, and stability. They let you focus on results rather than infrastructure maintenance.

Daily Tools

Beyond the core systems described above, I also rely on some daily tools to complete the workflow.

Atlas Browser: My primary browser for web reading, research, and information gathering. Noteworthy content is saved to my knowledge base via Obsidian Clipper.
Warp: My most frequently used terminal tool. Its modern interactive experience and AI capabilities make command-line work more efficient.
Typora: A Markdown editor I’ve used for a long time, ideal for immersive writing and long-form editing. Many blog posts and documents are completed here.
CodexBar: Used to monitor usage of ChatGPT, Codex, Claude Code, and other tools. For heavy AI users, token consumption has become a resource metric worth tracking.
Sogou Input Method: My primary voice input tool. Compared to keyboard input, voice better matches my thinking habits, especially when working remotely, writing, and communicating with AI.

These tools are not the core of the system themselves, but they form the foundational experience layer of the entire Personal AI Stack, making information acquisition, content creation, and daily development smoother.

My AI Usage Scale

Current approximate consumption:

GLM-5.1

Weekly:

600 million Tokens

Monthly:

2.4 billion Tokens

Codex

Weekly:

200 million Tokens

Monthly:

800 million Tokens

Total:

Approximately 3.2 billion Tokens per month

That’s roughly 100 million Tokens consumed per day. Through ChatGPT Plus and GLM Coding Plan subscriptions, this is much more cost-effective than paying per token — otherwise, these tokens would cost at least $500 per month.

How Much Does This System Cost Per Month?

The table below compares the fixed monthly costs.

Service	Monthly Cost
ChatGPT Plus	$19.99
GLM Coding Plan Max	¥422.10
iCloud+	$0.99

Table 6: Monthly Cost Breakdown

Approximately:

¥573 / month

OpenClaw runs on my Mac mini M4.

Hardware includes:

Mac mini M4
Samsung 990 Pro 1TB
HAGIBIS Dock

Total investment:

¥4,469

Amortized over four years:

Approximately ¥100 / month

Total Cost

The entire Personal AI Stack’s fixed cost is approximately:

¥700 CNY / month (~$100)

This cost pie chart shows the monthly spending structure of the Personal AI Stack.

Figure 24: Personal AI Stack Monthly Cost Breakdown

FAQ

Is This the Most Powerful Setup?

No.

This is not a “most powerful AI tool configuration guide.”

This is a completely self-funded Personal AI Stack designed for long-term individual use.

If your company reimburses expenses, or if you have a higher budget, you could choose Claude Max, Cursor, more API services, or even a local GPU workstation.

But my goal is not to pursue the absolute best — it’s to achieve stable, sustainable, and cumulative productivity within a personal budget.

Why Not Sync All ChatGPT Conversations to Obsidian?

Because I don’t want to turn Obsidian into a chat log repository.

What I care about more is:

Which content is worth preserving long-term?

Manual saving is itself a curation process.

That’s more important than automatically syncing everything.

Why Not Use Notion?

It’s not because Notion isn’t good.

It’s because I prefer Markdown First.

The benefits of Markdown include:

Local-first
Version controllable
Portable
AI-friendly
Suitable for long-term preservation

Why Use Telegram Only for OpenClaw?

Because Telegram’s Bot ecosystem and API are better suited as an Agent entry point.

WeChat, Lark, and Discord are more for person-to-person communication and community collaboration for me.

Why Emphasize Lark?

Because after joining my current company, I truly started using Lark.

The company’s entire workflow revolves around Lark.

For me, Lark isn’t a personal knowledge base — it’s a company knowledge and collaboration system.

Through Lark CLI, it can further become a workflow entry point that OpenClaw can operate.

Why Not Use a Local AI Workstation?

Because my main work is not training models or running inference services.

My main work is:

Thinking
Research
Writing
Open source community
Software development

Subscription models + a Mac mini is more than enough.

A local AI workstation requires significant investment, complex maintenance, and the model upgrade speed may not keep up with cloud services.

Why Not Buy a High-end NVIDIA GPU?

If the primary purpose is learning CUDA, GPU scheduling, or AI infrastructure, buying a GPU has value.

But if the primary purpose is daily productivity, a high-end GPU isn’t necessarily cost-effective.

For me:

Subscribing to models is more important than owning a GPU.

Workflow Matters More Than Models

My biggest takeaway from the past year is:

Many people believe the core of AI is the model.

But my experience suggests the opposite.

What truly affects productivity is often not the model leaderboard, but workflow design.

A 95-score model in an excellent workflow is usually more valuable than a 100-score model in a chaotic workflow.

In a sense, this is very similar to the evolution of the cloud-native world.

GPUs are important, but scheduling systems are equally important.

Models are important, but workflows are equally important.

Agents are important, but long-term memory and execution systems are equally important.

For me, the ultimate goal of the Personal AI Stack is not to replace people.

It’s to connect thinking, memory, and execution — freeing up more time for what truly matters.

Conclusion

This article defines my Personal AI Stack as a long-term runnable infrastructure, rather than a single tool or a single model.

The core conclusion is: in a self-funded scenario, stable workflows, continuous memory systems, and executable automation are more productive than chasing the most powerful model.

If your goal is to accumulate long-term value, investing time in “how things work together” often yields better returns than investing time in “model rankings.”

Token Is More Than a Billing Unit, It's Becoming the Resource Unit of the AI Era

Jimmy Song — Thu, 04 Jun 2026 06:42:40 +0000

The Linux Foundation recently announced plans to establish the Tokenomics Foundation.

Figure 1: Tokenomics Foundation

When I first saw the news, my immediate reaction wasn’t:

Yet another foundation.

It was:

AI infrastructure is shifting from “managing GPUs” to “managing Tokens.”

That shift may matter more than the foundation itself.

The official rationale from the Linux Foundation is straightforward:

As enterprises begin deploying generative AI and agents at scale, the Token has become the new unit of technology spend. The foundation will partner with the FinOps Foundation to establish Token cost management, benchmarks, open standards, and best practices.

If you stop there, most people would reduce it to:

FinOps for AI

Or:

AI cost management

But I think there’s more to it than that.

The Cloud Era Managed Resources

For the past two decades, the infrastructure industry has been managing resources.

We discussed:

CPU
Memory
Storage
Network
GPU

Kubernetes was no different.

Whether it was the scheduler, autoscaling, or resource quotas, everything fundamentally answered one question:

How do we allocate resources?

That was the central question of the cloud era.

The AI Era Starts Managing Outcomes

With the rise of AI, an interesting shift began.

Enterprises increasingly don’t care about:

How many GPUs were used

They care far more about:

How many Tokens were produced

For most enterprises:

GPU is cost.
Token is output.
GPU, CPU, and memory are merely means of production.
Token is the final deliverable.

J.R. Storment (former Executive Director of the FinOps Foundation) said something on LinkedIn that stuck with me:

Tokens are what all the hardware is being built to produce.

All hardware is ultimately producing Tokens.

From this perspective:

GPU
 ↓
Inference
 ↓
Token

A new value chain is forming.

Token Could Become the New Resource Model

Over the past few years, I’ve been tracking how Kubernetes is evolving in the AI era.

More and more signals suggest:

Kubernetes is evolving from a Compute Control Plane into an AI Control Plane.

Once AI becomes the dominant workload, the resources we manage may no longer be just:

cpu: 8
memory: 32Gi
gpu: 1

They may gradually become:

token
context
latency
throughput

Metrics much closer to business value.

The most interesting thing about the Tokenomics Foundation isn’t whether it will define new standards.

It’s that it implicitly acknowledges one thing:

Token has started to become a resource.

Just like CPU in the cloud era.

Implications for AI Infrastructure

For teams building AI infrastructure, this shift deserves serious thought.

Today, many projects (including HAMi) focus on:

GPU utilization
GPU sharing
GPU scheduling
GPU virtualization

These are all important.

But from an enterprise perspective, they ultimately don’t buy GPU utilization.

They buy:

Cost Per Token

Or even further:

Value Per Token

Given the same pool of GPUs:

Whoever can produce more Tokens,
Whoever can reduce the cost per million Tokens,
Whoever can increase the business value of each Token,

Will capture greater commercial value.

That’s why I believe the significance of the Tokenomics Foundation lies beyond the standards themselves.

It’s driving the entire industry from:

Resource management

Toward:

Value management

My Take

I don’t think the Tokenomics Foundation will become the next Kubernetes.

It’s more like the FinOps Foundation, OpenCost, or OpenTelemetry.

What it’s trying to define isn’t software.

It’s the metering system for the AI era.

The past decade:

CPU was the language of infrastructure

The next decade:

Token may become the language of AI infrastructure

If this trend holds, then the GPUs, inference frameworks, scheduling systems, and Agent Runtimes we discuss today are ultimately just parts of a larger Token economy.

And that may be the real signal the Linux Foundation is sending by launching the Tokenomics Foundation.

AI Native Landscape Launches as a Standalone Site

Jimmy Song — Thu, 04 Jun 2026 06:12:10 +0000

AI Native Landscape now has its own home. 600+ curated open-source projects, a brand-new standalone site, and direct search from your AI coding tools.

How It Started

Last July, I started collecting and organizing AI open-source projects on my personal website. The AI space was moving incredibly fast, with new projects popping up every day. GitHub is a treasure trove, full of tools and frameworks worth studying and using. I wanted to organize them systematically so others (and myself) could find the truly useful ones.

At first, everything lived on jimmysong.io. But as the catalog grew, maintaining it became a pain. The main site already had plenty of content, and AI-related pages mixed in meant that updating a single project required rebuilding the entire site. So this year I decided to spin it out as a standalone open-source project: ai-native-landscape.

All pages previously at jimmysong.io/en/ai/* now redirect to the new site at landscape.jimmysong.io. Existing links and bookmarks still work.

Not Just a List, But a Scored One

There are plenty of AI project directories out there, but most just list a name and a link and call it a day. That’s not enough.

GitHub has thousands upon thousands of projects. You see a name, click through, and find out the last commit was six months ago and nobody’s responding to Issues. You just spent time researching something that’s essentially abandoned.

That’s why AI Native Landscape does something different: every project gets a score.

The scoring is based on four dimensions:

Activity: commit frequency, release cadence, Issue response time
Community: contributor count, PR merge speed, discussion engagement
Quality: Stars, Forks, dependency relationships
Sustainability: maintenance history, team stability

These four dimensions combine into an overall health score. At a glance, you can tell whether a project is worth your time.

And the scores update daily, automatically. Today’s score might differ from tomorrow’s.

The project list itself is also human-curated. Projects that have gone inactive don’t make the cut. I want every project on this list to be “alive,” so the time you spend studying them won’t be wasted.

What’s Covered

The catalog spans eight major areas of AI:

Category	Description
Agent Systems	Frameworks, orchestration, and workflows
Knowledge & Context	RAG, vector databases, document processing, knowledge graphs
Models & Modalities	Foundation models, toolkits, speech and vision generation
Inference & Runtime	Model serving, inference engines, sandboxes, GPU acceleration
Training, Evaluation & Optimization	Frameworks, fine-tuning, benchmarks, observability
Developer Tooling	MCP protocols, coding agents, IDE tools, SDKs
Applications & Experience	Chat interfaces, workflow automation, low-code platforms
Platform & Infrastructure	Cloud-native AI, data platforms, security and operations

Currently 600+ open-source projects, each with bilingual descriptions in English and Chinese.

AI Skill Search

The landscape supports direct search from AI coding tools. No browser needed. Install with one command:

npx landscape-search

Works with Claude Code, Cursor, GitHub Copilot, OpenAI Codex, Cline, Aider, and other popular AI coding tools. After installation, just search in natural language:

“Find me an MCP-compatible agent framework”

No API key required. No local data files. Your AI tool fetches the live project catalog and returns ranked results.

Tech Stack

Built with Astro, deployed on Cloudflare Pages, with TypeScript for type safety. Project data lives in bilingual Markdown files. The build pipeline validates, indexes, and generates OG images in one pass.

Get Involved

GitHub is everyone’s treasure trove, and I hope this project can grow with the community’s help.

Star the repo: github.com/rootsongjc/ai-native-landscape

Submit a project: If you know an AI open-source project worth listing:

Open an Issue on GitHub
Or submit a PR directly, add bilingual Markdown files under data/projects/

See the contributing guide for details.

Links:

Live site: landscape.jimmysong.io
GitHub repo: rootsongjc/ai-native-landscape
Skill install: npx landscape-search

Kubernetes as the GPU Control Plane: HAMi v2.9 and Next-Gen AI Infra

Jimmy Song — Thu, 14 May 2026 06:34:19 +0000

Recently, I’ve been following the progress of domestic GPU scheduling and Kubernetes AI resource models. With the release of HAMi v2.9, I want to share several observations on how the AI Infra control plane is evolving.

Figure 1: Kubernetes as the GPU Control Plane for AI

Why Discuss This Topic Now

When DeepSeek R1 was released in early 2025, most people focused on the fact that it trained a model competitive with OpenAI o1 for just $5.6 million. What struck me, however, was that as inference costs plummeted, GPU utilization issues would quickly come to the forefront.

As models became more useful and inference demand exploded, “one GPU per model” rapidly became a luxury. Meanwhile, the NVIDIA H200 export saga accelerated the adoption of domestic compute. First, a sales ban; then, at the end of 2025, a 25% tariff under Trump; and by January 2026, Chinese customs had cleared zero units. Policy now mandates that over 40% of data center chips must be domestically produced by 2026.

The reality is harsh: not only are GPUs scarce, but you must also learn to use NVIDIA, Ascend, Cambricon, Hygon, and other very different platforms simultaneously.

That’s why I believe HAMi v2.9 is more significant than it appears on the surface.

GPUs Are No Longer Just About the Card

Kubernetes has always managed GPUs in a rather crude way:

resources:
 limits:
 nvidia.com/gpu: 1

This was sufficient in 2019, when the main question was simply whether a Pod needed a GPU or not.

But that’s no longer enough. An inference service might only need 4GB of VRAM, multiple small models can share a single card, training jobs care about GPU topology and interconnect bandwidth, and multi-tenancy requires fault domain isolation. Treating GPUs as integer resources is like using an abacus for statistical analysis—not impossible, but the mental model is out of sync with reality.

The most notable feature in HAMi v2.9 is the HAMi-core mode for the Ascend 910C. Previously, sharing Ascend cards relied on SR-IOV hardware virtualization, which was coarse-grained and inflexible. HAMi-core takes a different approach: it uses LD_PRELOAD to intercept ACL calls in user space, enabling memory isolation at the MB level and compute throttling by percentage.

In short: it’s managed by software, not hardware slicing.

This is reminiscent of how SDN abstracted the network control plane from hardware devices—GPU partitioning is shifting from a hardware capability to a cluster control plane capability. Considering that Huawei shipped 810,000 Ascend 910C cards last year—nearly half of all domestic chips—this capability has significant real-world impact.

DRA: Kubernetes Finally Has a Robust Device Resource Model

Kubernetes v1.34 (September 2025) officially promoted DRA (Dynamic Resource Allocation) to GA, and Red Hat OpenShift 4.21 followed suit. This is a big deal.

The Device Plugin solved “how to connect GPUs to K8s,” but not “how to express complex AI resource requirements.” Device Plugins only know how many cards are on a node, not how much VRAM you need, what topology, or what isolation level.

DRA standardizes device resource declaration, allocation, and management via ResourceClaim and DeviceClass. HAMi-DRA takes a pragmatic approach: it doesn’t require users to change how they declare resources. Instead, it uses a Mutating Webhook to automatically convert existing Device Plugin-style declarations into the DRA model. Legacy systems don’t need to change, but can still leverage new capabilities.

I liken this to what CSI did for storage: it didn’t eliminate vendor differences, but allowed Kubernetes to consume different storage capabilities in a unified way. DRA does the same for AI accelerators—NVIDIA, Ascend, AMD, Vastai cards will never be identical, but the scheduling layer should speak a common language.

A Complete Control Plane Path

If we look beyond individual features and consider HAMi-core, DRA, CDI, and the scheduler together, they actually correspond to different layers of GPU resource management:

HAMi-core: How to partition and isolate devices internally
DRA: How to declare, allocate, and bind resources
CDI: How to standardize device injection into container runtimes
Scheduler/Webhook: How to schedule, admit, and observe

Connecting these layers, from top to bottom, forms the complete Kubernetes GPU Control Plane:

Figure 2: Kubernetes as the GPU Control Plane for AI Workloads

This is a complete control plane path. In v2.9, Volcano vGPU was upgraded to v0.19 with enhanced CDI support. While this may seem like a minor improvement in device injection, it actually completes a critical link in this chain.

Heterogeneity Is the Main Battlefield for Domestic AI Infra

The reality for domestic AI clusters: you can’t build infrastructure around just one type of GPU.

Enterprise environments often have NVIDIA, Ascend, Biren, Cambricon, Hygon, Muxi, Kunlunxin, Vastai, and other devices coexisting. Each card has different drivers, runtimes, virtualization capabilities, and monitoring methods. HAMi v2.9 adds support for Vastai, covering more than ten types of heterogeneous compute devices. Mixed training and inference, online and offline workloads, domestic and overseas GPUs, multi-team and multi-tenant resource pools—in these scenarios, unified scheduling is far more important than single-card performance.

Key Judgments

GPU sharing will shift from a cost-saving measure to a default requirement. After the explosion of inference workloads, not every workload deserves exclusive access to an entire card. Exclusive allocation will increasingly become a luxury.

DRA is the way forward, but migration will be gradual. The Device Plugin ecosystem is too large to disappear overnight. HAMi-DRA’s compatibility layer shows the project team understands this.

Heterogeneous scheduling will become the core challenge for AI Infra. Whoever can abstract different vendor devices into a unified scheduling language will control the key position in the control plane.

Kubernetes will be reshaped by AI workloads. From scheduling semantics to resource models, AI requires much greater expressiveness than traditional web services. DRA, CDI, topology-aware scheduling—these are not isolated evolutions, but all point to one thing: Kubernetes is evolving from a container orchestrator to the control plane for AI computing.

Summary

The significance of HAMi v2.9 is not just in supporting a particular device or partitioning method, but in making one thing clear: the next generation of AI infrastructure competition is not just about model frameworks or GPU counts, but about the control plane.

GPUs are shifting from external devices on nodes to native resources within the Kubernetes control plane. Whoever defines the resource model for the AI era will define the long-term boundaries of AI Infra.

Kubernetes's Anxiety and Rebirth in the AI Wave

Jimmy Song — Fri, 03 Apr 2026 05:20:28 +0000

Kubernetes hasn’t been replaced by AI, but it’s being redefined by it. Anxiety is the prelude to rebirth.

After attending KubeCon EU 2026 in Amsterdam, I’ve been pondering a key question: Kubernetes isn’t obsolete, but it’s no longer “enough”; it hasn’t been replaced by AI, but it’s being redefined by AI.

Figure 1: KubeCon EU 2026 slogan: Keep Cloud Native Moving. This event had over 13,000 registrations, making it the largest KubeCon to date.

This was my third time attending KubeCon in Europe. Over the past few years, you can actually see the community’s mindset shift through the event slogans:

2024 Paris: La vie en Cloud Native

→ Cloud Native has become a “way of life,” the default state
2025 London: No slogan, just the 10th anniversary

→ Kubernetes reached a milestone, focusing on retrospection rather than moving forward
2026 Amsterdam: Keep Cloud Native Moving

→ But the question is: where is it moving?

The absence of a slogan in 2025 was a signal in itself:

When an ecosystem starts commemorating the past instead of defining the future, it’s already at an inflection point.

This article doesn’t recap the talks, but instead distills my observations at KubeCon into insights about Kubernetes’ anxiety and rebirth in the AI wave.

The Root of Anxiety: Is Kubernetes Facing a “Crisis”?

The biggest change at KubeCon was that AI has completely replaced traditional cloud native topics. The focus shifted from service optimization and microservices management to how to deploy and manage AI workloads on Kubernetes, especially inference tasks and GPU scheduling.

Figure 2: Before KubeCon officially started, the Maintainer Summit was all about AI.

Kubernetes, as the foundational infrastructure, was once the core of the cloud native world. With the explosive growth of AI models, the question now is whether Kubernetes can still serve as a “universal” platform for everything, which has become a new source of anxiety.

The AI boom brings real challenges: Can Kubernetes’ “universality” adapt to the complexity of AI workloads?

The Focus Brought by the AI Boom

AI’s popularity has shifted the cloud native spotlight entirely to artificial intelligence. AI coding, OpenClaw, large language models, and generative models have all drawn widespread attention. AI has become the core computing demand in the real world.

This surge in demand raises the question: Can Kubernetes continue to serve as the infrastructure platform for complex tasks? Especially with issues like GPU sharing, inference model scheduling, VRAM allocation, and device attribute selection, is the traditional Kubernetes resource model sufficient?

In the past, Kubernetes handled compute, storage, and networking as foundational infrastructure. But with the rapid development of AI, its “universality” is being challenged. Particularly for inference tasks, Kubernetes’ model appears thin.

Comparing with OpenStack: Will Kubernetes Repeat History?

OpenStack once aimed to be a complete open-source cloud platform, but ultimately failed to sustain growth due to complexity and a lack of flexibility in adapting to new technologies.

Will Kubernetes follow the same path? I believe Kubernetes has different strengths: as a container and microservices orchestration platform, it’s widely adopted and has strong community and vendor support. It doesn’t try to replace all cloud provider capabilities but serves as an infrastructure control plane to help users manage resources.

Figure 3: Cloud native contributors remain active. The crowd at the KubeCon EU 2026 Maintainer Summit shows the community’s vitality.

However, as AI workloads become mainstream, Kubernetes must find a new position to avoid being replaced by “AI-optimized platforms.”

Kubernetes’ Challenge: The GPU Resource Management Gap

At KubeCon, NVIDIA announced the donation of the GPU DRA (Dynamic Resource Allocation) driver to the CNCF, marking the upstreaming of GPU resource management. GPU sharing and scheduling have become urgent issues for Kubernetes.

Traditionally, Kubernetes relied on the Device Plugin model to schedule GPUs, only supporting allocation by device count (e.g., nvidia.com/gpu: 1). But for AI inference tasks, more information is needed for resource scheduling, such as VRAM size, GPU topology, and sharing strategies. NVIDIA DRA makes GPU resource management more flexible and intelligent, gradually easing the “GPU resource crunch” in AI workloads.

This shift means Kubernetes is no longer just a “container orchestration platform,” but is becoming the infrastructure layer for AI-specific resource scheduling.

Against this backdrop, both the community and industry are exploring finer-grained GPU resource abstraction and scheduling mechanisms. For example, the open-source project HAMi is building a GPU resource management layer for AI workloads on top of Kubernetes, supporting GPU sharing, VRAM-level allocation, and heterogeneous device scheduling.

Figure 4: HAMi demo at KubeCon EU 2026 Keynote

These efforts are not about replacing Kubernetes, but about filling the resource model gaps for the AI era. In the long run, this layer may evolve into a “GPU Abstraction Layer” similar to CNI/CSI, becoming a key part of AI-native infrastructure.

The Production “Gap”: Many AI PoCs, Few in Production

A common post-event summary was: Many PoCs, but “everyday production deployments” are still rare. Pulumi summarized it as:

lots of working demos, very few production setups people trust

This shows that while many AI workload solutions succeed in technical demos, the transition from experimentation to production remains difficult. Whether it’s GPU resource sharing or inference request scheduling, whether Kubernetes as the foundation can support this transformation is still an open question.

The Rise of Inference Systems: Kubernetes’ Scheduling Boundaries Are Challenged

Another major event at this KubeCon was llm-d being contributed to the CNCF as a Sandbox project.

If GPU DRA represents the upstreaming of device resource models, then llm-d represents another critical evolution: Distributed LLM inference capabilities are moving from proprietary engineering implementations to standardized, community-driven collaboration in cloud native.

This is significant not just because it’s another open-source project, but because it shows that Kubernetes’ challenges in the AI era are no longer just about “how to schedule GPUs,” but also “how to host inference systems themselves.” As prefill/decode separation, request routing, KV cache management, and throughput optimization move into the infrastructure layer, Kubernetes’ boundaries are being redefined.

Traditionally, the Kubernetes scheduler focused on Pod scheduling. But in AI inference scenarios, scheduling is not just about picking a node—it’s about selecting the most suitable inference instance based on request characteristics. Factors like model state, request queue depth, and cache hit rate all need to be considered. This process is increasingly managed by inference runtimes, forming new “request-level scheduling” systems.

This leads to an overlap between the Kubernetes scheduler and inference systems, forcing Kubernetes to rethink its role: should it keep expanding, or collaborate with inference systems?

AI-Native Infrastructure: The Key Challenge for Production

At the AI Native Summit, the real needs for AI-native infrastructure were especially clear. The focus was no longer “can it run on Kubernetes,” but how to make AI workloads routine, stable, and production-ready on Kubernetes.

Figure 5: At the AI Native Summit after KubeCon, Linux Foundation Chairman Jonathan said cloud native is entering the AI-native era.

The core challenge is delivery. Unlike traditional apps, AI model weights are often huge—tens of GB or even TB—making model delivery and data management extremely complex. Traditional container delivery systems (like image layers) struggle with such massive data and complex versioning.

A key direction for Kubernetes is to standardize model weight and data delivery, using ImageVolume and OCI artifacts to solve AI model delivery and version management on Kubernetes. This not only reduces “cold start” times but also provides infrastructure support for multi-tenancy and compliance.

Summary

Kubernetes won’t be replaced by AI, but it’s being reshaped as the core of infrastructure. This anxiety is the force driving its evolution—it’s moving from a “general-purpose infrastructure platform” to an “AI-powered multifunctional base”. Some even call it the AI operating system.

In the future, Kubernetes’ core competitiveness will no longer be just container management, but how effectively it can schedule and manage AI workloads, and how it can make AI a routine part of operations. This was my biggest takeaway from the AI Native Summit and KubeCon, and it’s what I look forward to in the Kubernetes ecosystem over the next few years.

References

Day One in Amsterdam: Kubernetes Is Rethinking AI

Jimmy Song — Sun, 22 Mar 2026 20:41:19 +0000

Today marks my first day at KubeCon Europe 2026. The most striking feeling is: the world is vast, but this community is truly small.

Figure 11: Jimmy on the first day of KubeCon EU 2026

One strong impression stands out:

The world is big, but this circle is really small.

Old Friends, New Cycle

At the Maintainer Summit, I met many familiar faces—

Colleagues from Ant Group, friends from Tetrate, and some people I’ve known for nearly a decade. Together, we’ve journeyed from the early days of Kubernetes, Service Mesh, and cloud native infrastructure to today.

In a sense, this generation has fully experienced:

The rise of Kubernetes
The standardization of Cloud Native
The microservices and service mesh boom
And now, the era of AI Infrastructure

This isn’t about “new people entering the field,” but rather—

The same group stepping into a new technology cycle.

What Is the Maintainer Summit Discussing?

If you ask:

What is the Kubernetes community most concerned about right now?

Today’s answer is very clear:

👉 How to run AI workloads better on Kubernetes

Figure 12: The Maintainer Summit’s main topic is AI Infra

Many topics at the Maintainer Summit revolved around:

Scheduling models for LLM / AI workloads
GPU / accelerator resource management
Integrating inference systems with Kubernetes
Redefining the roles of data plane vs. control plane
How observability tools like OTel monitor AI workloads

In other words:

Kubernetes hasn’t been replaced by AI; it’s actively “absorbing” AI.

Key Signal: GPUs Are Becoming the “Infrastructure Layer”

Today, I had an in-depth discussion with CNCF TOC, Red Hat, and the vLLM community.

The core question was:

How should GPUs be “platformized”?

Some consensus is already clear:

GPUs are no longer just devices
They are now a schedulable, partitionable, and shareable resource layer

Figure 13: TOC meeting discussing GPU resource management and LLM Serving integration

At the Maintainer Summit in Amsterdam, we had deep discussions with CNCF TOC, Red Hat, and the vLLM community about GPU resource management and LLM Serving integration in Kubernetes scenarios, and explored potential collaboration between vLLM and HAMi.

Behind this is a major paradigm shift:

Past	Now
GPU = Node resource	GPU = Infrastructure layer
Exclusive use	Multi-tenant sharing
Static binding	Dynamic scheduling
Managed within frameworks	Unified management at the platform layer

This is exactly what we’ve been working on in HAMi.

HAMi: From “Project” to “Reference Pattern”

Another interesting change today:

HAMi is no longer just a “community project”—it’s becoming:

A reference implementation (reference pattern) for AI Infra

Figure 14: Li Mengxuan, CTO of Dynamia, sharing HAMi’s design and practice at KubeCon EU 2026 Maintainer Summit

This is reflected in several ways:

Invited to present at the Maintainer Summit
Participating in CNCF TOC discussions
Involved in incubating review demos
Exploring joint content with the vLLM community (even discussing a joint blog 👀)

Especially in conversations with Red Hat and vLLM, a clear trend emerged:

GPU resource management and LLM serving are becoming coupled

That is:

Upper layer: vLLM / inference frameworks
Lower layer: GPU scheduling / sharing

A new “interface layer” is gradually forming.

This is a direction worth betting on.

Figure 15: At the TAG Workshop, HAMi was discussed as an Incubating demo

A Caution: The AI Infra Startup Boom Hasn’t Really Begun

At the same time, I have a somewhat “counterintuitive” observation:

We haven’t yet seen a large wave of AI Infra (K8s-focused) startups.

Most companies I saw today:

Many are pivoting from CI/CD, Service Mesh, or Gateway
Many are traditional cloud vendors extending into AI
Many are working on models, agents, or even lower-level tech

But those truly focused on:

“Making AI workloads run better on Kubernetes”

There are actually not many startups at this layer.

This could mean two things:

1) This Layer Isn’t Fully Formed Yet

Currently, most activity is at:

The model layer (LLM / foundation models)
The application layer (Agent / Copilot)

But not at:

The scheduling layer
The resource layer
The runtime layer

2) Or, the Barrier to Entry Is Very High

Because at its core, this is:

The intersection of Cloud Native × GPU × AI workload

It’s not just “wrapping AI,” but a fundamental re-architecture at the infrastructure level.

My Take

If we break down the AI technology stack:

Agent / Application
 ↓
LLM Serving (vLLM, etc.)
 ↓
AI Runtime / Scheduling
 ↓
GPU Resource Layer
 ↓
Hardware

Most innovation today is concentrated in:

The top two layers (Agent / LLM)

But the real long-term moat lies in:

The middle two layers (Runtime + Resource Layer)

And Kubernetes is very likely to remain:

The default platform for this middle layer

Summary

Today’s takeaway:

Kubernetes is not obsolete; it’s being redefined.

And our generation is shifting from:

“Cloud Native Builders”

to:

“AI Infrastructure Builders”

More to come tomorrow.

HAMi Website Refactor: Why HAMi Docs and Website Underwent a Complete Redesign

Jimmy Song — Tue, 17 Mar 2026 08:55:52 +0800

This redesign is more than a style update—it’s a step toward clearer technical communication and better user experience. Try the new HAMi website at https://project-hami.io and submit issues here.

Over the past two months, I conducted a thorough refactor of the documentation website (see GitHub). Externally, it looks like a “visual redesign”, but from the perspective of community maintainers and content builders, it’s a comprehensive upgrade of information architecture, content system, and frontend experience.

This article aims to systematically explain three things: why we did this refactor, what exactly changed, and what these changes mean for the HAMi community.

Why Refactor the Website and Documentation

HAMi is a CNCF-hosted open source project initiated and contributed by Dynamia, with growing influence in GPU virtualization, heterogeneous compute scheduling, and AI infrastructure. The community content is expanding, and user types are becoming more diverse: from first-time visitors to engineers and enterprise users seeking deployment docs, architecture diagrams, case studies, and ecosystem information.

The original site was functional, but as content grew, several issues became apparent:

The homepage lacked information density, making it hard to quickly grasp the project’s overall value.
Connections between docs, blogs, and community info were not smooth; content entry points were scattered.
Search experience was unstable; external solutions were not ideal in practice.
Mobile experience had many details needing improvement, especially navigation, card layouts, and footer areas.
Visual style was inconsistent, making it hard to convey community influence and engineering maturity.

For a fast-evolving open source community, the website is not just a “place for docs”, but the public interface of the community. It needs to serve as project introduction, knowledge gateway, adoption proof, community connector, and brand expression.

So the goal of this refactor was clear: not just superficial beautification, but to truly upgrade the website into HAMi’s systematic community entry point.

What Was Done in This Refactor

This update was not a single-point change, but a series of systematic improvements.

Homepage Redesign and Complete Information Architecture Overhaul

The most obvious change is the homepage.

We redesigned the homepage structure, moving away from simply stacking content blocks, and instead organizing the page around the main narrative: “Project Positioning → Core Capabilities → Ecosystem Entry → Content Accumulation → Community Trust”.

Specifically, the homepage received several key upgrades:

Rebuilt the Hero section to strengthen first-screen information delivery and action entry.
Optimized CTA design so users can quickly access docs, blogs, and resources.
Added and enhanced multiple homepage sections to showcase project value and community reach in a more structured way.
Adjusted visual hierarchy, background atmosphere, and scroll rhythm, transforming the homepage from a “content list” into a “narrative page”.

These changes include Hero animations and atmosphere layers, research/story sections, new resource entry sections, refreshed CTAs, unified background design, and ongoing reduction of visual noise. Together, they solve a core problem: enabling visitors to understand what HAMi is and why it’s worth exploring further within seconds.

Architecture Diagrams

Key diagrams were redrawn for clearer technical communication. This helps users grasp HAMi’s role in AI infrastructure.

Figure 1: HAMi website homepage architecture diagram

For HAMi, this change is critical. The community faces not just a single feature, but a set of system-level challenges involving Kubernetes, schedulers, GPU Operators, heterogeneous devices, and enterprise platforms. Improved diagrams make the website a better technical entry point.

Added Case Studies, Community, and Ecosystem Sections to Make Impact Visible

Another important direction was strengthening the “community proof” layer.

Many open source project sites fall into the trap of having complete docs, but users can’t tell if the project is truly adopted, if the community is active, or if the ecosystem is expanding. The HAMi website redesign consciously addresses this.

Figure 2: HAMi ecosystem and device support

Figure 3: HAMi adopters

Figure 4: HAMi contributor organizations

Blog & Reading Experience

Blog cards, lists, and metadata were unified for easier reading and sharing. Blogs are now a core communication layer.

Figure 5: HAMi website blog list page

Mobile Optimization

Navigation, card layouts, footer, and search were improved for smoother mobile browsing.

Figure 6: HAMi website mobile view

Footer layout was enhanced for better navigation and credibility. Built-in search replaced unreliable external solutions, improving content accessibility.

Figure 7: HAMi website footer

Figure 8: HAMi website built-in search

What This Redesign Means for the HAMi Community

From screenshots, it looks like “the website looks better”. But from a community-building perspective, its significance is deeper.

First, HAMi’s external expression is more systematic.

The website is no longer just a collection of scattered pages, but is forming a complete narrative chain: users can understand project value from the homepage, capability details from docs, practical paths from blogs, and community impact from ecosystem modules.

Second, community content assets are reorganized.

Previously, valuable articles, diagrams, and explanations existed but were hard to find. Now, through homepage sections, navigation, and search refactor, these contents are more effectively connected.

Third, HAMi’s community image is more mature.

A mature open source project needs not just an active code repository, but clear, stable, and sustainable website expression. Structure, style, and usability are part of the community’s engineering capability.

Fourth, this lays the foundation for expanding case studies, adopters, contributors, and ecosystem content.

With the framework sorted, adding more case studies, collaboration entry points, or showcasing more adopters and partners will be more natural and easier for users to understand.

As a Community Contributor, My Top Three Takeaways from This Redesign

In summary, I believe this refactor got three things right:

Upgraded the website from a “content dump” to a “community gateway”.
Combined visual optimization with information architecture adjustment, not just a skin change.
Improved basic experiences like search, mobile, navigation, and footer.

These may not be as flashy as launching a new feature, but they directly impact content dissemination, user comprehension, and the project’s long-term image.

For infrastructure projects like HAMi, technical capability is fundamental, but clearly communicating, organizing, and continuously presenting that capability is also a form of infrastructure.

Summary

This HAMi documentation and website refactor is essentially an upgrade to the community’s “expression layer” infrastructure.

It improves visual and reading experience, reorganizes content, homepage narrative, search paths, mobile access, and community signal display. Homepage redesign, architecture diagram redraw, unified blog style, mobile optimization, enhanced footer, and switching from external to built-in search together constitute a true “refactor”.

Externally, it helps more people quickly understand HAMi; internally, it provides a stable platform for the community to accumulate case studies, expand the ecosystem, and serve adopters and contributors.

The website is not an accessory to the open source community, but part of its long-term influence. HAMi’s redesign is about taking this seriously.

If you’re interested in Kubernetes GPU virtualization, add me on WeChat jimmysong or scan the QR code below.

Check out the HAMi project on GitHub

GTC 2026 Eve: AI is Becoming the New Infrastructure

Jimmy Song — Sun, 15 Mar 2026 11:34:06 +0800

AI is quietly reshaping the infrastructure landscape, and GTC 2026 may become a key node in this transformation.

Next week, one of the most important technology conferences in the AI industry, NVIDIA GTC 2026, will be held in San Jose, USA.

For many people, GTC is just a GPU technology conference. But if you follow the development of the AI industry over the past few years, you’ll find an interesting phenomenon:

Many important narratives about AI infrastructure are gradually taking shape at GTC.

From CUDA, DGX, to AI Factory, and most recently Jensen Huang’s proposed AI Five-Layer Cake, NVIDIA is constantly attempting to redefine the computing infrastructure of the AI era.

This is why many people call GTC:

AI’s “Woodstock.”

Figure 1: NVIDIA GTC Conference

This year’s GTC (March 16-19) is expected to cover various levels of the AI stack, including:

AI Chips
AI Data Centers
AI Agents
Robotics
Inference Computing

According to NVIDIA’s official blog, this year’s keynote will focus on the complete AI stack from chips to applications.

If we put these signals together, we can actually see a larger trend:

AI is transforming from an “applied technology” into “infrastructure.”

The Perspective of Industrial Revolutions

From a longer time scale, the technological revolutions in human history are essentially infrastructure revolutions.

We usually divide industrial revolutions into four times.

In the table below, you can see the infrastructure corresponding to each industrial revolution:

Industrial Revolution	Infrastructure
Steam Revolution	Steam Engine
Electrical Revolution	Power Grid
Digital Revolution	Computer
Internet Era	Network

Table 1: Industrial Revolutions and Corresponding Infrastructure

First Industrial Revolution: Steam

The steam engine allowed humans to utilize mechanical power on a large scale for the first time. Production no longer relied on human or animal power, but on machines.

Second Industrial Revolution: Electricity

Electricity changed not only the source of power, but also the organization of production. Assembly lines, large-scale manufacturing, and modern industrial systems are all built on the foundation of the power grid.

Third Industrial Revolution: Computers

Computers allowed information to be processed digitally. Software became a production tool.

Fourth Industrial Revolution: Internet and Intelligence

The internet connects all computers together. Cloud computing transforms computing resources into infrastructure. And AI gives machines a certain degree of “cognitive ability.”

The True Significance of AI

If we observe these industrial revolutions, we discover a pattern:

Each industrial revolution produces a new General Purpose Infrastructure.

And AI is likely to become the next-generation infrastructure.

NVIDIA even directly stated in a recent article:

AI is essential infrastructure, like electricity and the internet.

In other words:

AI is no longer just an applied technology, but a new factor of production.

NVIDIA’s Five-Layer Cake

Recently, Jensen Huang proposed a very interesting concept: AI Five-Layer Cake.

Figure 2: AI Five Layer Cake (Image source: NVIDIA)

AI is broken down into five layers:

Energy
Chips
AI Infrastructure
Models
Applications

This model actually illustrates one thing:

AI is a complete industrial system.

Jensen Huang even described AI at Davos as:

“One of the largest-scale infrastructure constructions in human history.”

Signals GTC 2026 May Release

This year’s GTC is expected to release several important directions.

Inference Computing

The focus of AI in the past was training. But the main load of AI in the future is likely to be Inference.

Analysts expect that by 2030, 75% of computing demand in the AI data center market will come from inference.

Agentic AI

The past AI model was:

User → Model → Answer

The Agent model is more complex:

User → Agent → Tools → Model → Action

The flowchart below shows the main interaction paths in the Agent model:

Figure 3: Agentic AI Interaction Flow

AI is no longer just answering questions, but executing tasks.

Agent Platform

Recent media reports suggest that NVIDIA may launch a new Agent platform: NemoClaw, aimed at helping enterprises deploy AI Agents.

If this project is truly released, it means NVIDIA’s stack will become the following structure:

Figure 4: NVIDIA Agent Platform Architecture

This is actually a complete AI stack.

Agents Change Computing Workloads

The emergence of Agents brings new computing workload issues.

Past AI workloads were mainly:

Training
Inference

But Agents bring a third type of workload:

Agent Workloads

The figure below shows the diverse workload types related to Agents:

Figure 5: Agent Workloads Structure

The characteristic of this workload is highly fragmented. GPUs are no longer occupied for long periods, but rather face many small requests. This poses new challenges for infrastructure.

AI-Native Infrastructure

For the past few years, I’ve been thinking about a question:

What is AI-native infrastructure?

It is clearly not just “Kubernetes with GPUs.” I’m more inclined to believe it needs to possess several characteristics.

GPU as a First-Class Resource

In the cloud computing era, CPU is the core resource. In the AI era, GPU is the core resource.

Heterogeneous Computing

Real-world AI chips are not limited to NVIDIA:

NVIDIA
Ascend
Cambricon
Metax
Moore Threads

Future AI infrastructure must be able to manage heterogeneous computing.

GPU is a very expensive resource. If it cannot be shared, utilization will be very low. This is why GPU virtualization and slicing are becoming increasingly important.

AI Scheduling

AI scheduling includes not only traditional CPU and Memory, but also:

GPU
VRAM
Topology
Bandwidth

A Possible AI Tech Stack

Combining the above trends, the future AI stack may present the following structure:

Figure 6: AI Tech Stack Evolution

This structure is very close to NVIDIA’s Five-Layer Cake.

My Judgment

Combining signals from GTC, AI Factory, Agents, and AI Five-Layer Cake, we can see a very obvious trend:

AI is rewriting computing infrastructure.

Future competition may not just be “who has the best model,” but:

Who has the best AI Infrastructure.

Just like the past few decades:

Electricity determines industrial capability
Internet determines information capability
Cloud computing determines software capability

The future may be:

AI Infrastructure determines intelligence capability.

Summary

If we stretch the time scale a bit longer, we may be in a new historical stage.

AI is no longer just a technological tool. It is becoming new infrastructure.

Just like:

Electricity
Internet
Cloud computing

And AI-native infrastructure is likely to become one of the most important technology directions for the next decade.

When GPUs Move Toward Open Scheduling: Structural Shifts in AI Native Infrastructure

Jimmy Song — Fri, 13 Feb 2026 14:32:46 +0000

The future of GPU scheduling isn’t about whose implementation is more “black-box”—it’s about who can standardize device resource contracts into something governable.

Figure 1: GPU Open Scheduling

Introduction

Have you ever wondered: why are GPUs so expensive, yet overall utilization often hovers around 10–20%?

Figure 2: GPU Utilization Problem: Expensive GPUs with only 10-20% utilization

This isn’t a problem you solve with “better scheduling algorithms.” It’s a structural problem - GPU scheduling is undergoing a shift from “proprietary implementation” to “open scheduling,” similar to how networking converged on CNI and storage converged on CSI.

In the HAMi 2025 Annual Review, we noted: “HAMi 2025 is no longer just about GPU sharing tools—it’s a more structural signal: GPUs are moving toward open scheduling.”

By 2025, the signals of this shift became visible: Kubernetes Dynamic Resource Allocation (DRA) graduated to GA and became enabled by default, NVIDIA GPU Operator started defaulting to CDI (Container Device Interface), and HAMi’s production-grade case studies under CNCF are moving “GPU sharing” from experimental capability to operational excellence.

This post analyzes this structural shift from an AI Native Infrastructure perspective, and what it means for Dynamia and the industry.

Why “Open Scheduling” Matters

In multi-cloud and hybrid cloud environments, GPU model diversity significantly amplifies operational costs. One large internet company’s platform spans H200/H100/A100/V100/4090 GPUs across five clusters. If you can only allocate “whole GPUs,” resource misalignment becomes inevitable.

“Open scheduling” isn’t a slogan—it’s a set of engineering contracts being solidified into the mainstream stack.

Standardized Resource Expression

Before: GPUs were extended resources. The scheduler didn’t understand if they represented memory, compute, or device types.

Figure 3: Open Scheduling Standardization Evolution

Now: Kubernetes DRA provides objects like DeviceClass, ResourceClaim, and ResourceSlice. This lets drivers and cluster administrators define device categories and selection logic (including CEL-based selectors), while Kubernetes handles the full loop: match devices → bind claims → place Pods onto nodes with access to allocated devices.

Even more importantly, Kubernetes 1.34 stated that core APIs in the resource.k8s.io group graduated to GA, DRA became stable and enabled by default, and the community committed to avoiding breaking changes going forward. This means the ecosystem can invest with confidence in a stable, standard API.

Standardized Device Injection

Before: Device injection relied on vendor-specific hooks and runtime class patterns.

Now: The Container Device Interface (CDI) abstracts device injection into an open specification. NVIDIA’s Container Toolkit explicitly describes CDI as an open specification for container runtimes, and NVIDIA GPU Operator 25.10.0 defaults to enabling CDI on install/upgrade—directly leveraging runtime-native CDI support (containerd, CRI-O, etc.) for GPU injection.

This means “devices into containers” is also moving toward replaceable, standardized interfaces.

On this standardization path, HAMi’s role needs redefinition: it’s not about replacing Kubernetes—it’s about turning GPU virtualization and slicing into a declarative, schedulable, governable data plane.

Data Plane Perspective

HAMi’s core contribution expands the allocatable unit from “whole GPU integers” to finer-grained shares (memory and compute), forming a complete allocation chain:

Device discovery: Identify available GPU devices and models
Scheduling placement: Use Scheduler Extender to make native schedulers “understand” vGPU resource models (Filter/Score/Bind phases)
In-container enforcement: Inject share constraints into container runtime
Metric export: Provide observable metrics for utilization, isolation, and more

This transforms “sharing” from ad-hoc “it runs” experimentation into engineering capability that can be declared in YAML, scheduled by policy, and validated by metrics.

Scheduling Mechanism: Enhancement, Not Replacement

HAMi’s scheduling doesn’t replace Kubernetes—it uses a Scheduler Extender pattern to let the native scheduler understand vGPU resource models:

Filter: Filter nodes based on memory, compute, device type, topology, and other constraints
Score: Apply configurable policies like binpack, spread, topology-aware scoring
Bind: Complete final device-to-Pod binding

This architecture positions HAMi naturally as an execution layer under higher-level “AI control planes” (queuing, quotas, priorities)—working alongside Volcano, Kueue, Koordinator, and others.

Figure 4: HAMi Scheduling Architecture (Filter → Score → Bind)

CNCF public case studies provide concrete answers: in a hybrid, multi-cloud platform built on Kubernetes and HAMi, 10,000+ Pods run concurrently, and GPU utilization improves from 13% to 37% (nearly 3×).

Figure 5: CNCF Production Case Studies: Ke Holdings 13%→37%, DaoCloud 80%+ utilization, SF Technology 57% savings

Here are highlights from several cases:

Case Study 1: Ke Holdings (February 5, 2026)

Environment: 5 clusters spanning public and private clouds
GPU models: H200/H100/A100/V100/4090 and more
Architecture: Separate “GPU clusters” for large training tasks (dedicated allocation) vs “vGPU clusters” with HAMi fine-grained memory slicing for high-density inference
Concurrent scale: 10,000+ Pods
Outcome: Overall GPU utilization improved from 13% to 37% (nearly 3×)

Case Study 2: DaoCloud (December 2, 2025)

Hard constraints: Must remain cloud-native, vendor-agnostic, and compatible with CNCF toolchain
Adoption outcomes:
- Average GPU utilization: 80%+
- GPU-related operating cost reduction: 20–30%
- Coverage: 10+ data centers, 10,000+ GPUs
Explicit benefit: Unified abstraction layer across NVIDIA and domestic GPUs, reducing vendor dependency

Case Study 3: Prep EDU (August 20, 2025)

Negative experience: Isolation failures in other GPU-sharing approaches caused memory conflicts and instability
Positive outcome: HAMi’s vGPU scheduling, GPU type/UUID targeting, and compatibility with NVIDIA GPU Operator and RKE2 became decisive factors for production adoption
Environment: Heterogeneous RTX 4070/4090 cluster

Case Study 4: SF Technology (September 18, 2025)

Project: EffectiveGPU (built on HAMi)
Use cases: Large model inference, test services, speech recognition, domestic AI hardware (Huawei Ascend, Baidu Kunlun, etc.)
Outcomes:
- GPU savings: Large model inference runs 65 services on 28 GPUs (37 saved); test cluster runs 19 services on 6 GPUs (13 saved)
- Overall savings: Up to 57% GPU savings for production and test clusters
- Utilization improvement: Up to 100% GPU utilization improvement with GPU virtualization
Highlights: Cross-node collaborative scheduling, priority-based preemption, memory over-subscription

These cases demonstrate a consistent pattern: GPU virtualization becomes economically meaningful only when it participates in a governable contract—where utilization, isolation, and policy can be expressed, measured, and improved over time.

Strategic Implications for Dynamia

From Dynamia’s perspective (and as VP of Open Source Ecosystem), the strategic value of HAMi becomes clear:

Two-Layer Architecture: Open Source vs Commercial

HAMi (CNCF open source project): Responsible for “adoption and trust,” focused on GPU virtualization and compute efficiency
Dynamia enterprise products and services: Responsible for “production and scale,” providing commercial distributions and enterprise services built on HAMi

Figure 6: Dynamia Dual Mechanism: Open Source vs Commercial

This boundary is the foundation for long-term trust—project and company offerings remain separate, with commercial distributions and services built on the open source project.

Global Narrative Strategy

The internal alignment memo recommends a bilingual approach:

First layer: Lead globally with “GPU virtualization / sharing / utilization” (Chinese can directly use “GPU virtualization and heterogeneous scheduling,” but English first layer should avoid “heterogeneous” as a headline)

Second layer: When users discuss mixed GPUs or workload diversity, introduce “heterogeneous” to confirm capability boundaries—never as the opening hook

Core anchor: Maintain “HAMi (project and community) ≠ company products” as the non-negotiable baseline for long-term positioning

The Right Commercialization Landing

DaoCloud’s case study already set vendor-agnostic and CNCF toolchain compatibility as hard constraints, framing vendor dependency reduction as a business and operational benefit—not just a technical detail. Project-HAMi’s official documentation lists “avoid vendor lock” as a core value proposition.

In this context, the right commercialization landing isn’t “closed-source scheduling”—it’s productizing capabilities around real enterprise complexity:

Systematic compatibility matrix
SLO and tail-latency governance
Metering for billing
RBAC, quotas, multi-cluster governance
Upgrade and rollback safety
Faster path-to-production for DRA/CDI and other standardization efforts

Forward View: The Next 2–3 Years

My strong judgment: over the next 2–3 years, GPU scheduling competition will shift from “whose implementation is more black-box” to “whose contract is more open.”

The reasons are practical:

Hardware Form Factors and Supply Chains Are Diversifying

OpenAI’s February 12, 2026 “GPT‑5.3‑Codex‑Spark” release emphasizes ultra-low latency serving, including persistent WebSockets and a dedicated serving tier on Cerebras hardware
Large-scale GPU-backed financing announcements (for pan-European deployments) illustrate the infrastructure scale and financial engineering surrounding accelerator fleets

These signals suggest that heterogeneity will grow: mixed accelerators, mixed clouds, mixed workload types.

Low-Latency Inference Tiers Will Force Systematic Scheduling

Low-latency inference tiers (beyond just GPUs) will force resource scheduling toward “multi-accelerator, multi-layer cache, multi-class node” architectural design—scheduling must inherently be heterogeneous.

Open Scheduling Is Risk Management, Not Idealism

In this world, “open scheduling” isn’t idealism—it’s risk management. Building schedulable governable “control plane + data plane” combinations around DRA/CDI and other solidifying open interfaces, ones that are pluggable, multi-tenant governable, and co-evolvable with the ecosystem—this looks like the truly sustainable path for AI Native Infrastructure.

The next battleground isn’t “whose scheduling is smarter”—it’s “who can standardize device resource contracts into something governable.”

Conclusion

When you place HAMi 2025 back in the broader AI Native Infrastructure context, it’s no longer just the year of “GPU sharing tools”—it’s a more structural signal: GPUs are moving toward open scheduling.

Figure 7: Open Scheduling Future Vision

The driving forces come from both ends:

Upstream: Standards like DRA/CDI continue to solidify
Downstream: Scale and diversity (multi-cloud, multi-model, even accelerators beyond GPUs)

For Dynamia, HAMi’s significance has transcended “GPU sharing tool”: it turns GPU virtualization and slicing into declarative, schedulable, measurable data planes—letting queues, quotas, priorities, and multi-tenancy actually close the governance loop.

AI Learning Resources: 44 Curated Collections from Our Cleanup

Jimmy Song — Sun, 08 Feb 2026 12:20:05 +0000

“The best way to learn AI is to start building. These resources will guide your journey.”

Figure 1: AI Learning Resources Collection

In my ongoing effort to keep the AI Resources list focused on production-ready tools and frameworks, I’ve removed 44 collection-type projects—courses, tutorials, awesome lists, and cookbooks.

These resources aren’t gone—they’ve been moved here. This post is a curated collection of those educational materials, organized by type and topic. Whether you’re a complete beginner or an experienced practitioner, you’ll find something valuable here.

Why Remove Collections from AI Resources?

My AI Resources list now focuses on concrete tools and frameworks—projects you can directly use in production. Collections, while valuable, serve a different purpose: education and discovery.

By separating them, I:

Keep the resources list actionable and focused
Create a dedicated space for learning materials
Make it easier to find what you need

📚 Awesome Lists (14 Collections)

Awesome lists are community-curated collections of the best resources. They’re perfect for discovering new tools and staying updated.

Must-Explore Awesome Lists

Awesome Generative AI

Models, tools, tutorials, and research papers
Great for: Comprehensive overview of generative AI landscape

Awesome LLM

LLM resources: papers, tools, datasets, applications
Great for: Deep dive into large language models

Awesome AI Apps

Practical LLM applications, RAG examples, agent implementations
Great for: Real-world implementation examples

Awesome Claude Code

Claude Code commands, files, and workflows
Great for: Maximizing Claude Code productivity

Awesome MCP Servers

MCP servers for modular AI backend systems
Great for: Building with Model Context Protocol

Specialized Awesome Lists

Awesome ChatGPT Prompts - Prompt examples for various scenarios
Awesome LLM Apps - LLM applications with code examples
Awesome Multimodal LLM - Multimodal model resources
Awesome MCP Clients - MCP client tools and SDKs
Awesome Claude Skills - Claude Skills and workflows
Awesome GitHub Copilot - Copilot customizations
Awesome Nano Banana Pro - Image model prompts and examples
Awesome SaaS - AI platform templates
Awesome Claude Code Subagents - Claude Code subagents

🎓 Courses & Tutorials (9 Curricula)

Structured learning paths from universities and tech companies.

Microsoft’s AI Curriculum

AI for Beginners

12 weeks, 24 lessons covering neural networks, deep learning, CV, NLP
Great for: Complete AI foundation
Format: Lessons, quizzes, projects

Machine Learning for Beginners

12-week, 26-lesson curriculum on classic ML
Great for: ML fundamentals without deep math
Format: Project-based exercises

Generative AI for Beginners

18 lessons on building GenAI applications
Great for: Practical GenAI development
Format: Hands-on projects

AI Agents for Beginners

11 lessons on agent systems
Great for: Understanding autonomous agents
Format: Project-driven learning

EdgeAI for Beginners

Optimization, deployment, and real-world Edge AI
Great for: On-device AI applications
Format: Practical tutorials

MCP for Beginners

Model Context Protocol curriculum
Great for: Building with MCP
Format: Cross-language examples and labs

Official Platform Courses

Hugging Face Learn Center

Free courses on LLMs, deep RL, CV, audio
Great for: Hands-on Hugging Face ecosystem
Format: Interactive notebooks

OpenAI Cookbook

Runnable examples using OpenAI API
Great for: OpenAI API best practices
Format: Code examples and guides

PyTorch Tutorials

Basics to advanced deep learning
Great for: PyTorch mastery
Format: Comprehensive tutorials

🍳 Cookbooks & Example Collections (5 Collections)

Practical code examples and recipes.

Claude Cookbooks

Notebooks and examples for building with Claude
Great for: Anthropic Claude integration
Format: Jupyter notebooks

Hugging Face Cookbook

Practical AI cookbook with Jupyter notebooks
Great for: Open models and tools
Format: Hands-on examples

Tinker Cookbook

Training and fine-tuning examples
Great for: Fine-tuning workflows
Format: Platform-specific recipes

E2B Cookbook

Examples for building LLM apps
Great for: LLM application development
Format: Recipes and tutorials

arXiv Paper Curator

6-week course on RAG systems
Great for: Production-ready RAG
Format: Project-based learning

📖 Guides & Handbooks (5 Resources)

In-depth guides on specific topics.

Prompt Engineering Guide

Comprehensive prompt engineering resources
Great for: Mastering prompt design
Format: Guides, papers, lectures, notebooks

Evaluation Guidebook

LLM evaluation best practices from Hugging Face
Great for: Assessing LLM performance
Format: Practical guide

Context Engineering

Design and optimize context beyond prompt engineering
Great for: Advanced context management
Format: Practical handbook

Context Engineering Intro

Template and guide for context engineering
Great for: Providing project context to AI assistants
Format: Template + guide

Vibe-Coding Workflow

5-step prompt template for building MVPs with LLMs
Great for: Rapid prototyping with AI
Format: Workflow template

🗂️ Template & Workflow Collections

Reusable templates and workflows.

Claude Code Templates

Code templates for various programming scenarios
Great for: Claude AI development
Format: Template collection

n8n Workflows

2,000+ professionally organized n8n workflows
Great for: Workflow automation
Format: Searchable catalog

N8N Workflows Catalog

Community-driven reusable workflow templates
Great for: Workflow import and versioning
Format: Template catalog

📊 Research & Evaluation

Academic and evaluation resources.

LLMSys PaperList

Curated list of LLM systems papers
Great for: Research on training, inference, serving
Format: Paper collection

Free LLM API Resources

LLM providers with free/trial API access
Great for: Experimentation without cost
Format: Provider list

🎨 Other Notable Resources

System Prompts and Models of AI Tools

Community-curated collection of system prompts and AI tool examples
Great for: Prompt and agent engineering
Format: Resource collection

ML Course CS-433

EPFL Machine Learning Course
Great for: Academic ML foundation
Format: Lectures, labs, projects

Machine Learning Engineering

ML engineering open-book: compute, storage, networking
Great for: Production ML systems
Format: Comprehensive guide

Realtime Phone Agents Course

Build low-latency voice agents
Great for: Voice AI applications
Format: Hands-on course

LLMs from Scratch

Build a working LLM from first principles
Great for: Understanding LLM internals
Format: Repository + book materials

💡 How to Use This Collection

For Complete Beginners

Start with: Microsoft’s AI for Beginners
Practice with: PyTorch Tutorials
Explore: Awesome AI Apps for inspiration

For Developers

Build skills: OpenAI Cookbook + Claude Cookbooks
Find tools: Awesome Generative AI + Awesome LLM
Learn workflows: n8n Workflows Catalog

For Researchers

Stay updated: Awesome Generative AI + LLMSys PaperList
Deep dive: Awesome LLM
Implement: Hugging Face Cookbook

For Product Builders

Find examples: Awesome AI Apps
Learn workflows: n8n Workflows Catalog
Study patterns: Awesome LLM Apps

🔄 What Was NOT Removed

Agent frameworks and production tools remain in the AI Resources list, including:

AutoGen - Microsoft’s multi-agent framework
CrewAI - High-performance multi-agent orchestration
LangGraph - Stateful multi-agent applications
Flowise - Visual agent platform
Langflow - Visual workflow builder
And 80+ more agent frameworks

These are functional tools you can use to build applications, not educational collections. They belong in the AI Resources list.

📝 Summary

I removed 44 collection-type projects from the AI Resources list to keep it focused on production tools:

14 Awesome Lists - Discover new tools and stay updated
9 Courses & Tutorials - Structured learning paths
5 Cookbooks - Practical code examples
5 Guides & Handbooks - In-depth resources
4 Template Collections - Reusable workflows
7 Other Resources - Research and evaluation

These resources remain incredibly valuable for learning and discovery. They just serve a different purpose than the production-focused tools in my AI Resources list.

Next Steps:

Bookmark this post for future reference
Explore the AI Resources list for production tools (agent frameworks, databases, etc.)
Check out my blog for more AI engineering insights

Acknowledgments: This collection was compiled during my AI Resources cleanup initiative. Special thanks to all the maintainers of these awesome lists, courses, and collections for their invaluable contributions to the AI community.

Standing on Giants' Shoulders: The Traditional Infrastructure Powering Modern AI

Jimmy Song — Sun, 08 Feb 2026 08:00:00 +0000

“If I have seen further, it is by standing on the shoulders of giants.” — Isaac Newton

Figure 1: Standing on Giants’ Shoulders: The Traditional Infrastructure Powering Modern AI

In the excitement surrounding LLMs, vector databases, and AI agents, it’s easy to forget that modern AI didn’t emerge from a vacuum. Today’s AI revolution stands upon decades of infrastructure work—distributed systems, data pipelines, search engines, and orchestration platforms that were built long before “AI Native” became a buzzword.

This post is a tribute to those traditional open source projects that became the invisible foundation of AI infrastructure. They’re not “AI projects” per se, but without them, the AI revolution as we know it wouldn’t exist.

The Evolution: From Big Data to AI

Era	Focus	Core Technologies	AI Connection
2000s	Web Search & Indexing	Lucene, Elasticsearch	Semantic search foundations
2010s	Big Data & Distributed Computing	Hadoop, Spark, Kafka	Data processing at scale
2010s	Cloud Native	Docker, Kubernetes	Model deployment platforms
2010s	Stream Processing	Flink, Storm, Pulsar	Real-time ML inference
2020s	AI Native	Transformers, Vector DBs	Built on everything above

Table 1: Evolution of Data Infrastructure

Big Data Frameworks: The Data Engines

Before we could train models on petabytes of data, we needed ways to store, process, and move that data.

Apache Hadoop (2006)

GitHub: https://github.com/apache/hadoop

Hadoop democratized big data by making distributed computing accessible. Its HDFS filesystem and MapReduce paradigm proved that commodity hardware could process web-scale datasets.

Why it matters for AI:

Modern ML training datasets live in HDFS-compatible storage
Data lakes built on Hadoop became training data reservoirs
Proved that distributed computing could scale horizontally

Apache Kafka (2011)

GitHub: https://github.com/apache/kafka

Kafka redefined data streaming with its log-based architecture. It became the nervous system for real-time data flows in enterprises worldwide.

Why it matters for AI:

Real-time feature pipelines for ML models
Event-driven architectures for AI agent systems
Streaming inference pipelines
Model telemetry and monitoring backbones

Apache Spark (2014)

GitHub: https://github.com/apache/spark

Spark brought in-memory computing to big data, making iterative algorithms (like ML training) practical at scale.

Why it matters for AI:

MLlib made ML accessible to data engineers
Distributed data processing for model training
Spark ML became the de facto standard for big data ML
Proved that in-memory computing could accelerate ML workloads

Search Engines: The Retrieval Foundation

Before RAG (Retrieval-Augmented Generation) became a buzzword, search engines were solving retrieval at scale.

Elasticsearch (2010)

GitHub: https://github.com/elastic/elasticsearch

Elasticsearch made full-text search accessible and scalable. Its distributed architecture and RESTful API became the standard for search.

Why it matters for AI:

pioneered distributed inverted index structures
Proved that horizontal scaling was possible for search workloads
Many “AI search” systems actually use Elasticsearch under the hood
Query DSL influenced modern vector database query languages

OpenSearch (2021)

GitHub: https://github.com/opensearch-project/opensearch

When AWS forked Elasticsearch, it ensured search infrastructure remained truly open. OpenSearch continues the mission of accessible, scalable search.

Why it matters for AI:

Maintains open source innovation in search
Vector search capabilities added in 2023
Demonstrates community fork resilience

Databases: From SQL to Vectors

The evolution from relational databases to vector databases represents a paradigm shift—but both have AI relevance.

Traditional Databases That Paved the Way

Dgraph (2015) - Graph database proving that specialized data structures enable new use cases
TDengine (2019) - Time-series database for IoT ML workloads
OceanBase (2021) - Distributed database showing ACID transactions could scale

Why they matter for AI:

Proved that specialized database engines could outperform general-purpose ones
Database internals (indexing, sharding, replication) are now applied to vector databases
Multi-model databases (graph + vector + relational) are becoming the norm for AI apps

Cloud Native: The Runtime Foundation

When Docker and Kubernetes emerged, they weren’t built for AI—but AI couldn’t scale without them.

Docker (2013) & Kubernetes (2014)

GitHub: https://github.com/kubernetes/kubernetes

Kubernetes became the operating system for cloud-native applications. Its declarative API and controller pattern made it perfect for AI workloads.

Why it matters for AI:

Model deployment platforms (KServe, Seldon Core) run on K8s
GPU orchestration (NVIDIA GPU Operator, Volcano, HAMi) extends K8s
Kubeflow made K8s the standard for ML pipelines
Microservice patterns enable modular AI agent architectures

Service Mesh & Serverless

Istio (2016), Knative (2018) - Service mesh and serverless platforms that proved:

Network-level observability applies to AI model calls
Scale-to-zero is essential for cost-effective inference
Traffic splitting enables A/B testing of ML models

Why they matter for AI:

AI Gateway patterns evolved from API gateways + service mesh
Serverless inference platforms use Knative-style autoscaling
Observability patterns (tracing, metrics) are now standard for ML systems

API Gateways: From REST to LLM

API gateways weren’t designed for AI, but they became the foundation of AI Gateway patterns.

Kong, APISIX, KGateway

These API gateways solved rate limiting, auth, and routing at scale. When LLMs emerged, the same patterns applied:

AI Gateway Evolution:

Traditional API Gateway (2010s)
 ↓
Rate Limiting → Token Bucket Rate Limiting
Auth → API Key + Organization Management
Routing → Model Routing (GPT-4 → Claude → Local Models)
Observability → LLM-specific Telemetry (token usage, cost)
 ↓
AI Gateway (2024)

Why they matter for AI:

Proved that centralized API management scales
Plugin architectures enable LLM-specific features
Traffic management patterns apply to prompt routing
Security patterns (mTLS, JWT) now protect AI endpoints

Workflow Orchestration: The Pipeline Backbone

Data engineering needs pipelines. ML engineering needs pipelines. AI agents need workflows.

Apache Airflow (2015)

GitHub: https://github.com/apache/airflow

Airflow made pipeline orchestration accessible with its DAG-based approach. It became the standard for ETL and data engineering.

Why it matters for AI:

ML pipeline orchestration (feature engineering, training, evaluation)
Proved that DAG-based workflow definition works at scale
Prompt engineering pipelines use Airflow-style orchestration
Scheduler patterns are now applied to AI agent workflows

n8n, Prefect, Flyte

Modern workflow platforms that evolved from Airflow’s foundations:

n8n (2019) - Visual workflow automation with AI capabilities
Prefect (2018) - Python-native workflow orchestration for ML
Flyte (2019) - Kubernetes-native workflow orchestration for ML/data

Why they matter for AI:

Multi-modal agents need workflow orchestration
RAG pipelines are essentially ETL pipelines for embeddings
Prompt chaining is DAG-based orchestration

Data Formats: The Lakehouse Foundation

Before we could train on massive datasets, we needed formats that supported ACID transactions and schema evolution.

Delta Lake, Apache Iceberg, Apache Hudi

These table formats brought reliability to data lakes:

Why they matter for AI:

Training datasets need versioning and reproducibility
Feature stores use Delta/Iceberg as storage formats
Proved that “big data” could have transactional semantics
Schema evolution handles ML feature drift

The Invisible Thread: Why These Projects Matter

What do all these projects have in common?

They solved scaling first - AI training/inference needs horizontal scaling
They proved distributed systems work - Modern AI is fundamentally distributed
They created ecosystem patterns - Plugin systems, extension points, APIs
They established best practices - Observability, security, CI/CD
They built developer habits - YAML configs, declarative APIs, CLI tools

The AI Native Continuum

Modern “AI Native” infrastructure didn’t replace these projects—it builds on them:

Traditional Project	AI Native Evolution	Example
Hadoop HDFS	Distributed model storage	HDFS for datasets, S3 for checkpoints
Kafka	Real-time feature pipelines	Kafka → Feature Store → Model Serving
Spark ML	Distributed ML training	MLlib → PyTorch Distributed
Elasticsearch	Vector search	ES → Weaviate/Qdrant/Milvus
Kubernetes	ML orchestration	K8s → Kubeflow/KServe
Istio	AI Gateway service mesh	Istio → LLM Gateway with mTLS
Airflow	ML pipeline orchestration	Airflow → Prefect/Flyte for ML

Table 2: From Traditional to AI Native

Why We’re Removing Them from AI Resources List

This post honors these projects, but we’re also removing them from our AI Resources list. Here’s why:

They’re not “AI Projects”—they’re foundational infrastructure.

Hadoop, Kafka, Spark are data engineering tools, not ML frameworks
Elasticsearch is search, not semantic search
Kubernetes is general-purpose orchestration
API gateways serve REST/GraphQL, not just LLMs

But their absence doesn’t diminish their importance.

By removing them, we acknowledge that:

AI has its own ecosystem - Transformers, vector DBs, LLM ops
Traditional infra has its own domain - Data engineering, cloud native
The intersection is where innovation happens - AI-native data platforms, LLM ops on K8s

The Giants We Stand On

The next time you:

Deploy a model on Kubernetes
Stream features through Kafka
Search embeddings with a vector database
Orchestrate a RAG pipeline with Prefect

Remember: You’re standing on the shoulders of Hadoop, Kafka, Elasticsearch, Kubernetes, and countless others. They built the roads we now drive on.

The Future: Building New Giants

Just as Hadoop and Kafka enabled modern AI, today’s AI infrastructure will become tomorrow’s foundation:

Vector databases may become the new standard for all search
LLM observability may evolve into general distributed tracing
AI agent orchestration may reinvent workflow automation
GPU scheduling may influence general-purpose resource management

The cycle continues. The giants of today will be the foundations of tomorrow.

Conclusion: Gratitude and Continuity

As we clean up our AI Resources list to focus on AI-native projects, we don’t forget where we came from. Traditional big data and cloud native infrastructure made the AI revolution possible.

To the Hadoop committers, Kafka maintainers, Kubernetes contributors, and all who built the foundation: Thank you.

Your work enabled ChatGPT, enabled Transformers, enabled everything we now call “AI.”

Standing on your shoulders, we see further.

Acknowledgments: This post was inspired by the need to refactor our AI Resources list. The 27 projects mentioned here are being removed—not because they’re unimportant, but because they deserve their own category: The Foundation.

My First Month at Dynamia: Why AI Native Infra Is Worth the Investment

Jimmy Song — Fri, 06 Feb 2026 12:56:35 +0000

Time flies—it’s already been a month since I joined Dynamia. In this article, I want to share my observations from this past month: why AI Native Infra is a direction worth investing in, and some considerations for those thinking about their own career or technical direction.

Introduction

After nearly five years of remote work, I officially joined Dynamia last month as VP of Open Source Ecosystem. This decision was not sudden, but a natural extension of my journey from cloud native to AI Native Infra.

But this article is not just about my personal choice. I want to answer a more universal question: In the wave of AI infrastructure startups, why is compute governance a direction worth investing in?

For the past decade, I have worked continuously in the infrastructure space: from Kubernetes to Service Mesh, and now to AI Infra. I am increasingly convinced that the core challenge in the AI era is not “can the model run,” but “can compute resources be run efficiently, reliably, and in a controlled manner.” This conviction has only grown stronger through my observations and reflections during this first month at Dynamia.

This article answers three questions: What is AI Native Infra? Why is GPU virtualization a necessity? Why did I choose Dynamia and HAMi?

What Is AI Native Infra

The core of AI Native Infrastructure is not about adding another platform layer, but about redefining the governance target: expanding from “services and containers” to “model behaviors and compute assets.”

I summarize it as three key shifts:

Models as execution entities: Governance now includes not just processes, but also model behaviors.
Compute as a scarce asset: GPU, memory, and bandwidth must be scheduled and metered precisely.
Uncertainty as the default: Systems must remain observable and recoverable amid fluctuations.

In essence, AI Native Infra is about upgrading compute governance from “resource allocation” to “sustainable business capability.”

Why GPU Virtualization Is Essential

Many teams focus on model inference optimization, but in production, enterprises first encounter the problem of “underutilized GPUs.” This is where GPU virtualization delivers value.

Structural idleness: Small tasks monopolize large GPUs, leaving them idle for long periods.
Pseudo-isolation risks: Native sharing lacks hard boundaries, so a single task OOM can cause cascading failures.
Scheduling failures: Some users queue for GPUs while others occupy but do not use them, leading to both shortages and idleness.
Fragmentation waste: There may be enough total GPU, but not enough full cards, making efficient packing impossible.
Vendor lock-in anxiety: Proprietary, tightly coupled solutions make migration costs uncontrollable.

In short: GPUs must not only be allocatable, but also splittable, isolatable, schedulable, and governable.

The Relationship Between HAMi and Dynamia

This is the most frequently asked question. Here is the shortest answer:

HAMi: A CNCF-hosted open source project and community focused on GPU virtualization and heterogeneous compute scheduling.
Dynamia: The founding and leading company behind HAMi, providing enterprise-grade products and services based on HAMi.

Open source projects are not the same as company products, but the two evolve together. HAMi drives industry adoption and technical trust, while Dynamia brings these capabilities into enterprise production environments at scale. This “dual engine” approach is what makes Dynamia unique.

What HAMi Provides

HAMi (Heterogeneous AI Computing Virtualization Middleware) delivers three key capabilities on Kubernetes:

Virtualization and partitioning: Split physical GPUs into logical resources on demand to improve utilization.
Scheduling and topology awareness: Place workloads optimally based on topology to reduce communication bottlenecks.
Isolation and observability: Support quotas, policies, and monitoring to reduce production risks.

Currently, HAMi has attracted over 360 contributors from 16 countries, with more than 200 enterprise end users, and its international influence continues to grow.

Market Trends: The AI Infrastructure Startup Wave

AI infrastructure is experiencing a new wave of startups. The vLLM team’s company raised $150 million, SGLang’s commercial spin-off RadixArk is valued at $4 billion, and Databricks acquired MosaicML for $1.3 billion—all pointing to a consensus: Whoever helps enterprises run large models more efficiently and cost-effectively will hold the keys to next-generation AI infrastructure.

Against this backdrop, the positioning of Dynamia and HAMi is even clearer. Many teams focus on “model performance acceleration” and “inference optimization” (like vLLM, SGLang), while we focus on “resource scheduling and virtualization”—enabling better orchestration of existing accelerated hardware resources.

The two are complementary: the former makes individual models run faster and cheaper, while the latter ensures that compute allocation at the cluster level is efficient, fair, and controllable. This is similar to extending Kubernetes’ CPU/memory scheduling philosophy to GPU and heterogeneous compute management in the AI era.

Why AI Native Infra Is Worth the Investment

My observations this month have convinced me that compute governance is the most undervalued yet most promising area in AI infrastructure. If you are considering a career or technical investment, here is my assessment:

First, this is a real and urgent pain point

Model training and inference optimization attract a lot of attention, but in production, enterprises first encounter the problem of “underutilized GPUs”—structural idleness, scheduling failures, fragmentation waste, and vendor lock-in anxiety. Without solving these problems, even the fastest models cannot scale in production. GPU virtualization and heterogeneous compute scheduling are the “infrastructure below infrastructure” for enterprise AI transformation.

Second, this is a clear long-term track

Frameworks like vLLM and SGLang emerge constantly, making individual models run faster. But who ensures that compute allocation at the cluster level is efficient, fair, and controllable? This is similar to extending Kubernetes’ success in CPU/memory scheduling to GPU and heterogeneous compute management in the AI era. This is not something that can be finished in a year or two, but a direction for continuous construction over the next five to ten years.

Third, this is an open and verifiable path

Dynamia chose to build on HAMi as an open source foundation, first solving general capabilities, then supporting enterprise adoption. This means the technical direction is transparent and verifiable in the community. You can form your own judgment by participating in open source, observing adoption, and evaluating the ecosystem—rather than relying on the black-box promises of proprietary solutions.

Fourth, this is a window of opportunity that is opening now

AI infrastructure is being redefined. Investing in its construction today will continue to yield value in the coming years. The vLLM team’s company raised $150 million, SGLang’s commercial spin-off RadixArk is valued at $4 billion, Databricks acquired MosaicML for $1.3 billion—all validating the same trend: Whoever helps enterprises run large models more efficiently will hold the keys to next-generation AI infrastructure.

I hope to bring my experience in cloud native and open source communities to the next stage of HAMi and Dynamia: turning GPU resources from a “cost center” into an “operational asset.” This is not just my career choice, but my judgment and investment in the direction of next-generation infrastructure.

Join the HAMi Community

Add me on WeChat (jimmysong) to join the HAMi community focused on GPU virtualization and heterogeneous compute scheduling.

If you are also interested in HAMi, GPU virtualization, AI Native Infra, or Dynamia, feel free to reach out.

Summary

From cloud native to AI Native Infra, my observations this month have only strengthened my conviction: The true upper limit of AI applications is determined by the infrastructure’s ability to govern compute resources.

HAMi addresses the fundamental issues of GPU virtualization and heterogeneous compute scheduling, while Dynamia is driving these capabilities into large-scale production. If you are also looking for a technical direction worth long-term investment, AI Native Infra—especially compute governance and scheduling—is a track with real pain points, a clear path, an open ecosystem, and an opening window of opportunity.

Joining Dynamia is not just a career choice, but a commitment to building the next generation of infrastructure. I hope the observations and reflections in this article can provide some reference for you as you evaluate technical directions and career opportunities.

If you are also interested in HAMi, GPU virtualization, AI Native Infra, or Dynamia, feel free to reach out.

The True Inflection Point of ADD: When Spec Becomes the Core Asset of AI-Era Software

Jimmy Song — Tue, 20 Jan 2026 07:51:36 +0000

The role of Spec is undergoing a fundamental transformation, becoming the governance anchor of engineering systems in the AI era.

The Essence of Software Engineering and the Cost Structure Shift Brought by AI

From first principles, software engineering has always been about one thing: stably, controllably, and reproducibly transforming human intent into executable systems.

Artificial Intelligence (AI) does not change this engineering essence, but it dramatically alters the cost structure:

Implementation costs plummet: Code, tests, and boilerplate logic are rapidly commoditized.
Consistency costs rise sharply: Intent drift, hidden conflicts, and cross-module inconsistencies become more frequent.
Governance costs are amplified: As agents can act directly, auditability, accountability, and explainability become hard constraints.

Therefore, in the era of Agent-Driven Development (ADD), the core issue is not “can agents do the work,” but how to maintain controllability and intent preservation in engineering systems under highly autonomous agents.

The ADD Era Inflection Point: Three Structural Preconditions

Many attribute the “explosion” of ADD to more mature multi-agent systems, stronger models, or more automated tools. In reality, the true structural inflection point arises only when these three conditions are met:

Agents have acquired multi-step execution capabilities

With frameworks like LangChain, LangGraph, and CrewAI, agents are no longer just prompt invocations, but long-lived entities capable of planning, decomposition, execution, and rollback.

Agents are entering real enterprise delivery pipelines

Once in enterprise R&D, the question shifts from “can it generate” to “who approved it, is it compliant, can it be rolled back.”

Traditional engineering tools lack a control plane for the agent era

Tools like Git, CI, and Issue Trackers were designed for “human developer collaboration,” not for “agent execution.”

When these three factors converge, ADD inevitably shifts from an “efficiency tool” to a “governance system.”

The Changing Role of Spec: From Documentation to System Constraint

In the context of ADD, Spec is undergoing a fundamental shift:

Spec is no longer “documentation for humans,” but “the source of constraints and facts for systems and agents to execute.”

Spec now serves at least three roles:

Verifiable expression of intent and boundaries

Requirements, acceptance criteria, and design principles are no longer just text, but objects that can be checked, aligned, and traced.

Stable contracts for organizational collaboration

When agents participate in delivery, verbal consensus and tacit knowledge quickly fail. Versioned, auditable artifacts become the foundation of collaboration.

Policy surface for agent execution

Agents can write code, modify configurations, and trigger pipelines. Spec must become the constraint on “what can and cannot be done.”

From this perspective, the status of Spec is approaching that of the Control Plane in AI-native infrastructure.

The Reality of Multi-Agent Workflows: Orchestration and Governance First

In recent systems (such as APOX and other enterprise products), an industry consensus is emerging:

Multi-agent collaboration no longer pursues “full automation,” but is staged and gated.
Frameworks like LangGraph are used to build persistent, debuggable agent workflows.
RAG (e.g., based on Milvus) is used to accumulate historical Specs, decisions, and context as long-term memory.
The IDE mainly focuses on execution efficiency, not engineering governance.

Figure 1: APOX user interface

APOX (AI Product Orchestration eXtended) is a multi-agent collaboration workflow platform for enterprise software delivery. Its core goals are:

To connect the entire process from product requirements to executable code with a governable Agentflow and explicit engineering artifact chain.
To assign dedicated AI agents to each delivery stage (such as PRD, PO, Architecture, Developer, Implementation, Coding, etc.).
To embed manual approval gates and full audit trails at every step, solving the “intent drift and consistency” governance problem that traditional AI coding tools cannot address.
The platform provides a VS Code plugin for real-time sync between local IDE and web artifacts, allowing Specs, code, tasks, and approval statuses to coexist in the repository.
Supports assigning different base models to different agents according to enterprise needs.

APOX is not about simply speeding up code generation, but about elevating “Spec” from auxiliary documentation to a verifiable, constrainable, and traceable core asset in engineering—building a control plane and workflow governance system suitable for Agent-Driven Development.

Such systems emphasize:

An explicit artifact chain from PRD → Spec → Task → Implementation.
Manual confirmation and audit points at every stage.
Bidirectional sync between Spec, code, repository, and IDE.

This is not about “smarter AI,” but about engineering systems adapting to the agent era.

The Long-Term Value of Spec: The Core Anchor of Engineering Assets

This is not to devalue code, but to acknowledge reality:

There will always be long-term differentiation in algorithms and model capabilities.
General engineering implementation is rapidly homogenizing.
What is hard to replicate is: how to define problems, constrain systems, and govern change.

In the ADD era, the value of Spec is reflected in:

Determining what agents can and cannot do.
Carrying the organization’s long-term understanding of the system.
Serving as the anchor for audit, compliance, and accountability.

Code will be rewritten again and again; Spec is the long-term asset.

Risks and Challenges of ADD: Living Spec and Governance Constraints

ADD also faces significant risks:

Can Spec become a Living Spec

That is, when key implementation changes occur, can the system detect “intent changes” and prompt Spec updates, rather than allowing silent drift?

Can governance achieve low friction but strong constraints

If gates are too strict, teams will bypass them; if too loose, the system loses control.

These two factors determine whether ADD is “the next engineering paradigm” or “just another tool bubble.”

The Trend Toward Control Planes in Engineering Systems

From a broader perspective, ADD is the inevitable result of engineering systems becoming “control planes”:

Engineering systems are evolving from “human collaboration tools” to “control systems for agent execution.”

In this structure:

Agent / IDE is the execution plane.
RAG / Memory is the state and memory plane.
Spec is the intent and policy plane.
Gates, audit, and traceability form the governance loop.

This closely aligns with the evolution path of AI-native infrastructure.

Summary

The winners of the ADD era will not be the systems with “the most agents or the fastest generation,” but those that first upgrade Spec from documentation to a governable, auditable, and executable asset. As automation advances, the true scarcity is the long-term control of intent.

AI Voice Dictation Input Methods Are Becoming the New Shortcut Key for the Programming Era

Jimmy Song — Sun, 18 Jan 2026 06:53:08 +0000

Voice input methods are not just about being “fast”—they are becoming a brand new gateway for developers to collaborate with AI.

Warning

On January 12, 2026, due to financial difficulties encountered during operations, the Miaoyan project announced the cessation of operations and the team was disbanded. The application will no longer be updated or maintained, but existing versions can continue to be used on the current device and system, and do not store any audio or transcription content.

Figure 1: Can voice input become the new shortcut for developers? My in-depth comparison experience.

AI Voice Input Methods Are Becoming the “New Shortcut Key” in the Programming Era

I am increasingly convinced of one thing: PC-based AI voice input methods are evolving from mere “input tools” into the foundational interaction layer for the era of programming and AI collaboration.

It’s not just about typing faster—it determines how you deliver your intent to the system, whether you’re writing documentation, code, or collaborating with AI in IDEs, terminals, or chat windows.

Because of this, the differences in voice input method experiences are far more significant than they appear on the surface.

My Six Evaluation Criteria for AI Voice Input Methods

After long-term, high-frequency use, I have developed a set of criteria to assess the real-world performance of AI voice input methods:

Response speed: Does text appear quickly enough after pressing the shortcut to keep up with your thoughts?
Continuous input stability: Does it remain reliable during extended use, or does it suddenly fail or miss recognition?
Mixed Chinese-English and technical terms: Can it reliably handle code, paths, abbreviations, and product names?
Developer friendliness: Is it truly designed for command line, IDE, and automation scenarios?
Interaction restraint: Does it avoid introducing distracting features that interfere with input itself?
Subscription and cost structure: Is it a standalone paid product, or can it be bundled with existing tool subscriptions?

Based on these criteria, I focused on comparing Miaoyan, Shandianshuo, and Zhipu AI Voice Input Method.

Miaoyan: Currently the Most “Developer-Oriented” Domestic Product

Miaoyan was the first domestic AI voice input method I used extensively, and it remains the one I am most willing to use continuously.

Figure 2: Miaoyan is currently my most-used Mac voice input method.

Command Mode: The Key Differentiator for Developer Productivity

It’s important to clarify that Miaoyan’s command mode is not about editing text via voice. Instead:

You describe your need in natural language, and the system directly generates an executable command-line command.

This is crucial for developers:

It’s not just about input
It’s about turning voice into an automation entry point
Essentially, it connects voice to the CLI or toolchain

This design is clearly focused on engineering efficiency, not office document polishing.

Usage Experience Summary

Fast response, nearly instant
Output is relatively clean, with minimal guessing
Interaction design is restrained, with no unnecessary concepts
Developer-friendly mindset

But there are some practical limitations:

It is a completely standalone product
Requires a separate subscription
Still in relatively small-scale use

From a product strategy perspective, it feels more like a “pure tool” than part of an ecosystem.

Note

Shandianshuo: Local-First Approach, Developer Experience Depends on Your Setup

Shandianshuo takes a different approach: it treats voice input as a “local-first foundational capability,” emphasizing low latency and privacy (at least in its product narrative). The natural advantages of this approach are speed and controllable marginal costs, making it suitable as a “system capability” that’s always available, rather than a cloud service.

Figure 3: Shandianshuo settings page

However, from a developer’s perspective, its upper limit often depends on “how you implement enhanced capabilities”:

If you only use it for basic transcription, the experience is more like a high-quality local input tool. But if you want better mixed Chinese-English input, technical term correction, symbol and formatting handling, the common approach is to add optional AI correction/enhancement capabilities, which usually requires extra configuration (such as providing your own API key or subscribing to enhanced features). The key trade-off here is not “can it be used,” but “how much configuration cost are you willing to pay for enhanced capabilities.”

If you want voice input to be a “lightweight, stable, non-intrusive” foundation, Shandianshuo is worth considering. But if your goal is to make voice input part of your developer workflow (such as command generation or executable actions), it needs to offer stronger productized design at the “command layer” and in terms of controllability.

Zhipu AI Voice Input Method: Stable but with Friction

I also thoroughly tested the Zhipu AI Voice Input Method.

Figure 4: Zhipu Voice Input Method settings interface

Its strengths include:

More stable for long-term continuous input
Rarely becomes completely unresponsive
Good tolerance for longer Chinese input

But with frequent use, some issues stand out:

Idle misrecognition: If you press the shortcut but don’t speak, it may output random characters, disrupting your input flow
Occasionally messy output: Sometimes adds irrelevant words, making it less controllable than Miaoyan
Basic recognition errors: For example, “Zhipu” being recognized as “Zhipu” (with a different character), which is a trust issue for professional users
Feature-heavy design: Various tone and style features increase cognitive load

Subscription Bundling: Zhipu’s Practical Advantage

Although I prefer Miaoyan in terms of experience, Zhipu has a very practical advantage:

If you already subscribe to Zhipu’s programming package, the voice input method is included for free.

This means:

No need to pay separately for the input method
Lower psychological and decision-making cost
More likely to become the “default tool” that stays

From a business perspective, this is a very smart strategy.

Main Comparison Table

The following table compares the three products across key dimensions for quick reference.

Dimension	Miaoyan	Shandianshuo	Zhipu AI Voice Input Method
Response Speed	Fast, nearly instant	Usually fast (local-first)	Slightly slower than Miaoyan
Continuous Stability	Stable	Depends on setup and environment	Very stable
Idle Misrecognition	Rare	Generally restrained (varies by version)	Obvious: outputs characters even if silent
Output Cleanliness/Control	High	More like an “input tool”	Occasionally messy
Developer Differentiator	Natural language → executable command	Local-first / optional enhancements	Ecosystem-attached capabilities
Subscription & Cost	Standalone, separate purchase	Basic usable; enhancements often require setup/subscription	Bundled free with programming package
My Current Preference	Best experience	More like a “foundation approach”	Easy to keep but not clean enough

Table 1: Core Comparison of Miaoyan, Shandianshuo, and Zhipu AI Voice Input Methods

User Loyalty to AI Voice Input Methods

The switching cost for voice input methods is actually low: just a shortcut key and a habit of output.

What really determines whether users stick around is:

Whether the output is controllable
Whether it keeps causing annoying minor issues
Whether it integrates into your existing workflow and payment structure

For me personally:

The best and smoothest experience is still Miaoyan
The one most likely to stick around is probably Zhipu
Shandianshuo is more of a “foundation approach” and worth watching for how its enhancements evolve

These points are not contradictory.

Summary

Miaoyan is more mature in engineering orientation, command capabilities, and input control
Zhipu has practical advantages in stability and subscription bundling
Shandianshuo takes a local-first + optional enhancement approach, with the key being how it balances “basic capability” and “enhancement cost”
Who truly becomes the “default gateway” depends on reducing distractions, fixing frequent minor issues, and treating voice input as true “infrastructure” rather than an add-on feature

The competition among AI voice input methods is no longer about recognition accuracy, but about who can own the shortcut key you press every day.

From Spatial Data to AI Open Source: Technical Standards, Data Sovereignty, and the Global Divide

Jimmy Song — Sun, 11 Jan 2026 03:29:28 +0000

The divide in technical standards and data sovereignty determines the global competitive landscape of infrastructure open source in the AI era.

In this article, I will use the differences in air quality data presentation in Apple Maps and Weather as a starting point to explore how technical standards and data sovereignty influence the open source paths of AI in different countries. I will further analyze why, in the AI era, infrastructure-level open source has become the key battleground for ecosystem dominance.

Author’s Note

This article originates from a very everyday observation: Why is air quality data in China shown as “points” in Apple Maps and Weather, while in other countries it is often displayed as “areas”?

Figure 1: Air quality map in Apple Weather, showing point-based data in China and area-based data in other countries

At first glance, it seems like a product experience difference. But when I reconsidered this issue in the context of engineering, standards, and system design, I realized it actually points to a much bigger question: how different countries understand the relationship between technology, standards, openness, and sovereignty.

As an engineer who has long worked in cloud native, AI infrastructure, and open source ecosystems, I gradually realized that this difference is not limited to air quality or map data. In the AI era, it is further amplified, directly affecting how we open source models, build infrastructure, and whether we can participate in the formulation of global rules.

Writing this article is not about judging right or wrong, but about using a concrete example to explain a structural difference and discuss the long-term impact and real opportunities this difference may bring in the AI era.

What is especially important: at the level of AI infrastructure and infra-level open source, the competition has just begun. China is not without opportunities, but the choice of path will become more critical than ever.

Differences in Air Quality Data Presentation: A Microcosm of Technical Standards and Sovereignty

The following image illustrates the divide between spatial data, AI open source, and technical standards. By comparing how air quality data is presented in Apple Maps and Weather in different countries, you can intuitively feel the differences in technical standards and sovereignty strategies behind the scenes.

Figure 2: The divide between spatial data, AI open source, and technical standards

If you regularly use global products such as maps, weather, traffic, or various data services, you may notice a recurring phenomenon that is rarely discussed seriously: the way data is presented in China often differs significantly from global mainstream standards.

A very intuitive example comes from the air quality display in Apple Maps or Weather. In China, air quality is usually shown as discrete points; in the US, Europe, Japan, and other countries, it is often rendered as continuous coverage areas.

At first glance, this seems like a product experience difference, and may even lead people to mistakenly believe that “China’s data is incomplete.” But if you treat it as an engineering or system design issue, you will find: this is not a matter of data capability, but a different choice in technical standards, data sovereignty, and openness strategies.

And this choice is not limited to air quality.

Air Quality Is Just a Slice: Greater Differences in Spatial Public Data

Air quality is just a highly visible and relatively low-risk example. Similar differences have long existed in broader spatial and public data domains.

Maps and coordinate systems
Surveying and high-precision spatial data
Real-time traffic and population movement
Remote sensing, environmental, and urban operation data

In global mainstream systems, such data is usually regarded as public information infrastructure. It is standardized, gridded, API-ified, allows interpolation, modeling, and redistribution, and is widely used in research, business, and product innovation.

In China, this data often takes another form: hierarchical, discrete, strictly defined, and with centralized interpretation authority.

This is not a technical preference in a single field, but a systemic logic of technology and governance.

Three Global Paths

Placing China in a global context, we can see that there are roughly three different paths worldwide regarding “how public data and technical standards are opened.”

Engineering-Open Type: Standards and Ecosystem First

Represented by the US and some European countries, the core features of this system are:

Public data prioritized as infrastructure
Standards and interfaces come first
Encourages engineering autonomy and ecosystem evolution
Tolerates model inference and uncertainty

This path directly shaped the global landscape of foundational software and infrastructure-level open source. Linux, Kubernetes, and the cloud native system are essentially products of openness at the rules layer.

Governance-Sovereignty Type: Control and Auditability First

Represented by China, this path emphasizes:

Sensitivity of spatial and public data
Data as part of governance capability
Standards, definitions, and release methods are highly bound
Emphasizes traceability, accountability, and controllability

In this system, “point data” is not a sign of technological backwardness, but a governable technical form. When a technical system is designed as a governance system, its primary goal is not reusability, but controllability.

Compromise-Coordinated Type: Cautious Openness, Engineering Internationalization

Some countries try to find a balance between the two, maintaining caution in spatial data while being highly internationalized in engineering and industry. This shows that the difference is not about being advanced or backward, but about different objective functions.

The following diagram compares the core characteristics, typical cases, and advantages/challenges of these three paths from a global perspective. The “Engineering-Open Type” on the left shapes the global infrastructure software landscape through standards and ecosystems; the “Governance-Sovereignty Type” in the middle emphasizes data sovereignty and security controllability but has limitations in influence at the rules layer; the “Compromise-Coordinated Type” on the right attempts to find a balance between security and openness. The divide between these three paths directly affects the infrastructure competition landscape of various countries in the AI era.

Figure 3: Global Perspective: Three Paths for Public Data and Technical Standards

The Essence of “Point” vs. “Area” in Air Quality

Among all spatial public data, air quality is an ideal observation window:

Does not directly involve military or core economic security
Highly visible, updated daily, and perceptible to everyone

China does not lack air quality data; on the contrary, the density of monitoring stations is among the highest in the world. The real difference lies in:

Whether interpolation is allowed
Whether model inference is allowed
Whether platforms are allowed to reinterpret the data

“Point” means authenticity and traceability; “area” means models, inference, and redistribution of interpretive authority. This is precisely the watershed between technical standards and data sovereignty.

The following diagram compares two different technical paths. The left side, “Governance-Sovereignty Type,” emphasizes data traceability and controllability, using discrete point-based data presentation. The right side, “Engineering-Open Type,” allows model interpolation and inference, providing more user-friendly experience through continuous area-based coverage. The essence of this difference lies not in the level of technical capability, but in the different choices made between data sovereignty, governance capability, and open ecosystems.

Figure 4: Technical Standards and Sovereignty Divide in Spatial Data Presentation

The Amplification Effect in the AI Era

With the above logic in mind, many phenomena in the AI era become less confusing.

For example:

Why are Chinese AI companies more willing to open source large language model (LLM) weights, while American companies have clearly shifted toward closed source in recent years?
Why is foundational software and infrastructure-level open source still mainly led by the US?

The key is not “whether to open source,” but “which layer is open sourced.”

Model weights are static, declarable assets
Infrastructure, runtimes, protocols, and standards are dynamic, evolving system rules

Open sourcing weights is essentially openness at the asset layer; infrastructure-level open source means relinquishing control over operating rules and interpretive authority.

The following diagram compares two different layers of AI open source. The left side shows “Model Weight Layer Open Source,” which is a typical feature of Chinese path—opening static digital assets with low cost and controllable risk, but not involving rule-making. The right side shows “Infrastructure Layer Open Source,” which is a core strategy of US path—by open sourcing development tools, protocol standards, runtimes, and compute scheduling and other infrastructure, defining how AI is used, thereby mastering ecosystem rules and interpretive authority. Key insight: Open sourcing model weights does not equal mastering AI ecosystem, and the real competitive focus is shifting to the infrastructure layer of “how AI runs.”

Figure 5: Two Layers of AI Era Open Source: Model Weights vs Infrastructure

The US Approach: Focusing on Rules and Runtime Layers

In the past year or two, US-led AI open source and ecosystem initiatives have shown a highly consistent direction: not rushing to open source the strongest models, but focusing on defining “how AI is used.”

The Linux Foundation established AAIF (Agentic AI Foundation), focusing on AI infrastructure, standards, and toolchain collaboration
Protocols like MCP (Model Context Protocol) aim to define common interaction methods between agents and tools/systems
Major tech companies are generally focusing on APIs, platforms, runtimes, and ecosystem binding

The commonality of these actions: competing in model capability, but controlling the usage rules.

China’s Shift: From Model-Oriented to Infrastructure-Oriented

It is important to emphasize that this difference does not mean China is unaware of the issue.

Whether in policy discussions or within industry and research institutions, the risk of “only open sourcing models without controlling infrastructure and standard dominance” has been repeatedly discussed.

The real challenge lies in how to achieve a directional shift within the existing governance logic and risk framework. This shift has already appeared in some concrete practices.

Exploration and Practice at the Infrastructure Layer

In the AI era, infrastructure often starts with the most engineering-driven problems.

HAMi Project

Projects like HAMi do not focus on model capability, but on:

Abstraction, allocation, and isolation of GPU resources
How multi-tenant AI workloads are run
How computing power transitions from hardware assets to governable system resources

The significance of such projects is not about being “SOTA,” but about entering the domain of “how AI runs.”

AI Runtime Reconstruction from a System Software Review

Exploration at the research institution level is also noteworthy. The FlagOS initiative by the Beijing Academy of Artificial Intelligence is a clear signal: AI is being redefined as a system software issue, not just a model or algorithm problem.

Long-Term Tech Stack Investment by Industry Players

In the industry, Huawei’s strategy reflects a similar direction: not simply open sourcing models, but attempting to build a complete, controllable AI tech stack, from computing power to frameworks, platforms, and ecosystems. This is a slower, heavier, but more infrastructure-competitive path.

Realistic Assessment: The Starting Point of AI Infrastructure Competition

Taking a longer view, we find an easily overlooked fact:

At the level of AI infrastructure and infra-level open source, there is no settled pattern between China and the US.

The US advantage lies in:

Mature engineering culture
Standard organizations and foundation mechanisms
High proficiency in openness at the rules layer

China’s variables include:

Huge AI application scenarios
Extreme demand for computing power and system efficiency
Ongoing directional adjustments

The real uncertainty is not “whether we can catch up,” but whether it is possible to gradually open up space for engineering autonomy and standard co-construction while maintaining governance bottom lines.

Summary

The “points” and “areas” of air quality, model weights and the world of operations—behind these appearances lies not a simple technical route dispute, but how a country finds its own balance between openness, standards, and sovereignty.

In the AI era, this issue will not disappear, but will become more concrete and more engineering-driven. And this is precisely where there are still opportunities for China’s AI infrastructure open source.

Joining Dynamia: Embarking on a New Journey in AI Native Infrastructure

Jimmy Song — Wed, 07 Jan 2026 07:49:21 +0000

Compute governance is the critical bottleneck for AI scaling. From hardware consumption to core asset, this long-undervalued path needs to be redefined.

Figure 1: Dynamia.ai

A New Beginning

I have officially joined Dynamia as Open Source Ecosystem VP, responsible for the long-term development of the company in open source, technical narrative, and AI Native Infrastructure ecosystem directions.

Why I Chose Dynamia

I chose to join Dynamia not because it’s a company trying to “solve all AI problems,” but precisely the opposite—it’s because Dynamia focuses intensely on one unavoidable, yet long-undervalued core issue in AI Native Infrastructure: compute, especially Graphics Processing Units (GPU), are evolving from “technical resources” into infrastructure elements that require refined governance and economic management.

Through years of practice in cloud native, distributed systems, and AI infrastructure (AI Infra), I’ve formed a clear judgment: as Large Language Models (LLM) and AI Agents enter the stage of large-scale deployment, the real bottleneck limiting system scalability and sustainability is no longer just model capability itself, but how compute is measured, allocated, isolated, and scheduled, and how a governable, accountable, and optimizable operational mechanism is formed at the system level. From this perspective, the core challenge of AI infrastructure is essentially evolving into a “resource governance and Token economy” problem.

About Dynamia and HAMi

Dynamia is an AI-native infrastructure technology company rooted in open source DNA, driving efficiency leaps in heterogeneous compute through technological innovation. Its leading open source project, HAMi (Heterogeneous AI Computing Virtualization Middleware), is a Cloud Native Computing Foundation (CNCF) sandbox project providing GPU, NPU and other heterogeneous device virtualization, sharing, isolation, and topology-aware scheduling capabilities, widely adopted by 50+ enterprises and institutions.

Dynamia’s Technical Approach

In this context, Dynamia’s technical approach—starting from the GPU layer, which is the most expensive, scarcest, and least unified abstraction layer in AI systems, treating compute as a foundational resource that can be measured, partitioned, scheduled, governed, and even “tokenized” for refined accounting and optimization—aligns highly with my long-term judgment on AI-native infrastructure.

This path doesn’t use “model capabilities” or “application innovation” as selling points in the short term, nor is it easily packaged into simple stories. However, with rising compute costs, heterogeneous accelerators becoming the norm, and AI systems moving toward multi-tenant and large-scale operations, these infrastructure-level capabilities are gradually becoming prerequisites for the establishment and expansion of AI systems.

Future Focus

As Dynamia’s Open Source Ecosystem VP, I will focus on technical narrative of AI-native infrastructure, open source ecosystem building, and global developer collaboration, promoting compute from “hardware resource being consumed” to governable, measurable, and optimizable AI infrastructure core asset, laying the foundation for the scaling and sustainable evolution of AI systems in the next stage.

Summary

Joining Dynamia is an important milestone in my career and a concrete action demonstrating my long-term optimism about AI-native infrastructure. Compute governance is not a short-term trend that yields quick results, but an infrastructure proposition that cannot be bypassed for AI large-scale deployment. I look forward to exploring, building, and landing solutions on this long-undervalued path with global developers.

References

Running Parallel AI Agents on My Mac: Hands-On with Verdent's Standalone App

Jimmy Song — Sun, 04 Jan 2026 02:25:48 +0000

I’ve been spending more time recently experimenting with vibe coding tools on real projects, not demos. One of those projects is my own website, where I constantly tweak content structure, navigation, and layout.

During this process, I started using Verdent’s standalone Mac app more seriously. What stood out was not any single feature, but how different the experience felt compared to traditional AI coding tools.

Figure 1: Verdent Standalone App UI

Verdent doesn’t behave like an assistant waiting for instructions. It behaves more like an environment where work happens in parallel.

A Different Starting Point: Tasks, Not Chats

Most AI coding tools begin with a conversation. Verdent begins with tasks.

When I opened my website repository in the Verdent app, I didn’t start with a long prompt. I created multiple tasks directly: one to rethink navigation and SEO structure, another to explore homepage layout improvements, and a third to review existing content organization.

Each task immediately spun up its own agent and workspace. From the beginning, the app encouraged me to think in parallel, the same way I normally would when sketching ideas on paper or jumping between files.

This framing alone changes how you work.

Built for Multitasking, Without Losing Context

Switching contexts is unavoidable in real development work. What usually breaks is continuity.

Verdent handles this well. Each task preserves its full context independently. I could stop one task mid-way, switch to another, and come back later without re-explaining the problem or reloading files.

For example, while one agent was analyzing my site’s navigation structure, another was exploring layout options. I moved between them freely. Nothing was lost. Each agent remembered exactly what it was doing.

This feels closer to how developers think than how chat-based tools operate.

Safe Parallel Coding with Workspaces

Parallel work only becomes truly safe when code changes are isolated. When parallelism moves from discussion to actual code modification, risk management becomes essential.

Verdent solves this with Workspaces. Each workspace is an isolated, independent code environment with its own change history, commit log, and branches. This isn’t just about separation—it’s about making concurrent code changes manageable.

What this means in practice:

Multiple tasks can write code simultaneously
Changes remain isolated from each other
If conflicts arise, they’re visible and cleanly resolvable

I intentionally let different agents operate on overlapping parts of my project: one modifying Markdown content and links, another adjusting CSS and layout logic. Both ran in parallel. No conflicts emerged. Later, I reviewed the diffs from each workspace and merged only what made sense.

This kind of isolation removes significant anxiety from AI-assisted coding. You stop worrying about breaking things and start experimenting more freely, knowing that each change exists in its own contained environment.

Parallel Agent Execution Feels Like Delegation

Parallelism doesn’t mean that all agents complete the same phase of work at the same time—instead, by isolating and overlapping phases, what was once a strictly sequential process is compressed into a more efficient, collaborative mode.

In Verdent, each agent runs in its own workspace, essentially an automatically managed branch or worktree. In practice, I often create multiple tasks with different responsibilities for the same requirement, such as planning, implementation, and review. But this doesn’t mean they all complete the same phase simultaneously.

These tasks are triggered as needed, each running for a period and producing clear artifacts as boundaries for collaboration. The planning task generates planning documents or constraint specifications; the implementation task advances code changes based on those documents and produces diffs; the review task, according to the established planning goals and audit criteria, performs staged reviews of the generated changes. By overlapping phases around artifacts, the originally strict sequential process is compressed into a workflow that more closely resembles team collaboration.

The value of splitting into multiple tasks is not parallel execution, but parallel cognition and clear collaboration boundaries.

While it’s technically possible to put multiple roles into a single task, this causes planning, implementation, and review to share the same context, which weakens role isolation and the auditability of results.

Configurability and Design Trade-offs

Beyond the workflow model itself, Verdent exposes a surprisingly rich set of configurable capabilities.

It allows users to customize MCP settings, define subagents with configurable prompts, and create reusable commands via slash (/) shortcuts. Personal rules can be written to influence agent behavior and response style, and command-level permissions can be configured to enforce basic security boundaries. Verdent also supports multiple mainstream foundation models, including GPT, Claude, Gemini, and K2. For users who prefer a lightweight coding experience without a full IDE, Verdent offers DiffLens as an alternative review-oriented interface. Both subscription-based and credit-based pricing models are supported.

Figure 2: Verdent Settings

That said, Verdent makes a clear set of trade-offs. It is not built around tab-based code completion, nor does it offer a plugin system. If it did, it would start to resemble a traditional IDE - which does not seem to be its goal. Verdent is not designed for direct, fine-grained code manipulation; most changes are mediated through conversational tasks and agent-driven edits. This makes the experience clean and focused, but it also means that for large, highly complex codebases, Verdent may function better as a complementary orchestration layer rather than a full-time development environment.

Where Verdent Fits Today

There are many AI-assisted coding tools emerging right now. Some focus on smarter editors, others on faster generation.

Verdent feels different because it focuses on orchestration, not just assistance.

It doesn’t try to replace your editor. It sits one level above, coordinating planning, execution, and review across multiple agents.

That makes it particularly suitable for exploratory work, refactoring, and early-stage design - exactly the kind of work I was doing on my website.

Final Thoughts

Using Verdent’s standalone app didn’t just speed things up. It changed how I structured work.

Instead of doing everything sequentially, I started thinking in parallel again - and letting the system support that way of thinking.

Verdent feels less like an AI feature and more like an environment that assumes AI is already part of how development happens.

For developers experimenting with AI-native workflows, that shift is worth paying attention to.

2025 Annual Review: The Transformation Journey from Cloud Native to AI Native

Jimmy Song — Wed, 31 Dec 2025 10:02:01 +0000

The waves of technology keep evolving; only by actively embracing change can we continue to create value. In 2025, I chose to move from Cloud Native to AI Native—this year marked a key turning point for personal growth and system reinvention.

2025 was a turning point for me. This year, I not only changed my technical direction but also the way I approach problems. Moving from Cloud Native infrastructure to AI Native Infrastructure was not just a migration of content, but an upgrade in mindset.

Figure 1: Farewell 2025!

This year, I conducted a large-scale refactoring of the website and systematically organized the content. Beyond the technical improvements, I want to share my thoughts and changes throughout the year.

A Bold Shift: Embracing the AI Native Era

At the beginning of 2025, I made an important decision: to reposition myself from a Cloud Native Evangelist to an AI Infrastructure Architect. This was not just a change in title, but a strategic transformation after careful consideration.

As I witnessed the surge of AI technologies and the rise of Agent-based applications reshaping software, I realized that clinging to the boundaries of Cloud Native might mean missing an era. So, I systematically adjusted the website’s content structure, shifting the focus toward AI Native Infrastructure.

This transformation was not about abandoning the past, but extending forward from the foundation of Cloud Native. Classic content like Kubernetes and Istio remains and is continuously updated, but new topics such as AI Agent and the AI Native Landscape have been added, forming a more complete knowledge map.

Content Creation: From Technical Details to Ecosystem Perspective

AI Agent: Building Systematic Knowledge

Agents represent a major evolution in software for the AI era. When I tried to understand Agent design principles, I found fragmented information everywhere but lacked a systematic knowledge base.

So I created content that analyzes the Agent context lifecycle and control loop mechanisms, summarizing several proven architectural patterns. To make complex knowledge easier to digest, I organized it into logical sections so readers can learn step by step.

AI Tool Ecosystem: Mapping the Open Source Landscape

AI tools and frameworks are emerging rapidly, with new projects appearing daily. To help readers quickly grasp the ecosystem, I built a comprehensive AI OSS database.

This database covers everything from Agent frameworks to development tools and deployment services. I not only included active projects but also established an archive mechanism, preserving detailed information on over 150 historical projects. More importantly, I developed a scoring system to objectively evaluate projects across dimensions like quality and sustainability, helping readers decide which tools are worth investing time in.

Blogging: Capturing Technology Trends Faster

In 2025, I wrote over 120 blog posts. Compared to previous years, these articles focused more on observing and reflecting on technology trends, rather than just technical tutorials.

I started paying attention to deeper questions: How will AI infrastructure evolve? What does Beijing’s open source initiative mean for the AI industry? What ripple effects might a tech acquisition trigger? These articles allowed me and my readers to not only see “what” technology is, but also “why” and “what’s next.”

User Experience: Making Knowledge Easier to Discover and Consume

No matter how good the content is, if it can’t be easily found and read, its value is greatly diminished. In 2025, I invested significant effort into website functionality, with one goal: to provide readers with a smoother reading experience.

Comprehensive Search Upgrade

As the volume of content grew, the original search function could no longer meet demand. I redesigned the search system to support fuzzy search and result scoring, and optimized index loading performance. More importantly, the new search interface is more user-friendly, supporting keyboard navigation and category filtering so users can find what they want faster.

Multi-Device Experience Optimization

Mobile reading experience has improved significantly. I refactored the mobile navigation and table of contents, making reading on phones much smoother. Dark mode is now more refined, fixing several display issues and ensuring images and diagrams look good on dark backgrounds.

Efficiency Revolution in Content Distribution

A major change was optimizing the WeChat Official Account publishing workflow. Previously, publishing website content to WeChat required manual handling of many details; now, it’s almost one-click export. This workflow automatically processes images, metadata, styles, and all details, reducing a half-hour task to just a few minutes.

Additionally, I added a glossary feature for technical term highlighting and tooltips; improved SEO and social sharing metadata; and cleaned up outdated content. These seemingly minor improvements quietly enhance the user experience.

Content Evolution: More Dimensional Knowledge Expression

Looking back at content creation in 2025, I found clear changes in several dimensions.

From Tutorials to Observations

Early content leaned toward technical tutorials and practical guides, showing “how to do.” This year, I focused more on “why” and “what are the trends.” I wrote more technology trend analyses, ecosystem maps, and in-depth case studies. These may not directly teach you how to use an API, but they help you understand the direction of technological evolution.

From Chinese to Bilingual

AI is a global wave and cannot be limited to the Chinese-speaking world. In 2025, I wrote bilingual documentation for almost all new AI tools, and important blog posts also have English versions. This increased the workload, but allowed the content to reach a broader audience.

From Text to Multimedia

Text is efficient, but not all knowledge is best expressed in words. This year, I used many architecture and schematic diagrams to explain complex concepts, adding 59 new charts. These visual elements lower the barrier to understanding, making abstract concepts more intuitive. I also optimized image display in dark mode to ensure consistent visual experience.

Development Approach: Embracing AI-Assisted Programming

2025 was not only a year of shifting content themes toward AI, but also a year of deep practice in AI-assisted programming.

I developed a VS Code plugin and created many prompts to automate repetitive tasks. I experimented with various AI programming tools and settled on a toolchain that suits me. I even migrated the website to Cloudflare Pages and used its edge computing services to develop a chatbot. These practices greatly improved development efficiency, giving me more time to focus on thinking and creating rather than mechanical coding.

This made me realize: AI will not replace developers, but developers who use AI well will replace those who do not. I also shared more insights to help others master AI-assisted programming.

Looking Ahead to 2026: Keep Moving Forward

Looking back at 2025, the site underwent a profound transformation—from a Cloud Native tech blog to an AI infrastructure knowledge base. But this is just the beginning, not the end.

Looking forward to 2026, I plan to continue deepening in several areas:

Enhancing the knowledge system: Continue to supplement GPU infrastructure and AI Agent content, especially practical cases and performance tuning knowledge.
Tracking ecosystem evolution: AI tools and frameworks iterate rapidly; I need to keep up with this fast-changing ecosystem and update content in a timely manner.
Deepening engineering practice: Share more practical AI engineering experience to help readers turn theory into practice.
Exploring knowledge connections: Consider building a knowledge graph to connect different content sections, providing smarter navigation and recommendations.

Summary

2025 was a year of change and growth. From Cloud Native to AI Native, from technical practice to ecosystem observation, both the content and functionality of the site have made qualitative leaps.

What makes me happiest is that this transformation allowed me and my readers to stand at the forefront of the technology wave. We are not just learning new technologies, but thinking about how technology changes the world and the way we write software.

The waves of technology keep evolving; only by actively embracing change can we continue to create value. Thank you to every reader for your companionship and support. I look forward to sharing more insights and practices in 2026.

Further Reading:

AI Native Landscape
2025 Blog Posts

The Butterfly Effect After Manus Was Acquired by Meta

Jimmy Song — Tue, 30 Dec 2025 03:30:51 +0000

The success or failure of AI applications often lies not in the technology itself, but in the ability to scale delivery and create a closed loop.

Figure 1: The Butterfly Effect After Manus Was Acquired by Meta

When “Those Who Discuss It” Are Not “Those Who Pay for It”

On December 30, 2025, a piece of news went viral: Manus was acquired by Meta for billions of dollars (Manus Joins Meta for Next Era of Innovation). This startup, founded in China and under pressure from tech giants since its inception, completed a whirlwind journey in less than a year—from explosive growth, relocating to Singapore, to being acquired by a global giant.

According to Manus’s official statement, its products and subscriptions will continue to be available via the app and website, and the company will remain operational in Singapore. The team will join Meta to provide general Agent capabilities for Meta’s consumer and enterprise products (including Meta AI).

Rather than focusing on “who won,” I’m more interested in the chain reaction this event triggered: it activated completely opposite judgment systems among different groups, and this split is reshaping the growth paths and strategies for AI applications and startups.

Two Public Opinion Arenas: Blessings and Doubts Coexist

After Manus was acquired, the mainstream sentiment in social circles was one of congratulations and excitement. Many saw it as a stellar example of a Chinese team going global—achieving remarkable results in the most competitive field in a very short time.

Meanwhile, the comment sections of public accounts became “venting valves for counter-narratives,” with skepticism centering on three main points:

Whether the technology has real barriers (e.g., “there are countless similar products,” “it’s not hard for big companies to build their own”).
Valuation and bubble concerns (e.g., “another case of the AI bubble”).
Distrust in the buyer’s judgment (e.g., “giants making desperate bets,” “history repeating itself”).

This divergence isn’t about who understands AI better, but about different evaluation frameworks: social circles focus on “trajectory and outcome,” while comment sections focus on “legitimacy and worthiness.”

Many people are impressed by Manus’s marketing buzz and controversies, which can lead to skepticism. But if it achieved a “strict $100M ARR” in 10 months, one fact is clear: its revenue doesn’t depend on broad consensus, but comes from a highly concentrated group of global users with strong willingness to pay.

Manus’s core user profile is closer to “individuals as production units,” including freelancers, indie developers, independent researchers, and key deliverers in small and medium businesses. They don’t care about debates over “wrapping” or not; they care about “can I deliver end-to-end tasks,” and “can this help me hire one less person, work fewer late nights, or avoid juggling ten tools.”

This leads to a counterintuitive phenomenon: those who discuss the most may not pay, while those who pay steadily are often silent.

For these users, tools are not identity badges—they are profit levers.

Three Lessons for Entrepreneurs: The Growth Paradigm in the AI Application Era Has Changed

Based on the above, the Manus case offers three lessons for entrepreneurs:

Growth No Longer Equals Positive Reviews

AI applications can commercialize first and build consensus later. Public opinion can remain divided for a long time, but cash flow doesn’t wait for unified recognition.

“Heavy Marketing” Is Becoming a Capability, Not a Stigma

As foundational models and capabilities spread rapidly, differentiation is quickly erased. Being seen, understood, and paid for is itself part of the moat. Not all marketing deserves respect, but “distribution and mindshare” have become unavoidable battlegrounds for AI applications.

Globalization Is No Longer a Bonus, but May Be a Survival Strategy

From payment willingness, compliance boundaries, talent density to valuation systems, market structure means many teams “can only complete the loop overseas.” It’s not romantic, but it’s reality.

A Personal Reflection

As someone long engaged in cloud native and AI infrastructure, I’m used to evaluating products by their “technical barriers.” But cases like Manus remind me: at the AI application layer, barriers may not first appear in models or code, but often in organizational speed, productization capability, delivery loop, and distribution efficiency.

When a system can reliably turn “capability” into “results,” it has built a commercial moat—even if its tech stack doesn’t meet outsiders’ ideals of “purity.”

The biggest butterfly effect of Manus being acquired by Meta may not be the deal itself, but making more entrepreneurs realize: in the AI era, the winning move is shifting from “what model you use” to “whether you can deliver results at scale.”

Summary

The acquisition of Manus by Meta is not just a convergence of capital and technology, but also a microcosm of the changing growth paradigm in the AI application era. For entrepreneurs, understanding and mastering “user structure,” “distribution capability,” and “global closed loops” will be key to future competition.

AI Infra Open Source in China: Analysis of Beijing and Shanghai's Plans

Jimmy Song — Thu, 25 Dec 2025 10:01:13 +0000

Institutionalized open source marks a new starting point for China’s AI Infra, but true breakthroughs and risks lie in the engineering and governance details.

Perspective on Beijing and Shanghai’s Open Source Plans

Using the simultaneous release of open source ecosystem plans by Beijing and Shanghai as a lens, and drawing on China’s past foundation practices and international open source governance experience, this article explores the real opportunities, structural constraints, and potential risks as AI Infrastructure (AI Infra, Artificial Intelligence Infrastructure) enters a new phase of institutionalized open source.

Figure 1: Beijing and Shanghai successively launch open source ecosystem construction plans

Why Compare Beijing and Shanghai Together

It is rare for me to write an article solely because of a local policy document. However, during Christmas, both Beijing and Shanghai’s Bureaus of Economy and Information Technology released their respective open source ecosystem construction plans:

This time, the fact that both cities released their plans on the same day sends a signal worth serious attention: China is attempting to advance open source in a more systematic and institutionalized way, especially regarding open source capabilities related to AI Infra.

If you only look at Beijing’s plan, it is easy to interpret it as a local industrial policy upgrade. But when you consider both Beijing and Shanghai’s plans together, it looks more like a clearly defined “dual-center structure.”

The question is no longer whether to develop open source, but:

In the AI era, what institutional forms, engineering paths, and governance models will open source take?

Open Source as “Industrial Infrastructure Engineering”

Both Beijing and Shanghai’s plans reflect a highly consistent judgment:

Open source is no longer seen as a spontaneous community activity, but as an industrial infrastructure capability that requires systematic construction.

This is especially evident in the field of AI Infra.

Issues such as computing power scheduling, model evaluation, toolchains, data elements, license compliance, and supply chain security—previously hidden in “engineering details”—are now systematically incorporated into policy language for the first time. This at least shows that decision-makers have realized:

AI competition is not only about model parameter scale
It is even more about toolchains, infrastructure, evaluation systems, and engineering capabilities
These capabilities are naturally more suitable for building public foundations through open source

In this respect, Beijing and Shanghai are highly aligned.

Two Open Source Paths: Infra vs. Platform

When we zoom in, the differences between the two plans become clear.

Beijing: “Foundation-Oriented” Open Source Path for AI Infra

Beijing’s plan focuses on:

Heterogeneous computing power scheduling
Model evaluation toolchains
Data elements and data governance
RISC-V software-hardware collaboration
SBOM, license compatibility, open source compliance
Supply chain security and industrial resilience

This is a typical perspective of “treating AI as an infrastructure problem.”

It is less concerned with the number of projects or community size, and more with:

Whether reusable engineering capabilities can be formed
Whether these can be trusted by industry and government over the long term
Whether they can stand up to scrutiny in terms of security, compliance, and governance

To some extent, Beijing is answering the question:

How can open source become a “governable, auditable, and scalable public capability”?

Shanghai: “Scale and Internationalization” Path for AI Platform

In contrast, Shanghai’s plan has a different focus:

Building an international open source community for artificial intelligence
Covering the entire platform chain from development, training, testing, hosting, to operation
Overseas sites, multilingual support, international activities
Resource linkage through computing vouchers and model vouchers
“Open source platform first release / global simultaneous release” dual-release mechanism
Clear targets for community, enterprise, and developer scale

Shanghai cares more about:

How open source can achieve scale effects
How it can support the growth of commercial enterprises
How it can be seen and adopted globally

This is a path of “treating open source as a global digital product and platform capability.”

Together: A Complete but Tension-Filled Structure

When viewed together, Beijing and Shanghai’s plans form a more complete picture:

Beijing is responsible for “making open source solid,” while Shanghai is responsible for “taking open source global.”

Structurally, this is a clear division of labor:

Beijing focuses on institutions, governance, and foundational capabilities
Shanghai focuses on community, commercialization, and international communication

These two paths are not in conflict; in theory, they are even complementary. The real question is whether they can form positive feedback in practice, rather than operating in silos.

Cautious Attitude Toward “Institutionalized, Platformized Open Source”

Precisely because both plans are so “systematic,” I am even more cautious.

The reason is simple: this is not China’s first attempt to promote open source through foundations, associations, or platforms.

Over the past decade, we have seen similar paths repeatedly, and recurring structural problems:

The difficulty of establishing neutrality and multi-party trust is extremely high
There is a huge gap between showcase metrics (quantity, activities, certifications) and ecosystem strength
Commercialization and long-term maintenance mechanisms are hard to sustain

These problems will not disappear just because the plans are more comprehensive.

Four Risks to Watch Under the Dual Plans

If we are to “listen to their words and watch their actions,” I would focus on the following four risks:

Will Metrics Hijack Engineering Reality

When “internationally influential projects,” “star projects,” and “first-release projects” become hard metrics, will this induce packaging, migration, and short-term hype, rather than truly solving engineering problems?

Will It Slide Toward Platform Centralism

The long-term pattern of AI Infra is closer to a model that prioritizes protocols, standards, and interoperability. If it eventually evolves into “a few platforms concentrating resources and discourse power,” it may be efficient in the short term but will suppress external participation and international collaboration in the long run.

Is Internationalization Underestimated as an “Operational Issue”

True international collaboration is never just about language, sites, or events; it also involves governance structures, compliance boundaries, and supply chain trust.

Will Application Demonstrations Become One-Off Projects

If “first plans” and “computing vouchers” are just procurement tactics without continuous iteration and community feedback mechanisms, the long-term benefit to the ecosystem will be very limited.

What Are the “Hard Results” of AI Infra Open Source After Three Years

If we review the success of this round of institutionalized open source after three years, I would look for three types of results:

Whether de facto standards and interoperable ecosystems have emerged, including scheduling interfaces, evaluation benchmarks, Agent tool invocation protocols, and observability semantics.
Whether compliance and supply chain security have become public capabilities—SBOM, license compatibility, vulnerability monitoring—truly productized and service-oriented.
Whether a sustainable maintenance business mechanism has been established, allowing core maintainers to stay long-term, rather than relying on passion and subsidies.

If I were to use a North Star metric to measure the success of these plans, it would be the emergence of several outstanding open source commercial companies rooted in China and serving the world.

Summary

The open source ecosystem plans of Beijing and Shanghai mark a new phase of institutionalization and engineering for AI Infra open source in China. Over the next three years, the real achievements will not be about meeting targets, but about forming sustainable engineering capabilities, de facto standards, and maintenance mechanisms. Only through continuous participation and practice can open source become the public foundation of AI infrastructure.

References

From 2025 Onwards, Software Engineering Shifts from Code-Centric to Runtime and Cost-Centric

Jimmy Song — Wed, 24 Dec 2025 14:59:11 +0000

In 2025, the core of software engineering is no longer just about code itself, but about runtime controllability and cost governance. This shift is fundamentally reshaping the industry’s underlying logic.

Looking back at 2025, I became increasingly aware that this year was not about “code becoming unimportant,” but rather that the value coordinates of engineering have shifted as a whole. For more than a decade, software engineering has focused on code quality, architectural evolution, and delivery efficiency. But starting in 2025, the key to system success is shifting—towards whether the runtime is controllable and whether costs are governable.

This is not just a slogan, but a conclusion repeatedly validated by my real-world experiences throughout the year.

My 2025: From “Platform Engineering” to “Runtime Challenges”

In my annual review, I noted a clear change: I spent less time on “how to write a good system,” and more time on “how to keep the system running stably, reliably, and affordably.”

This shift in focus is a natural extension of a decade of cloud native evolution.

The following timeline diagram illustrates how my focus has changed over recent years:

Figure 1: My Focus Shift Timeline

My focus shifted from cloud native platform engineering to LLM application engineering, then to AI infrastructure, and finally to Agentic Runtime with governance and cost control.

When AI workloads truly enter business scenarios, the core challenges engineers face also change:

Are inference, training, and evaluation competing for the same compute pool?
Is GPU utilization consistently below expectations?
Does cost scale linearly and uncontrollably with concurrency?
Does the system have failure isolation and replay capabilities?

These issues go far beyond the code level.

Industry Consensus: AI Is Shifting the Focus of Engineering

By 2025, an industry consensus is emerging: AI is rewriting software engineering. But the real change is not happening in the IDE or code completion speed—it is reflected in the migration of engineering complexity.

Previously, complexity was concentrated in code and interfaces, and problems were solved through abstraction, refactoring, and testing.

Now, complexity has shifted to the runtime, resource, and cost layers, and must be addressed through scheduling, isolation, observability, and governance.

This is why the same AI tools:

Serve as “accelerators” for junior engineers
But act as “magnifiers” for senior engineers

AI tools amplify whether you truly understand how systems run in production.

Why “Cost” Becomes a First Principle

In traditional cloud native systems, low CPU utilization is often just an efficiency issue; but in AI systems, low GPU utilization is often a cash flow problem.

In 2025, I repeatedly encountered scenarios like:

Resources “seem insufficient,” but utilization is not actually high
Scaling up to solve queuing issues ends up increasing unit costs
The system lacks clear budget and quota boundaries, so throttling becomes the only way to stop the bleeding

The root cause of these phenomena is not model selection, but the lack of a runtime and cost control plane tailored for AI workloads.

The following flowchart visually illustrates the cyclical relationship between GPU resources and cost pressures:

Figure 2: GPU Resource and Cost Cycle in AI Systems

In AI systems, limited GPU supply leads to queuing and waiting, which causes throughput to drop. Attempts to solve this through blind scaling only increase unit costs and create budget pressure, ultimately forcing the adoption of finer scheduling and governance strategies.

Engineering problems ultimately manifest as cost issues.

The Rise of Agents: The Real Challenge Is at Runtime

In 2025, Agent (Intelligent Agent, Agent, Intelligent Agent) became a hot topic; by 2026, it will enter the “can it actually run” stage.

The challenge for Agents has never been about “how smart they are,” but rather:

Whether there are clear permission and data boundaries
Whether they run in an isolated execution environment
Whether they can be observed, evaluated, and replayed
Whether they are subject to explicit cost and budget constraints

These capabilities form the outline of Agentic Runtime (Agentic Runtime, Intelligent Agent Runtime) that I have been trying to clarify throughout the year.

The following flowchart shows the core capability layers of Agentic Runtime:

Figure 3: Agentic Runtime Capability Layers

Agentic Runtime builds from the foundation of Agents and workflows, connecting through orchestration and tool protocols, with the runtime managing state, memory, and evaluation. It provides secure execution environments (Sandbox and Policy), and ultimately implements a resource and cost control plane that unifies GPU, quota, and billing management.

Without a runtime, an Agent is just a demo; without cost constraints, an Agent is just a risk amplifier.

Outlook for 2026: The “Foundation” of Engineering Matters Again

Looking ahead to 2026, I remain cautiously optimistic.

I do not believe the future belongs to “those who write the best prompts,” but more likely to:

Those who understand runtime boundaries
Those who can govern compute as a constrained resource
Those who design AI systems as long-running systems

From 2025 onwards, software engineering is no longer code-centric, but runtime and cost-centric. This is not a regression, but a return: a return to being responsible for the whole system and for real-world constraints.

For me personally, this is both a year-end summary and the direction I will continue to invest in for the coming years.

Summary

In 2025, the focus of software engineering has shifted from code itself to runtime and cost governance. The rise of AI and Agents has not diminished the value of engineering, but has pushed complexity to a higher level. In the future, understanding runtime, managing compute and cost will become the new core competencies for engineers. I hope this year-end review provides some inspiration and reflection for fellow professionals.

From Cloud Native to AI Native: Why Kubernetes Is the Foundation for Next-Gen AI Agents

Jimmy Song — Wed, 24 Dec 2025 12:25:52 +0000

As a long-time practitioner in the cloud native field, I am increasingly convinced of one thing: AI Agents are not just a change in application form, but a migration of infrastructure paradigms.

As artificial intelligence evolves from demos and copilots to systems that truly take on tasks and responsibilities, AI Agents are becoming the new execution units in enterprise IT architectures. They not only “think,” but also act: they can invoke tools, access systems, and collaborate to achieve goals.

This raises an important question:

What kind of infrastructure should such systems run on?

In my view, Kubernetes remains a solid choice for large-scale scenarios—but only if we reimagine Kubernetes in an AI-native way.

Cloud Native Challenges for Production-Grade AI Agents

In real production environments, AI Agents expose infrastructure needs that are fundamentally different from traditional microservices. Agents are not “just another HTTP service”; they have three distinct characteristics:

Behavior is non-deterministic (driven by model inference)
Execution paths are dynamic (tool invocation cannot be fully enumerated in advance)
Decisions must be auditable, constrained, and reviewable

If we simply apply existing cloud native infrastructure, we quickly hit bottlenecks.

The following table summarizes the main challenges and risks AI Agents face in cloud native environments:

Challenge Category	Real Needs of Agents	What Happens If Missing
Policy & Security	Dynamic control of tool and data access based on context, identity, and task	Agents have “superuser” privileges, risks are uncontrollable
Observability	Not just “did it succeed,” but also why was this decision made	Hard to debug, hard to review, hard to hold accountable
Governance & Consistency	Platform-level guardrails enforce organizational policies	Each Agent could become a “shadow AI”

Table 3: Challenges and Risks for AI Agents in Cloud Native Environments

All these issues point to one conclusion:

AI Agents must be treated as first-class citizens in Kubernetes, not just ordinary workloads.

Core Architecture: Making Agents Native Kubernetes Objects

Looking back at the evolution of cloud native technologies, we’ve gone through similar stages:

Physical machines → Virtual machines
Virtual machines → Containers
Containers → Microservices
Microservices → Declarative, governable platforms

AI Agents are simply the next step.

A production-ready AI Agent architecture requires at least three layers:

Agent Orchestration Layer: Declaratively define Agents
Tool Service-ization Layer (MCP Services): Turn capabilities into governable services
AI Native Data Plane / Gateway: Unify policy, security, and protocols

Agent Orchestration Layer: Declarative Agent Management

Agents should no longer be “runtime objects” inside an SDK—they should be managed like Pods or Deployments.

Key concepts:

Agents as Kubernetes Resources

Agents are defined using CRD (CustomResourceDefinition)
Lifecycle managed via kubectl or GitOps
Agent models, tools, and policies are all explicitly declared

A typical Agent definition includes:

Agent logic (inference loop)
Model configuration (specifying which large language model to use)
Callable toolset

This closely mirrors how we once decomposed “applications” into Deployments, Services, and ConfigMaps.

Tool Service-ization Layer: MCP Services Are Essential

In Agent architectures, tools are where real “actions” happen.

Early MCP tools were often:

Local processes
Tightly coupled to a single Agent
Lacking versioning, permissions, and auditing

This is unsustainable in enterprise environments.

The Essence of MCP Service-ization

Tools → Remote services
Services → Kubernetes native workloads
Capabilities → Reusable, governable, auditable

This step is fundamentally similar to how we once turned scripts into microservices.

AI Native Gateway: The “Control Plane Entry” for the Agent World

As the number of Agents grows and tools/models diversify, connectivity itself becomes a system risk.

Traditional API Gateways do not understand scenarios like:

MCP
Agent-to-Agent (A2A) communication
Model invocation context

Thus, we need an AI native gateway dedicated to mediation and governance.

It must understand at least three types of traffic:

A2T: Agent → Tool
A2L: Agent → LLM
A2A: Agent ↔ Agent

And enforce, across these paths:

Identity and authorization
Policy and guardrails
Auditing and rate limiting

Architecture Overview

The diagram below illustrates the core layers and traffic paths of an AI-native system on Kubernetes:

Figure 3: AI Native Architecture Layers and Traffic Paths

Summary

AI Agents do not negate cloud native; on the contrary:

AI Agents are the natural extension of cloud native in the era of intelligence.

Declarative → Agent definitions
Service → MCP Services
Service Mesh → AI Native Gateway

If Kubernetes is the “automated factory,” then AI Agents are the intelligent workers who actually get things done.

And the AI native gateway is the security and governance system tailored for these intelligent workers.

This is not an optional architecture—it is the only path for AI to reach production.

AI Open Source Landscape: A One-Stop Guide to AI Project Navigation and Scoring System

Jimmy Song — Tue, 23 Dec 2025 08:34:05 +0000

The AI Open Source Landscape is not just a project directory, but an innovative attempt to bring transparency and quantifiability to the AI open source ecosystem.

Note: This article is intended for general readers and focuses on platform features and usage scenarios. If you want to see the technical details and formulas behind the scoring, please refer to: AI Project Scoring and Inclusion Criteria.

Project Background and Positioning

The AI Open Source Landscape aims to provide developers, researchers, and enterprise users with a one-stop navigation and evaluation platform for AI open source projects. With the rapid development of large language models (LLM, Large Language Model), multimodal models (Multimodal Model), and other AI technologies, the open source community has seen a surge of innovative projects. However, information is scattered and quality varies, making it difficult for users to filter and make decisions.

Figure 1: AI Open Source Landscape

The AI Open Source Landscape systematically collects mainstream AI open source projects. As of the time of writing, it has included 851 open source projects. This landscape combines a multi-dimensional scoring system to help users efficiently discover, compare, and select the most suitable AI tools and frameworks for their needs. The platform not only focuses on models themselves, but also covers datasets, inference engines, evaluation tools, application frameworks, and the entire ecosystem chain, striving to promote transparency, quantifiability, and sustainable development in the AI open source ecosystem.

Main Interface and Feature Highlights

The platform homepage presents project distribution in both landscape and list views, supporting category filtering, keyword search, and tag navigation to help users quickly locate target projects.

Figure 2: Open Source Project Detail Page

For general readers, the main experience points include:

Card view: One-sentence overview, star rating, and overall score for quick browsing and comparison.
Health card: Displays overall health and key dimensions (activity, community, influence, sustainability) on the project page or sidebar, with the latest update marked for easy assessment of maintenance status.
Detail page: Provides more background information, project links, and application scenarios to help you evaluate suitability for your needs.
Smart badges: Visually display labels such as “Active”, “New Project”, “Popular”, “Archived” on cards, helping you quickly capture key project features.

If you are interested in the specific rules for badge determination or scoring, detailed explanations are available on the Scoring Rules Page.

Scoring and Ranking Mechanism

The platform uses multi-dimensional scores to reflect the overall health and popularity of projects. The main dimensions include: Activity, Community, Quality, Sustainability, and the comprehensive Health score. These scores help you quickly judge whether a project is suitable for production or experimentation.

Data Sources and Update Mechanism

The platform’s data mainly comes from GitHub, project lists, official documentation, and community recommendations. We regularly and automatically synchronize and update metrics to ensure that the “last updated” and scores displayed on the interface reflect the current maintenance status of projects. Projects that have not been updated for a long time or are determined to be “inactive” are moved to the Archived Page. Archived projects remain searchable and retain historical scores, but will not appear in the default view of active rankings, making it easier for readers to focus on projects that are still maintained and active.

For general readers, the key points are:

The page displays key metrics and “last updated” time, helping you quickly judge whether a project is still maintained.
The AI Open Source Landscape continuously iterates on the scoring model to improve fairness and differentiation.

How to Contribute and Correct Data

If you want a project to be included or its data updated, you can:

Submit an AI Open Source Project Inclusion Request
Keep the project’s README, License, documentation, and other information complete in the repository to facilitate our data collection and assessment.
For faster synchronization or if you encounter data issues, contact the maintainers via project issues or raise a request in the site discussion area.

Typical Use Cases or User Feedback

The AI Open Source Landscape has been widely used in various scenarios such as AI developer selection, enterprise technology research, and academic studies. For example:

Developers can quickly filter models or tools that meet their needs through the platform, saving significant research time.
Enterprise technical teams use the ranking lists for competitor analysis and technology planning.
Educational and research institutions refer to the landscape to understand trends in the AI open source ecosystem, supporting course design and topic selection.

Some users have commented that the platform is “comprehensive, well-structured, and fair in scoring,” greatly improving the efficiency of AI project selection and learning. Community suggestions continue to drive ongoing improvements in platform features and content.

Summary

The AI Open Source Landscape systematically and quantitatively organizes the AI open source ecosystem: the backend worker is responsible for reliable data collection and scoring calculations (supporting backfill and migration), while frontend components handle fast rendering and visualization (including smart badges, health cards, and metric explanations).

If you want to learn more about the scoring details or participate in improvements:

The community is welcome to join in evaluation, backfilling historical data, and refining scoring rules, working together to make the AI open source ecosystem more transparent and sustainable.

AI 2026: Infrastructure, Agents, and the Next Cloud-Native Shift

Jimmy Song — Fri, 19 Dec 2025 03:54:31 +0000

The real turning point for AI in 2026 is not autonomy, but the maturity of infrastructure - where agentic runtimes, GPU efficiency, and organizational design will decide who wins.

Introduction: 2026 Is Not an AI Moment, It Is an Infrastructure Moment

Over the past fifteen years, every major shift in software has followed a familiar arc. Microservices were adopted not out of love for distributed systems, but because monoliths reached organizational limits. Kubernetes succeeded not because containers were novel, but because infrastructure finally matched how teams operated. Cloud native was never about YAML—it was about operability at scale.

AI now stands at a similar inflection point.

The central question for 2026 is not whether models will become more autonomous. That debate overlooks the core issue. Instead, the real question is whether AI can become operable, governable, and economically sustainable within real systems.

Most organizations today are limited not by intelligence, but by infrastructure: inefficient GPU utilization, escalating inference costs, fragile agent demos, and a tendency to treat AI as a feature rather than a runtime. The next phase of AI will be shaped not by model breakthroughs, but by the maturity of AI infrastructure and its ability to absorb responsibility.

From Automation to Capability Multiplication — A Familiar Cloud-Native Pattern

Reflecting on early cloud adoption, the dominant narrative was cost reduction: fewer servers, lower CapEx, elastic scaling. Yet, the true payoff emerged later, when teams realized cloud enabled entirely new operating models.

AI is repeating this pattern.

The following diagram illustrates the shift from automation to capability multiplication.

Figure 1: From Automation to Capability Multiplication

The first wave of AI focused on labor replacement. The second wave reframes AI as capability multiplication: the same team, observing more signals, covering broader areas, and acting sooner.

This mirrors the evolution of monitoring, tracing, and SRE practices. Rather than reducing engineers, these systems enabled continuous observation instead of occasional sampling.

Preemptive AI systems—monitoring every interaction, log, and signal—are only viable if the underlying infrastructure can support them. This exposes a critical constraint: AI capability scales faster than AI infrastructure.

Without efficient scheduling, isolation, and utilization, multiplying capability simply multiplies cost.

Agents Are Becoming Distributed Systems, Whether We Admit It or Not

The industry often discusses agents as products. In reality, agents are evolving into distributed systems.

The diagram below highlights this architectural shift.

Figure 2: Agents Are Becoming Distributed Systems

Single-agent designs resemble early monoliths: impressive demos, fragile behavior, and opaque failure modes. As tasks grow in complexity, systems must decompose work into planning, execution, verification, and review—making coordination inevitable.

This is not merely a philosophical change, but an architectural one.

Multi-agent systems introduce challenges familiar from the microservices era:

Coordination and orchestration
Resource contention
Fault isolation
Observability and rollback
Deterministic artifacts between stages

Labeling this as “multi-agent collaboration” can be misleading. What is actually occurring is workload decomposition and control-plane emergence. Agents are transitioning from tools to workloads competing for limited resources.

Recognizing this clarifies why agent progress is inseparable from infrastructure maturity.

AI Infra Is the Missing Layer Between Models and Organizations

Cloud native taught us that abstractions only scale when a control plane exists.

Currently, AI lacks a mature control plane.

The following image demonstrates the gap between models and organizations.

Figure 3: AI Infra Is the Missing Layer Between Models and Organizations

Models are powerful, but the surrounding infrastructure—scheduling, isolation, quota enforcement, cost attribution, observability—remains primitive, especially at the GPU layer.

GPUs are expensive, scarce, and often underutilized. In many environments, utilization remains below 30–40%, while inference costs continue to rise. Training pipelines monopolize resources, inference workloads spike unpredictably, and organizations must choose between waste and throttling innovation.

This is not a model problem. It is fundamentally an AI infrastructure problem.

The next phase of AI will depend on treating GPUs as we learned to treat CPUs:

Fine-grained allocation
Fair sharing
Preemption and prioritization
Clear ownership and accounting

Until GPU utilization becomes a primary design goal, AI systems will remain economically fragile.

Domain Expertise Matters Because Infrastructure Finally Exposes It

As models plateau in general reasoning, differentiation shifts elsewhere.

The diagram below illustrates how infrastructure exposes domain expertise.

Figure 4: Domain Expertise Matters Because Infrastructure Finally Exposes It

In cloud-native systems, competitive advantage eventually moved from frameworks to operational excellence: superior runbooks, incident response, and cost control. AI is following a similar trajectory.

High-value AI systems must operate within dense, rule-heavy domains such as finance, healthcare, manufacturing, and infrastructure operations. What matters is not abstract intelligence, but the ability to encode domain constraints, exceptions, and failure patterns.

Here, domain experts become central—not as prompt engineers, but as system shapers. Their decisions define agent permissions, human intervention points, and error containment strategies.

Infrastructure determines whether this expertise can be safely operationalized.

Simulation Is Becoming the New Staging Environment for AI

One of the most important lessons from cloud-native operations: distributed systems are not tested in production.

AI systems that act, plan, and modify state are no exception.

The following image shows simulation as the new staging environment.

Figure 5: Simulation Is Becoming the New Staging Environment for AI

Training and validating agents directly in live environments is unsustainable. The future lies in simulation-first AI development—sandboxed environments that mirror real systems, workloads, and constraints.

This approach is analogous to staging clusters, chaos engineering, and load testing, but elevated for decision-making systems. Evaluation shifts from static benchmarks to behavioral metrics: intervention rates, rollback frequency, and cost impact.

Organizations that build these environments will advance faster and safer. Those that do not may remain limited by conservative deployments and restricted autonomy.

Summary

Technological revolutions succeed not on novelty alone, but when infrastructure, tooling, and organizational models align.

AI is nearing that pivotal moment.

The leaders in 2026 will be those who:

Treat AI as a runtime, not just a feature
Optimize for resource efficiency, especially GPUs
Recognize agents as distributed systems
Redesign organizations around continuous learning systems
Invest in infrastructure ahead of autonomy

AI is no longer just a model problem. It is an infrastructure challenge—and the next phase will be decided not in labs, but in production systems.

What I Saw at COSCon'25: The Real State of Open Source in China

Jimmy Song — Thu, 18 Dec 2025 06:14:51 +0000

Attending COSCon'25 in Beijing, I observed firsthand how open source in China is shifting: AI is now the default context, discussions are grounded in real engineering, and the community is embracing long-term thinking. These are not just trends—they are the new reality.

In early December this year, I attended COSCon'25, the China Open Source Annual Conference, in Beijing. Although I have worked in open source for many years, this was my first time participating in an event organized by the Open Source Society—and I joined as a sub-forum producer. Previously, I thought such conferences were too high-level or disconnected from reality, but after actually taking part, I found there was much to gain.

A quick note: this article is not an official conference summary or review. The organizers have already published detailed information about the event’s scale, attendee numbers, and forum sessions. If you’re interested in those details, please refer to the official article: COSCon'25: The 10th China Open Source Annual Conference Successfully Concludes in Beijing—A Comprehensive Recap!

Figure 1: 10th COSCon Venue

What I want to share is this: Standing on site, on the engineering front lines, and as an organizer rather than an audience member, I saw real changes happening in Chinese open source.

This COSCon: No More Trying to “Prove Open Source Matters”

One clear impression: Almost no one spent time arguing “why do open source” anymore.

In earlier years, common narratives included:

Is open source safe?
Can open source be commercialized?
Can China create its own open source projects?

But at COSCon'25, these questions were basically assumed as “background conditions.” The focus shifted to those already doing open source, and what comes next.

This doesn’t mean the issues have disappeared, but it does mean:

In China’s engineering circles, open source is no longer a “philosophical choice”—it’s a practical way of working.

AI as Background Noise, Not the Main Character

The theme of this year’s conference was Open Source × Open Intelligence, but interestingly, AI did not take center stage.

Instead, it was more like background noise— Almost every topic touched on AI, but no one was giving talks solely “about AI.”

You would see it repeatedly in areas like:

Cloud native scheduling, focusing on GPU / NPU / heterogeneous resources
Storage and data, focusing on data paths for training and inference
Serverless, focusing on LLM cold starts and elasticity
Observability, focusing on what to do when system complexity gets out of hand

AI was not treated as a “hot trend,” but as a new workload reality. This is a significant change, though not one easily captured in press releases.

Real Impressions as a Cloud Native Sub-forum Producer

I helped organize the cloud native open source sub-forum at this year’s conference. This role gave me a perspective very different from that of a typical attendee.

First, Topics Clearly Converged on “Engineering Problems”

There were almost no talks about Kubernetes concepts; Very few about “architectural philosophies.”

Instead, the focus was on:

What pitfalls did you encounter at what scale?
Why did you choose this solution over another?
Which problems remain unsolved?

Many presentations weren’t “pleasant to hear,” but they were very real.

Second, The Boundary Between Academia and Industry Is Thinning

This was especially evident this year.

Some talks from universities and research institutes were no longer just “from a paper’s perspective,” but directly addressed core issues in industrial systems, such as:

Cold start of serverless LLMs
The real value of RDMA in inference paths
Whether prefill/decode separation is truly feasible in engineering

These topics may not be immediately applicable, but they are now colliding head-on with engineering problems, rather than talking past each other.

Third, Open Source Is No Longer Just About Code

In many discussions, “governance,” “maintenance cost,” and “community collaboration” came up frequently.

This is a signal: When a project is truly being used, code is no longer the hardest part.

Main Forum: More Questions, Not Answers

If I had to sum up the main forum in one sentence: It kept raising questions, but wasn’t in a hurry to provide answers.

For example:

Has the boundary of open source changed in the AI era?
Should models, data, and chips become part of the open source core?
Are developers’ roles being redefined?

There are no standard answers to these questions, but the fact that they are being raised repeatedly shows they have become common concerns, not just the thoughts of a few.

Exhibition Area and Sub-forums: Closer to the Real Ecosystem

Compared to the main forum, I personally paid more attention to the sub-forums and exhibition area.

There, you would see:

Many projects no longer emphasize “who they want to replace”
More discussions about “who they can work with”
Several communities are seriously discussing long-term maintenance, not just releasing versions

This may not be glamorous, but it’s important.

A Personal Judgment

If I had to make a judgment about COSCon'25, I would say:

Chinese open source is shifting from “can we do it” to “can we sustain it for the long term.”

This is a more difficult, but also more realistic, stage.

This COSCon did not try to create a grand narrative. Instead, it felt like a “status exposure” at a particular stage: There are more questions, participants are more diverse, but the discussions are also closer to the real world.

Open source doesn’t depend on a single conference to move forward, but being on site helps you see more clearly: Where exactly are we standing right now?

Decoding Goose: Why It Joined AAIF and What This Means for Agentic Runtime

Jimmy Song — Fri, 12 Dec 2025 08:16:48 +0000

Goose is not a project that excites you at first glance in this wave of Agent innovation, but its entry into AAIF signals a deeper shift in how we think about Agentic Runtime and AI-Native infrastructure.

At first glance, Goose is not a project that immediately excites people.

Figure 1: Goose App UI

It doesn’t have flashy demos, nor does it showcase overwhelming multimodal capabilities, and it certainly doesn’t look like an AI product aimed at consumers. Yet, this seemingly “plain” project became one of the first donations to the Agentic AI Foundation (AAIF), standing alongside Anthropic’s MCP and OpenAI’s AGENTS.md.

This fact alone is worth a closer look.

This article does not aim to prove how powerful Goose is, but rather to answer three more practical questions:

What overlooked but long-term critical problems does Goose actually solve?
Why was it Goose, and not another Agent framework, that entered AAIF?
What does this mean for Agentic Runtime and AI-Native infrastructure, which I care about?

Goose’s True Positioning: It’s Not an IDE or a Chatbot

If you only look at its surface features, Goose is easily mistaken for one of two things:

A “multi-model AI desktop client”
Or an “intelligent programming assistant that can run commands”

But inside Block, it was never designed as a “tool” from the start.

Goose’s origin is closely tied to Block’s engineering environment.

Block (formerly Square) is a classic engineering-driven company: complex systems, high automation needs, many internal tools, and very high execution costs in real production environments. In its recent AI transformation, Block did not focus on “which model to choose” or “which AI tool to introduce,” but directly targeted the engineering execution layer itself.

Goose was born in this context.

Its goal is not to “help people code faster,” but to enable models to stably and controllably take action: run tests, modify code, drive UIs, call internal systems, and operate reliably in real engineering environments.

In short:

Goose is more like an executable Agent Runtime than a conversation-centric product.

Block’s AI Transformation Started with Organization, Not Tools

To understand Goose, you can’t ignore a key organizational shift at Block.

In an interview with Block’s CTO, one signal was very clear: the starting point for AI transformation was not buying tools or stacking models, but the organizational structure itself.

Block shifted from a business-line GM model to a more functionally oriented structure, making engineering and design the company’s core scheduling units again. This is essentially a proactive response to Conway’s Law.

If the organizational structure doesn’t allow technical capabilities to be orchestrated centrally, Agents will ultimately remain “personal assistants” or “engineering toys.”

From this perspective, Goose is not just a tool, but a cultural signal:

Every employee can use AI to build and execute real system behaviors.

This also explains a fact many overlook: Goose was not packaged as SaaS, nor was it rushed to commercialization, but was open-sourced and rapidly standardized.

Because its role inside Block is closer to an “operating system for execution models” than a product that can be sold separately.

Why Did Goose Enter AAIF? Not Because It’s “Technically Strongest”

This is what confuses outsiders the most.

If you only look at flashy features, model support, or community popularity, Goose doesn’t stand out. But AAIF’s choice was not about “maximum capability,” but about whether the position is right.

Looking at the first batch of AAIF projects, a clear chain emerges:

MCP (Anthropic): Defines how models safely and standardly call tools
AGENTS.md (OpenAI): Defines behavioral conventions for Agents in code repositories
Goose (Block): A real, runnable, open-source Agent execution framework

Goose’s role is not to set new protocols, but to serve as the practical carrier and reference implementation for these protocols.

It proves one thing:

MCP is not just a paper standard
Agents are not just research concepts
In real enterprise environments, they can actually run

From this angle, Goose’s “ordinariness” is actually an advantage.

It is not tied to Block’s business moat, nor does it have irreplaceable private APIs; it can be forked, replaced, audited—“boring” enough, and neutral enough.

And that is the most important trait of public infrastructure.

Goose’s Value Lies Not in Today, But in 2–3 Years

From a longer-term perspective, Goose’s value becomes clearer.

What we’re experiencing now is much like the early days of containers: Most Agent projects today are demos, IDE plugins, or workflow wrappers, but what’s really missing is a sustainable, schedulable, observable execution layer.

Goose is already moving in this direction.

Block’s metrics for Goose’s success are straightforward:

How many human hours are saved each week
How much non-technical teams reduce their dependence on engineering teams

Behind this is a judgment I’m increasingly convinced of:

What enterprises truly need is not “smarter models,” but “cheaper execution.”

The long-term value of Agents is not in generation quality, but in execution substitution rate.

AAIF Is an Attempt at Infrastructure-Level Consensus

Just as CNCF did for cloud native, AAIF is not guaranteed to succeed.

But it at least marks a shift: Agents are no longer just application-layer innovations, but are beginning to enter the stage of infrastructure-layer collaboration.

As a reference implementation, Goose is likely to remain in this ecosystem for a long time—even if it is replaced, rewritten, or evolved in the future.

Summary

If you see Goose as a “product,” it is indeed not dazzling.

But if you place it in the long-term evolution path of Agentic AI, its significance becomes clear:

It is not the end, but a necessary intermediate state.

For me, the emergence of Goose further confirms one thing:

Agentic Runtime is not a conceptual problem, but an engineering and organizational one.

And that is one of the most worthwhile directions to invest energy in over the next few years.

ARK: Multi-Agent Systems Are Finally Entering the Engineer's World

Jimmy Song — Thu, 11 Dec 2025 13:19:42 +0000

The deep integration of cloud native and AI, with the ARK platform, provides a new paradigm for engineering multi-agent systems.

Introduction

AI Agents are moving from the “single agent demo” stage to “large-scale operation.” The real challenge does not lie in the model itself, but in engineering issues at runtime: model management, tool invocation, state maintenance, elastic scaling, team collaboration, observability, deployment, and upgrades. These are problems that traditional agent libraries struggle to solve.

ARK (Agentic Runtime for Kubernetes) provides a fully operational, observable, governable, and continuously deliverable multi-agent operating system. It is not a Python library, but a complete runtime platform.

Figure 15: ARK Dashboard

Note: In this article, ARK refers to McKinsey’s open-source ARK Agent Runtime for Kubernetes.

This article, from an engineer’s perspective, will reorganize ARK’s core capabilities and answer the following questions:

What engineering challenges does ARK actually solve?
Why is it worth special attention in the cloud native field?
How is it fundamentally different from frameworks like LangChain and CrewAI?
What insights does it offer for the Agentic Runtime ecosystem?

ARK Architecture: Treating Agents as Kubernetes-Native Workloads

The core idea of ARK is: An agent is not a script, but a schedulable, governable, and observable Kubernetes workload.

The following architecture diagram illustrates ARK’s underlying structure.

Figure 16: ARK Overall Architecture

This diagram highlights ARK’s key design points:

CRDs declare requirements (Agent, Model, Team, Tool, Memory, etc.)
The Controller translates declarations into actual Pods/Services
The API provides a unified communication entry point and team orchestration
Memory supports long-term state management for agents
MCP Server enables external systems to become tools
Dashboard provides visual management and observability

ARK adopts the typical cloud-native Operator pattern and applies it to multi-agent systems.

CRD: ARK’s “Abstraction Layer”

Unlike traditional agent frameworks where “code is logic,” ARK uses CRDs (Custom Resource Definitions) to abstract the components of agent applications.

The main CRD types in ARK include:

Model
Agent
Team
Tool
Memory
Evaluation

These CRDs correspond to all the key components of an agent system.

The following diagram shows the structure of the CRDs:

Figure 17: CRD Structure (Simplified)

Through CRDs, ARK achieves the following engineering features:

All resources are GitOps-ready, supporting declarative management
Changes are auditable, reversible, and continuously deliverable
The evolution of models, tools, and agents does not require business code changes

This is the key gene of ARK’s engineering-oriented system.

Agent Execution Flow: From Query to Tool Invocation

The following image shows how to view query details in the ARK Dashboard.

Figure 18: Viewing Query Details in ARK Dashboard

In ARK, the complete execution flow for an agent receiving a query is as follows:

Figure 19: Agent Execution Flow

This flow has the following characteristics:

Memory modules are naturally involved in the execution flow, without code specialization
Large language model (LLM, Large Language Model) and tool invocation are governed by the runtime
Agents can reside in Pods long-term, not just as one-off processes

This makes ARK more like an “agent microservice platform.”

Below is an example of a request and response:

The True Value of Multi-Agent: Team Orchestration

ARK’s Team CRD allows multiple agents to be woven into a higher-level “system,” enabling multi-agent collaboration.

The following diagram shows the collaboration model of a multi-agent team:

Figure 20: Multi-Agent Team Collaboration

The engineering value of Team is reflected in:

Making “expert collaboration” declarative and configurable
Flexible strategies (such as polling, role assignment, routing, etc.)
A2A Gateway handles message passing
The Team itself is observable (every round of collaboration is logged)

For enterprises, this means the “agent organizational structure” can be standardized, replayed, and tuned.

Fundamental Differences Between ARK and Other Frameworks

Many engineers, upon first seeing ARK, may wonder:

“Is it just LangChain or CrewAI wrapped in Kubernetes?”

In fact, there are fundamental differences. The following diagram compares the structural differences between ARK and mainstream agent frameworks:

Figure 21: ARK vs LangChain / AutoGPT / CrewAI

The table below further summarizes the key differences:

Dimension	Traditional Agent Libraries	ARK
Core Pattern	Write Python code	Write CRDs (declarative)
Deployment	Local/Container	Kubernetes-native scheduling
State	Managed inside code	Memory CR + Service
Tools	Integrated at code level	Tool CR + MCP
Multi-Agent	Dialog managed in code	Team CR + A2A protocol
Observability	Almost none	OTel / Langfuse / Dashboard
Use Cases	Demo / Prototype / Single Agent	Enterprise production / Multi-Agent Systems

Table 3: ARK vs Traditional Agent Libraries

In short:

LangChain is a “library for building agents,” while ARK is a “platform for running agents.”

The two are not in conflict and are, in fact, highly complementary.

The Engineering Value of ARK

To summarize ARK’s engineering value in simple terms:

Turns agents into governable workloads
Unifies models, tools, and memory as reusable resources
Makes multi-agent collaboration structured, observable, and tunable
Brings agent upgrades and iteration into CI/CD + GitOps mode
Enables enterprises to manage agents like microservices

This is a clear evolution path:

Agent → Service → Platform → Runtime → Operating System

ARK is currently positioned at the fourth stage: Runtime.

Insights for Agentic Runtime

ARK provides three direct insights for building Agentic Runtimes:

Unified Scheduling System

The agent runtime must run on a unified scheduling system (Kubernetes, MicroVM, Wasmtime, etc.)

Declarative Capability Boundaries

Must use declarative abstractions to split capability boundaries, including:
- Model Layer
- Tool Layer
- Memory Layer
- Workflow Layer
- Team Layer
- State Layer

Observability

Observability is essential; otherwise, multi-agent systems cannot be engineered
- Langfuse
- OTel
- Logs / Events
- Structured JSON

ARK demonstrates a direction:

Multi-agent systems are an engineering problem, not a prompt engineering problem.

Summary

If you only need to build a simple agent, frameworks like LangChain, CrewAI, and AutoGPT are sufficient.

But if you want to operate a system composed of dozens or hundreds of agents that need to collaborate, run long-term, and support continuous delivery and governance, runtimes like ARK are the inevitable trend.

It provides Agentic AI with:

A cloud-native runtime model
Observable execution paths
Governable abstraction layers
Extensible, componentized architecture

Therefore, ARK deserves to be regarded as an early model for engineering multi-agent systems.

Can Open Source Suddenly Disappear? An AI Chat Dev Tool Went 404 Overnight

Jimmy Song — Thu, 11 Dec 2025 05:20:12 +0000

“Open source” in the AI era is no longer a trustworthy promise. Commercial projects can withdraw their code at any time, and developers must be wary of the gap between appearances and reality.

The Disappearance of Lunary’s Repository: A Real Case of Open Source “Vanishing”

While updating the AI open source project library on my website, I encountered a situation that left me stunned for the first time: An “open-source AI tool” that still promotes itself, with an active website and commercial services, suddenly vanished from GitHub—its repository went straight to 404.

The project is called Lunary.

Original repository address: https://github.com/lunary-ai/lunary It now returns a 404 Not Found.

Notably, the official site lunary.ai remains online, but the core promise of an “open-source codebase” has disappeared.

Lunary’s Positioning and Features

Here is an overview of Lunary’s main features and positioning to help understand its role in the AI tool ecosystem.

Lunary claims to be an Observability and Evaluations platform for large language model (LLM, Large Language Model) applications, focusing on:

LLM conversation and feedback logs
Cost, latency, and metrics analysis
Prompt version management
Distributed tracing
Evaluations
Supports both self-hosted and managed modes
Provides JS / Python SDKs, integrates with LangChain

Its overall positioning is clear: “Development and debugging tools for AI applications.”

In fact, products like this have emerged rapidly over the past year, forming a new AI DevTool track.

The Reality and Risks Behind the “Open Source” Label

The core issue is not the tool itself, but its claim to be “open source.”

Lunary has consistently emphasized:

“Lunary is an open-source platform for developers.”

This statement is great for attracting users, as open source implies transparency, trustworthiness, self-hosting, and community participation.

But now the repository is gone, with only the website continuing its promotion—raising many questions.

Lunary is not a niche hobby project, but a commercial company-led initiative. If an individual suddenly deletes a repo, it’s not surprising, but for a company operating publicly, this move is extremely rare.

This is the first time I’ve truly seen a reality in the AI DevTools space: “Open source” is being used as a branding term, not a commitment.

Possible Industry Reasons for Repo Deletion

Let’s analyze some common industry reasons for deleting a repository to help developers understand the motivations behind such actions.

Increased commercial pressure: These tools often struggle with sustainable business models, prompting teams to shift to closed-source SaaS.
Pivoting: The company finds the original direction unprofitable and prepares to change course.
Team changes: Acquisition, key member departures, or funding issues can all lead to repo shutdowns.
Compliance or legal risks: Observability products involve user data, which may require public code to be taken down.

Regardless of the reason, the impact on users is the same: it is no longer an “open-source product.”

The “Pseudo Open Source” Phenomenon in AI Tools

The most noteworthy aspect is not Lunary itself, but the rapid spread of this phenomenon in the AI tool space.

Many projects use “open source” as a user acquisition strategy but lack open governance and long-term commitment.

High substitutability, homogeneity, and commercial pressure mean these DevTools have low survival rates.

When commercial teams lead open source, a single decision can make the repository disappear instantly.

In the cloud native era, we’ve already seen a wave of “pseudo open source.” In the AI era, this trend is accelerating.

Three Practical Lessons for Developers

Based on this case, here are three practical lessons for developers:

The “open source label” does not guarantee trustworthiness: Open source projects led by commercial companies without community or foundation backing can be withdrawn at any time.
AI DevTools are far less stable than infrastructure: These tools are not essential, highly replaceable, and have short lifecycles.
Tool usability should take precedence over “open source status”: Because it may stop being open source at any moment.

My First Experience Maintaining an AI Project List and Facing Repo Deletion

After collecting hundreds of projects over the past two years, this is the first time I’ve encountered a “commercial open source project disappearing, official repo 404” case.

To me, this is an industry signal: the AI open source world is entering a period of drift, and commercial projects’ open source commitments are increasingly unstable.

It also reminds everyone making technical choices: in the AI era, open source is no longer a label you can automatically trust.

Summary

The disappearance of the Lunary repository is not an isolated incident, but a reflection of the “pseudo open source” phenomenon in the AI tool space. Developers should be cautious about the actual commitments behind the “open source” label, paying attention to project governance and sustainability. In the future, the boundary between open source and commercial will become even more blurred, and rational judgment and risk awareness will be essential for technical decision-making.

Lunary’s sudden disappearance highlights the instability of open source projects in the AI DevTools space. For developers, technical choices should focus more on project usability and community governance, rather than relying solely on the “open source” label. As the industry evolves, similar incidents may become more frequent. Only rational judgment and risk awareness can help you stand firm in the fast-changing tech landscape.

CNCF in the AI Native Era? The Agentic AI Foundation Is Officially Established

Jimmy Song — Wed, 10 Dec 2025 03:25:38 +0000

The standardization and open collaboration of the agent ecosystem is no longer a luxury, but the critical watershed for whether AI Native can be engineered and implemented.

The establishment of AAIF (Agentic AI Foundation) is the result of leading vendors staking out the “agent protocol layer” in advance.
The real challenge is not technical, but how organizations transition from “human execution + AI assistance” to “agent execution + human supervision”.
Successful agent adoption requires a phased adoption path, not just a bunch of protocols and demos.
CNCF and AAIF are complementary: CNCF manages “what infrastructure agents run on”, AAIF manages “how agents collaborate”. This matches the system I am building in ArkSphere.

Cloud Native Problems Are Solved, AI Native Problems Are Just Beginning

Over the past decade, Cloud Native technologies like Kubernetes, Service Mesh, and microservices have standardized “how applications run in the cloud”. But AI Native faces a completely different challenge: It’s not about “how to deploy a service”, but “how many behaviors in the system can be handed over to agents to execute themselves”.

CNCF’s Cloud Native AI (CNAI) addresses infrastructure-level issues: “How can model training/inference/RAG run at scale and securely on Kubernetes?”

But what AI Native truly lacks is another layer: How do agents collaborate, access tools, get governed, and audited?

This is exactly the gap AAIF aims to fill.

AAIF’s Three Weapons: Protocol + Runtime + Development Standard

AAIF hosts three core technologies contributed by its founding members:

① Anthropic’s Model Context Protocol (MCP)

https://github.com/modelcontextprotocol

A “system call interface for agents”:

Unified definition for how agents access databases, APIs, files, and external tools.
Designed to be more like an AI version of gRPC + OAuth.
Already integrated by Claude, Cursor, ChatGPT, VS Code, Microsoft Copilot, Gemini, and others.

It may not be the flashiest technology, but it could become the plumbing for the entire Agentic ecosystem.

② Block’s Goose Framework

https://github.com/block/goose

Reference runtime for MCP:

Local-first, composable agent workflow engine.
Enables enterprises to pilot agents in small scopes without betting on a specific vendor.
Serves as an engineering template for protocol implementation.

③ OpenAI’s AGENTS.md

https://agents.md

A simple but effective standard:

Place an AGENTS.md file in the project repository.
Clearly document build steps, testing, constraints, and context rules.
Any agent that understands AGENTS.md can operate the codebase using the same instructions.

This makes agent behavior more predictable and auditable.

Why Is AAIF in Such a Hurry? This Is a Race for Standards

Let’s compare with history:

Kubernetes’ predecessor Borg ran internally at Google for over a decade; K8s was open sourced and donated to CNCF two years later.
PyTorch joined the Linux Foundation six years after its release.
MCP was donated to AAIF just over one year after its launch.

AAIF is not about “mature technology entering a foundation”, but staking out the key position early.

The reasons are practical:

Prevent agent ecosystem fragmentation Today, there are many competing “tool invocation protocols”, which could become incompatible silos in three years.
Protocol layer is easier to reach global consensus than model layer Model competition is inevitable, but protocols can be standardized, open sourced, and avoid vendor lock-in.
A necessary move in global tech competition Putting the foundational standards for Agentic AI into the Linux Foundation is both a gesture of cooperation and a strategic move.

AAIF vs CNCF: Not Competition, But Two Pieces of the Puzzle

CNCF’s role:

“What infrastructure do agent workloads run on?” Kubernetes, Service Mesh, observability, AI Gateway, RAG Infra—all at this layer.

AAIF’s role:

“How do agents collaborate, invoke tools, and get governed?” Protocols, runtimes, and behavioral standards—all at this layer.

Analogy:

Domain	Responsibilities
AAIF	Semantic and collaboration layer of Agentic Runtime
CNCF/CNAI	Resource and execution layer of AI Native Infra

Table 3: AAIF vs CNCF Comparison

This matches the upper semantic and lower infrastructure layers in my ArkSphere architecture diagram.

In the long run, the two sides will be tightly coupled: CNCF’s KServe, KAgent, and AI Gateway will natively support MCP / AGENTS.md, AAIF’s Runtime will run on Cloud Native infrastructure by default.

The Real Challenge: Not Protocols, But Organizations and People

Most enterprises will get stuck on:

How much responsibility can agents actually take?
Who is accountable when things go wrong?
How are audit, SLOs, and compliance defined?
How is multi-agent collaboration visualized?
How are tool invocation permissions controlled?

In other words, agent adoption is not a “technical migration”, but an “organizational migration”.

If AAIF cannot provide:

Phased adoption methodologies
Typical organizational migration paths
Engineering best practices
Failure cases and anti-patterns

It will be difficult for AAIF to achieve the industry impact that CNCF did.

Summary: AAIF Is the Moment When Boundaries Are Drawn

For me, the establishment of AAIF feels like:

“The battlefield boundaries of the agent world have finally been drawn. Now it’s up to the engineering community to make it work.”

CNCF solved “how to run Cloud Native”, AAIF is now trying to solve “how agents collaborate”.

In the next five years, whoever can truly connect these two worlds will stand at the gateway to the next generation of infrastructure.

That’s why I started a dedicated “Agentic Runtime + AI Native Infra” research track in ArkSphere.

The ‘Three-Body’ Architecture of the AI Native Era

Finally, a personal note—my thoughts on ArkSphere.

Figure 3: AAIF × CNCF: Three-Layer Architecture of Agentic AI in the AI Native Era

This diagram shows the three-layer structure of the AI Native era:

CNCF (bottom layer): Provides the Cloud Native foundation required for agent operation, including Kubernetes, Service Mesh, GPU scheduling, and security systems.
AAIF (middle layer): Defines the runtime semantics and standards for agents, including the MCP protocol, Goose reference runtime, and AGENTS.md behavioral standard.
ArkSphere (bridging layer): Aligns the “Agentic Runtime semantic layer” with the “AI Native Infra infrastructure layer”, forming an engineerable agent architecture standard.

In short:

Infra is responsible for “running”, Runtime for “how to act”, and ArkSphere for “how to assemble a system”.

Bun Acquired by Anthropic: A Structural Signal for AI-Native Runtimes

Jimmy Song — Wed, 03 Dec 2025 05:21:28 +0000

The shifting ownership of runtimes is reshaping the underlying logic of AI programming and infrastructure.

After the announcement of Bun’s acquisition by Anthropic, my focus was not on the deal itself, but on the structural signal it revealed: general-purpose language runtimes are now being drawn into the path dependencies of AI programming systems. This is not just “a JS project finding a home,” but “the first time a language runtime has been actively integrated into the unified engineering system of a leading large model company.”

This event deserves a deeper analysis.

Bun’s Engineering Features and Current Status

Before examining Bun’s industry significance, let’s outline its runtime characteristics. The following list summarizes Bun’s main engineering capabilities:

High-performance JavaScript/TypeScript runtime
Built-in bundler, test framework, and package manager
Single-file executable
Extremely fast cold start
Node compatibility without Node’s legacy dependencies

These capabilities have formed measurable performance barriers.

However, it should be noted that Bun currently lacks the core attributes of an AI Runtime, including:

Permission model
Tool isolation
Capability declaration protocol
Execution semantics understandable by models
Sandbox execution environment

Therefore, Bun’s “AI Native” properties have not yet been established, but Anthropic’s acquisition provides an opportunity for it to evolve in this direction.

The Significance of a Leading Model Company Acquiring a General-Purpose Runtime

Historically, it is not uncommon for model companies to acquire editors, plugins, or IDEs, but in known public cases, mainstream large model vendors have never directly acquired a mature general-purpose language runtime. Bun × Anthropic is the first clear event pulling the runtime into the AI programming system landscape. This move sends two engineering-level signals:

The speed of AI code generation continues to increase, amplifying the need for deterministic execution environments. The generate→execute→validate→destroy cycle intensifies the problem of environment non-repeatability.
Models require a “controllable execution substrate” rather than a traditional operating system. Agents are not suited to run tools in an uncontrollable, unpredictable OS layer.
The runtime needs to be embedded into the model’s internal engineering pipeline. Future IDEs, agents, and auto-repair pipelines may directly invoke the runtime’s API.

This is not a short-term business integration, but a manifestation of the trend toward compressed engineering pipelines.

Runtime Requirements Differentiation in the AI Coding Era

Based on observations of agentic runtimes over the past year, runtime requirements in the AI coding era are diverging. The following list summarizes the main engineering abstractions trending in this space:

Determinism: AI-generated code is not reviewed line by line; execution results must be consistent across machines and over time.
Minimal distribution unit: Users no longer install language environments and numerous dependencies. Verifiable, replicable, and portable single execution units are becoming the norm.
Tool isolation: Models cannot directly access all OS capabilities; the context and permissions visible to tools must be strictly defined.
Short-lived execution: Agent invocation patterns resemble “batch jobs” rather than long-running services.
Capability declaration: The runtime must expose “what I can do,” rather than the entire OS interface.
Embeddable self-testing pipeline: After generating code, models need to immediately execute tests, collect errors, and iterate. The runtime must provide observability and diagnostic primitives.

These requirements are not unique to Bun, nor did Bun originate them, but Bun’s “monolithic and controllable” runtime structure is more conducive to evolving in this direction.

Figure 3: Minimal execution loop of an AI-native runtime

Bun’s Potential Role Within Anthropic’s System

If Bun is seen merely as a Node.js replacement, the acquisition is of limited significance. But if it is viewed as the execution foundation for future AI coding systems, the logic becomes clearer:

Code is generated by models
Building is handled by the runtime’s built-in toolchain
Testing, validation, and repair are performed by models repeatedly invoking the runtime
All execution behaviors are defined by the runtime’s semantics
The runtime forms Anthropic’s internal “minimal stable layer”

This model is similar to the relationship between Chrome and V8: the execution engine and upper-layer system co-evolve over time, with performance and semantics advancing in sync.

Whether Bun can fulfill this role depends on Anthropic’s architectural choices, but the event itself has opened up possibilities in this direction.

Industry Trends and Future Evolution

Combining facts, signals, and engineering trends, the following directions can be anticipated:

The “Agent Runtime” category will gradually become more defined
The boundaries between bundler, runtime, and test runner will continue to blur
Cloud vendors will launch controllable runtimes with capability declarations
Permission models and secure sandboxes will move down to the language runtime layer
Runtimes will become part of the model toolchain, rather than an external environment

These trends will not all materialize in the short term, but they represent the inevitable path of engineering evolution.

Summary

The combination of Bun × Anthropic is not about “an open-source project being absorbed,” but about a language runtime being actively integrated into the engineering pipeline of a large model system for the first time. Competition at the model layer will continue, but what truly reshapes software is the structural transformation of AI-native runtimes. This is a foundational change worth long-term attention.

Agentic Runtime Realism: Insights from McKinsey Ark on 2026 Infrastructure Trends

Jimmy Song — Tue, 02 Dec 2025 12:07:45 +0000

Statement

ArkSphere has no affiliation or association with McKinsey Ark.

The value of Agentic Runtime lies not in unified interfaces, but in semantic governance and the transformation of engineering paradigms. Ark is just a reflection of the trend; the future belongs to governable Agentic Workloads.

Recently, the ArkSphere community has been focusing on McKinsey’s open-source Ark (Agentic Runtime for Kubernetes). Although the project is still in technical preview, its architecture and semantic model have already become key indicators for the direction of AI Infra in 2026.

This article analyzes the engineering paradigm and semantic model of Ark, highlighting its industry implications. It avoids repeating the reasons for the failure of unified model APIs and generic infrastructure logic, instead focusing on the unique perspective of the ArkSphere community.

Ark’s Semantic Model and Engineering Paradigm

Ark’s greatest value is in making Agents first-class citizens in Kubernetes, achieving closed-loop tasks through CRD (Custom Resource Definition) and controllers (Reconcilers). This semantic abstraction not only enhances governance capabilities but also aligns closely with the Agentic Runtime strategies of major cloud providers.

Ark’s main resources include:

Agent (inference entity)
Model (model selection and configuration)
Tools (capability plugins/MCP, Model Capability Plugin)
Team (multi-agent collaboration)
Query (task lifecycle)
Evaluation (assessment)

The diagram below illustrates the semantic relationships in Agentic Runtime:

Figure 5: Agentic Runtime Semantic Relationships

Architecture and Community Activity

Ark’s architecture adopts a standard control plane system, emphasizing unified runtime semantics. The community is highly active, engineer-driven, and the codebase is well-structured, though production readiness is still being improved.

Figure 6: Ark Architecture and Control Plane Flow

ArkSphere’s Boundaries and Inspirations

The emergence of Ark has clarified the boundaries of ArkSphere. ArkSphere does not aim for unified model interfaces, multi-cloud abstraction, a collection of miscellaneous tools, or a comprehensive framework layer. Instead, it focuses on:

The semantic system of Agentic Runtime (tasks, states, tool invocation, collaboration graphs, etc.)
Enterprise-grade runtime governance models (permissions, auditing, isolation, multi-tenancy, compliance, cost tracking)
Integration capabilities for domestic ecosystem tools
Engineering paradigms from a runtime perspective

ArkSphere is an ecosystem and engineering system at the runtime level, not a “model abstraction layer” or an “agent development framework.”

Key Changes in 2026

2026 will usher in the era of Agentic Runtime, where Agents are no longer just classes but workloads that require governance rather than mere importation. Ark is just one example of this trend, and the direction is clear:

Semantic models and governability become highlights
Closed-loop tasks are the core value

Summary

Ark’s realism teaches us that the future belongs to runtime, semantics, governability, and workload-level Agents. The industry will no longer pursue unified APIs or framework implementations, but will focus on governable runtime semantics and engineering paradigms.

In-Depth Analysis of Ark: Kubernetes for the AI Era or a New Engineering Paradigm Shift?

Jimmy Song — Tue, 02 Dec 2025 10:54:34 +0000

Statement

ArkSphere has no affiliation or association with McKinsey Ark.

The greatest value of Ark lies in reshaping engineering paradigms, not just its features. It points the way for AI Infra and leaves vast space for community ecosystems.

Recently, many members in our ArkSphere community have started exploring McKinsey’s open-source Ark (Agentic Runtime for Kubernetes).

Some see it as radical, some think it’s just a consulting firm’s experiment, and others quote a realistic maxim:

What we need now is “agentic runtime realism,” not “unified model romanticism.”

I strongly agree with this sentiment.

I’ve spent some time analyzing Ark’s source code, architecture, and design philosophy, combined with our community discussions. My conclusion is:

Ark’s significance is not in its features, but in its paradigm.

It’s not the answer, but it points toward the future of AI Infra.

Below is my interpretation of Ark, focusing on engineering, architecture, trends, and its inspiration for ArkSphere.

What Exactly Is Ark?

Ark’s core positioning is: A runtime that treats Agents as Kubernetes Workloads.

It’s not a framework, not an SDK, not an AutoGen-style multi-agent tool, but a complete system including:

Control plane (Controller)
Custom resource models (CRD, Custom Resource Definition)
API service
Dashboard
CLI
Python SDK

Essentially, Ark is the control plane for Agents.

Ark defines seven core CRDs in Kubernetes. The following flowchart shows the relationships among these resources:

Figure 7: Ark CRD Resource Relationships

Through this set of CRDs, Ark makes Agent systems resource-oriented and declarative, enabling capabilities such as:

Lifecycle management
Multi-tenant isolation
RBAC (Role-Based Access Control)
Observability
Upgradability
Extensibility (tools, models, MCP)

In other words, Ark is not about “how to write Agents,” but “how to operate Agents in enterprise-grade systems.”

Three-Layer Architecture: Mixed Languages and Components, but a Complete System

Ark’s overall architecture is divided into three layers, each with different tech stacks and responsibilities. The following flowchart illustrates the relationships among components in each layer:

Figure 8: Ark Three-Layer Architecture Components

This is not a “wrapper project,” but a fully operational AI Runtime system, with a level of engineering far beyond most agent frameworks on the market.

Is It the Kubernetes of the AI Era?

Let’s revisit Kubernetes’ core value:

Kubernetes was never about “unifying cloud APIs”; it unified the “application runtime model.”

Cloud provider APIs aren’t unified, nor are networking or storage. What is unified: Pod, Deployment, Service—these application models.

Kubernetes succeeded because:

It provides a stable application abstraction on top of diversity.

Ark’s goal is not to unify all large language models (LLMs), MCPs, or tool formats, but rather:

Agent resource model (CRD) + control plane (Reconciler) + lifecycle.

From this perspective, Ark offers a prototype of a “declarative application model” for the AI era.

Whether it will become “Kubernetes for AI” is still too early to say, but it has already planted a seed.

Comparison with Other Frameworks: Not on the Same Level

Current mainstream agent frameworks like LangChain, CrewAI, AutoGen, MetaGPT, etc., address problems fundamentally different from Ark.

The table below compares the positioning and limitations of each framework:

Name	What Problem Does It Solve	Core Limitation
LangChain	Agent/Tool composition	Doesn’t address deployment or governance
AutoGen	Multi-agent conversations	Lacks control plane and lifecycle
CrewAI	Workflow-style multi-agent	Missing scheduling, RBAC, resource model
MetaGPT	Agent SOP	Just execution logic, not a platform
OpenDevin	AI IDE/Dev Assistant	Not an Agent Runtime
Ark	Agent control plane + resource system	Functionality not yet mature

Table 3: Mainstream Agent Frameworks vs. Ark

In short:

Other tools focus on “how to write Agents.”
Ark focuses on “how Agents run, schedule, govern, observe, and extend.”

That’s an architectural difference.

Execution Flow: Agents Scheduled Like Pods

Ark’s execution flow closely resembles the Kubernetes controller model. The following sequence diagram shows the core process:

Figure 9: Ark Agent Execution Flow

You can see Ark’s process logic is transparent, with a clear engineering path, bringing agent systems into a “controllable” state.

Production Readiness: Right Direction, Still a Tech Preview

According to official notes and code maturity, Ark currently offers:

Runnable
Learnable
Extensible
But not recommended for large-scale production use yet

Main reasons include:

CRD structures may change
APIs are not yet stable
MCP ecosystem is still forming
Memory service is still basic
Multi-agent team execution strategies are primitive

But the engineering system is already taking shape, which is crucial.

Community Activity: Small but Elite, Strong McKinsey Drive

From GitHub data:

Stars: 222
Forks: 50
Contributors: 48
Commit frequency is steady
The vast majority of contributions come from within McKinsey

Note: Data as of December 2, 2025.

High stability, but limited openness.

This is also ArkSphere’s opportunity:

The paradigm is right, but the ecosystem needs community-driven growth.

Trends for 2026: From Framework Era to Runtime Era

After deep analysis, I’m increasingly convinced:

2023–2024: Large model API call era
2024–2025: Agent framework era
2025–2027: Agent Runtime / Control Plane era (Ark’s direction)

While everyone is writing Python scripts for agents, the real value lies in:

Multi-agent task scheduling
Tool registration and governance
Session/Memory lifecycle
Result reproducibility
RBAC, auditing, tenant isolation
Observability
Enterprise internal personalized agent systems

Ark is providing a practical path forward.

Inspiration for ArkSphere

Ark’s inspiration for ArkSphere is both critical and direct:

ArkSphere Should Focus on “Paradigm Building,” Not “Feature Stacking”

Ark offers a prototype for future Agentic Runtime:

Resource model
Control plane
Tool registration
Multi-agent collaboration
Evaluation and governance

ArkSphere’s role should be:

Aggregate paradigms, produce standards, incubate ecosystems, not rewrite Ark itself.

This is the “CNCF (Cloud Native Computing Foundation) for the AI-native era.”

Huge Potential for Localization in China

Localization opportunities include but are not limited to:

Integration with domestic large language models (e.g., Qwen, DeepSeek, Zhipu)
Enterprise privatization scenarios
Local tool/MCP discovery ecosystem
Multi-cluster/edge inference
Enterprise-grade RBAC, auditing, data isolation
AgentSpec enhancements for industrial scenarios
Enhanced versions of Runtime/Controller

In other words:

Ark solves the “model,” while ArkSphere can solve the “ecosystem.”

What We Need Is Not “Kubernetes for the LLM Era,” But an “Industry-Grade Cognition System for AI Runtime”

The biggest takeaway from dissecting Ark:

The future of AI-native is not a pile of tools, but an engineering system.

ArkSphere can be the initiator of this system.

Summary

Ark is not a “universal runtime,” nor is it the “ultimate Kubernetes for the AI era.”

But it has done one crucial thing right:

It abstracts all the pain points people faced when writing Python agent scripts into Kubernetes resources and controllers.

It represents engineering, not just a demo.

It’s not mature yet, but it’s heading in the right direction.

It’s not the end, but it gives us a clear roadmap.

For the ArkSphere community I’m running, Ark provides a clear inspiration:

The future belongs to Runtime, to Control Plane, to governable agent systems.

And the ones who can truly scale this system are not McKinsey, but the community.

From Using AI to Relying on AI: Why the Era of AI Engineering Has Yet to Begin

Jimmy Song — Sat, 29 Nov 2025 12:40:54 +0000

The real inflection point for AI engineering is not “how many people use it,” but “how many people cannot do without it.” Only when not using AI leads to direct loss of opportunity and efficiency, can we say the era of AI engineering has truly arrived.

Starting Point: Predictions for AI in 2026

Recently, I came across two predictions for 2026 from Amazon CTO Werner Vogels that struck me the most:

Renaissance Developer: Developers must span code, product, business, and social impact.
Personalized Learning: AI will reshape education, focusing on differentiated paths rather than a unified curriculum.

Both point to the same trend: AI is not just a tool, but is redefining how people grow and how they are defined.

There is a gap between prediction and reality, and it is worth exploring.

Correction: Will AI Really Be “Saturated” by 2026?

My initial prediction was that AI usage would reach saturation by 2026. Reality has shown me this is too optimistic.

By the end of 2025, even among internet professionals, most people’s use of AI remains at the “heard of it” or “tried it a few times” stage. It is still far from being a daily workflow necessity.

More importantly, this judgment is conditional: infrastructure supply, regulation, and compute costs must not reverse in the next 3–6 years. If any variable breaks down (costs double, models go offline, policy shifts), the adoption curve will be disrupted.

The Truth About the Inflection Point: From “Using” to “Relying On”

“Relying on” is a vague term. A more precise definition requires measurable indicators.

Here is a diagram that visualizes the metrics for being truly dependent on AI:

This diagram visualizes the quantitative metrics for being truly dependent on AI, comparing target thresholds with current status:

Figure 1: Quantitative Definition of AI Dependency

Most industries have not reached the “cannot operate without” stage, unlike the internet, mobile, or payment inflection points. Most metrics are still far below the threshold, which is why the most likely outcome for 2026 is: more people will use AI, but those who truly rely on it will remain a minority.

Using ≠ Building: The Five-Level Capability Ladder

This difference is not binary, but a clear progression.

The following table shows the five-level model of AI capability maturity.

Level	Name	Description	Scarcity
1	Tool User	ChatGPT/Claude, Coding, Copywriting, Accelerator, Optional	Low
2	Integrator	LLM API + Vector DB, AI layered on existing systems, Usable, not critical	Low
3	Settler	Restructuring data flow, business decisions, AI becomes critical path	Rising
4	Engineering Abstraction	Extracting frameworks, runtimes, providing infra for ecosystem	Extremely High
5	Autonomous System	Self-feedback, self-optimizing, redefining human-AI relationship	Future

Table 1: Five-Level Model of AI Capability Maturity

Currently, the biggest gap is at Level 3 and Level 4. Most people are stuck at Level 1 or 2, with very few reaching Level 4. This means high-value scarcity will not disappear, but will continue to rise.

Why the Era of AI Engineering Has Not Arrived: Three-Dimensional Delaying Factors

It is not technology alone that is holding things back, but constraints in three dimensions.

The following diagram illustrates the three main constraints delaying AI engineering maturity:

This diagram illustrates the three main constraints (technical, institutional, and organizational) that are delaying AI engineering maturity, along with their delay metrics:

Figure 2: Three-Dimensional Constraints on AI Engineering Maturity

The key observation: If any one dimension is stuck, the entire ecosystem’s maturity will be delayed. Currently, none of the three dimensions have fully mature solutions.

The Realistic Window: Three Paths for Capability Advancement

The next three years will not be “winner takes all,” but rather a period where multiple capability levels appreciate simultaneously.

Below is a table comparing the value and bottlenecks of different capability advancement paths:

Capability Path	Short-Term Value	Long-Term Outlook	Bottleneck
Level 1→2 (Tool→Integration)	⭐⭐ Rapid Depreciation	⭐ Saturation	Low barrier, fierce competition
Level 2→3 (Integration→Settlement)	⭐⭐⭐⭐ Scarce	⭐⭐⭐⭐ Continual Appreciation	Requires industry depth, long-term iteration
Level 3→4 (Settlement→Abstraction)	⭐⭐⭐⭐⭐ Extremely Scarce	⭐⭐⭐⭐⭐ Defines Ecosystem	Large cognitive leap, needs community influence

Table 2: AI Capability Advancement Paths and Value Comparison

Key conclusion: While the number of “AI users” is rapidly increasing (depressing Level 1 value), due to the three-dimensional delaying factors, scarcity at Level 3 and 4 will only rise.

What I’m Doing on arksphere.dev

Based on the above judgment, I focus on exploring the architectural evolution of AI Native Infrastructure. The goal is not to catalog model usage, but to study the foundational capability stack supporting scalable intelligent systems: scheduling, storage, inference, Agent Runtime, autonomous control, observability, and reliability.

The content is no longer a collection of courses or tips, but a continuous record of evolution around Infra → Runtime → System Abstraction. arksphere.dev is the site for this experiment and settlement.

Summary

The inflection point for the era of AI engineering is not “how many people use it,” but “how many people cannot do without it.” The latter requires five measurable indicators to reach their thresholds, and we are still far from that.

“Using ≠ Building” is not a binary, but a five-level progression. Scarcity at Level 3 and 4 will rise as the number of Level 1 users increases—this is the biggest opportunity window in the next three years.

But the width of this window depends largely on how technology, institutions, and organizations evolve together. I hope more people working on AI engineering will not only focus on technical innovation, but also invest equal thought into institutional development, talent growth, and risk governance—these “invisible engineering” challenges.

Antigravity VS Code Setup Guide: Build a Practical AI IDE Workflow

Jimmy Song — Thu, 20 Nov 2025 03:55:30 +0000

The biggest pain point when switching IDEs is user habits. By installing a series of plugins and tweaking configurations, you can make Antigravity feel much more like VS Code—preserving familiar workflows while adding Open Agent Manager capabilities.

If you searched for a practical Antigravity VS Code setup, this walkthrough is optimized for that exact use case. The goal is not to replicate VS Code pixel by pixel, but to restore a familiar extension marketplace, keep your daily coding ergonomics, and still use Antigravity’s stronger agent-style execution. I focus on the concrete setup steps that materially change productivity: marketplace migration, AMP and CodeX installation, editor behavior alignment, and the trade-offs versus GitHub Copilot in real daily work.

Figure 1: Antigravity IDE UI

Continue Reading

Qoder AI IDE review and hands-on comparison
Open-source AI Agent and workflow platform comparison
Free Vibe Coding tools I actually use
Oh My OpenCode in AI Native Landscape

Below are the configurations and steps I actually use. Feel free to follow along.

First Impressions of Antigravity

A few subjective observations:

The interface is split between agent management and editor views, somewhat like AgentHQ + VS Code.
Agents modify code very quickly, with a much higher completion rate than typical “chat-based” assistants.
The editor and context windows are large, ideal for long diffs and logs.
By default, it uses OpenVSX / OpenVSCode Gallery, so the extension ecosystem isn’t identical to my VS Code setup.

All subsequent steps focus on one goal: keep Antigravity’s agent features while maintaining my VS Code workflow.

Switching the Extension Marketplace to VS Code Official

Antigravity is essentially a VS Code fork, so you can directly change the Marketplace configuration.

In Antigravity:

Go to Settings -> Antigravity Settings -> Editor, and update the following URLs to point to VS Code:

Marketplace Item URL:

https://marketplace.visualstudio.com/items

Marketplace Gallery URL:

https://marketplace.visualstudio.com/_apis/public/gallery

Figure 2: VSCode Marketplace Configuration

Restart Antigravity.

After this change, searching and installing extensions works just like the official VS Code Marketplace. Installing AMP, GitHub Theme, VS Code Icon, etc., all follow this process.

Installing the AMP Extension

AMP isn’t officially supported on Antigravity yet, but you can install it directly via the VS Code Marketplace.

Steps:

Open the Extensions panel (the same icon as in VS Code).
Search for the AMP extension and install it as usual.
Log in using your AMP API Key.

Currently, Antigravity doesn’t support one-click account login like VS Code; you have to use the API key.

Summary

Once installed, AMP works almost identically in Antigravity as in VS Code—completion and refactoring features are available. The only difference is manual login configuration.

I recommend AMP because it offers a free mode. In my experience, it’s great for writing documentation, running scripts, and as a daily command-line tool. It’s fast, and especially useful for optimizing prompts.

Importing the CodeX Extension

CodeX doesn’t provide a direct VSIX download link on the web. My approach is to export it from VS Code and then import it into Antigravity.

Figure 3: Exporting Codex Extension in VS Code

Steps:

Install the CodeX extension in VS Code (if you haven’t already).
In VS Code’s extension manager, find CodeX and export it as a .vsix file.
Switch to Antigravity, open the Extensions panel, and select “Install from VSIX”.
Choose the exported codex-x.x.x.vsix file to complete installation.

Tip

Since my local VS Code is already logged into CodeX, importing it into Antigravity automatically reuses the login state—I didn’t need to log in again.

Optimizing Editor Settings

Beyond the marketplace and plugins, a few tweaks make the experience even closer to VS Code:

Theme: Choose the same color scheme as VS Code to minimize visual switching. I use GitHub Theme and vscode-icons.
Editor Settings: In “Open Editor Settings”, set indentation, formatting, line width, etc., to match your VS Code preferences. I define these in the workspace’s settings.json, so no migration is needed.

After these changes, the editing area is essentially “VS Code with an agent console”.

Remaining Issues

To fully migrate from VS Code/GitHub Copilot to Antigravity, I think there are still several key challenges:

Limited Customization: Antigravity can’t support custom prompts and agents like Copilot Chat. Currently, only “rules” configuration is available, which limits workflow flexibility.
Model Ecosystem Needs Improvement: Antigravity hasn’t natively integrated the latest models from major vendors (OpenAI, Anthropic, Microsoft, xAI, etc.), whereas GitHub Copilot excels here.
Cost Considerations:
- Future pricing may start at $20/month.
- No free models are supported, unlike GitHub Copilot (even Copilot Pro users have free model options).
Stability Issues: Agents often encounter “Agent terminated due to error” during operation, requiring manual retries or new sessions. This affects workflow smoothness, though I expect improvements in the future.

GitHub Copilot VS. Antigravity

Although Antigravity excels in several areas, there is still significant room for improvement compared to the combination of GitHub Copilot and VS Code.

The large language models (LLMs, Large Language Models) I frequently use are all supported in VS Code:

Figure 4: Copilot-supported LLMs (partial)

My long-accumulated custom prompts:

Figure 5: Copilot Chat enables quick access to custom prompts

My collection of agents:

Figure 6: Copilot Chat allows selection of custom agents

Here are some personal experiences using VS Code and Copilot that, for now, are hard to replace with other IDEs:

The Ask/Edit/Agent/Plan workflow perfectly fits my working habits.
Support for custom prompts and agents is essential. Many of my prompts and agents have been refined over time and are deeply integrated into my daily workflow—it’s hard to find alternatives elsewhere.
New models are integrated at lightning speed. Whenever a new model is released, GitHub Copilot is among the first to support it.
The integration with VS Code is seamless—no extra configuration required, making it extremely convenient.
Frequent updates: just a few days ago, a bug I reported to VS Code was fixed the same night.
Copilot Chat’s keyboard shortcuts make it easy to quickly access various features.
GitHub has granted me a free Pro account. Although the monthly premium quota is only 300 calls, combining Copilot with other plugins like AMP, Codex, Droid, and Qwen enables a highly efficient workflow. Even if I upgrade to a paid account in the future, the $10/month fee is very cost-effective compared to similar products.

Practical Experience

A few subjective tips from my actual usage—take them as reference:

Don’t treat Antigravity as “VS Code + chat box”. Use its agent features for complete tasks: let the agent propose a plan, then execute changes.
For major changes, always create a new Git branch and restrict agent actions to that branch. Handle all diffs via standard Pull Request (PR) workflows.
Ask agents to produce “artifacts” (plans, proposals, test descriptions), not just final code. This makes it easier to review and track changes.
Plugins you’re already comfortable with in VS Code (like AMP, CodeX) can be migrated directly, reducing cognitive load and letting you focus on new agent workflows.

Summary

My current experience: Antigravity delivers powerful agent capabilities and multi-view consoles. By following these steps to align the interface and plugin ecosystem with VS Code, you can smoothly transition your daily development workflow.

Cloudflare November 18 Global Outage: The Dangers of Implicit Assumptions in Modern Infrastructure

Jimmy Song — Wed, 19 Nov 2025 18:56:34 +0800

The greatest risks to modern internet infrastructure often aren’t in the code itself, but in those implicit assumptions and automated configuration pipelines that go undefined. Cloudflare’s outage is a wake-up call every Infra/AI engineer must heed.

Yesterday (November 18), Cloudflare experienced its largest global outage since 2019. As this site is hosted on Cloudflare, it was also affected—one of the rare times in eight years that the site was inaccessible due to an outage (the last time was a GitHub Pages failure, which happened the year Microsoft acquired GitHub).

Figure 1: jimmysong.io was down for 27 minutes due to the Cloudflare outage

This incident was not caused by an attack or a traditional software bug, but by a seemingly “safe” permissions update that triggered the weakest link in modern infrastructure: implicit assumptions (Implicit Assumption) and automated configuration pipelines (Automated Configuration Pipeline). Cloudflare has published a blog post Cloudflare outage on November 18, 2025 explaining the cause.

Here is the chain reaction process of the outage:

A permissions adjustment led to metadata changes;
The metadata change doubled the lines in the feature file;
The doubled lines triggered the proxy module’s memory limit;
The memory limit caused the core proxy to panic;
The proxy panic led to a cascade failure in downstream systems.

This kind of chain reaction is the most typical—and dangerous—systemic failure mode at today’s internet scale.

Root Cause: Implicit Assumptions Are Not Contracts

Let’s first look at the core hidden risk in this incident. The Bot Management feature file is automatically generated every five minutes, relying on a default premise:

The system.columns query result contains only the default database.

This assumption was not documented or validated in configuration—it existed only in the engineer’s mental model.

After a ClickHouse permissions update, the underlying r0 tables were exposed, instantly doubling the query results. The file size exceeded the FL2 preset of 200 features in memory, ultimately causing a panic.

Once an implicit assumption is broken, the system lacks a buffer and is highly prone to cascading failures.

Configuration Pipelines Are Riskier Than Code Pipelines

This incident was not caused by code changes, but by data-plane changes:

SQL query behavior changed;
Feature files were automatically generated;
The files were broadcast across the network.

A typical phenomenon in modern infrastructure: data, schema, and metadata are far more likely to destabilize systems than code.

Cloudflare’s feature file is a “supply chain input,” not a regular configuration. Anything entering the automated broadcast path is equivalent to a system-level command.

Language Safety Can’t Eliminate Boundary Layer Complexity

A former Cloudflare engineer summarized it well:

Rust can prevent a class of errors, but the complexity of boundary layers, data contracts, and configuration pipelines does not disappear.

The FL2 panic stemmed from a single unwrap(). This isn’t a language issue, but a lack of system contracts:

No upper-bound validation for feature count;
File schema lacked version constraints;
Feature generation logic depended on implicit behavior;
Core proxy error mode was panic, not graceful degradation.

Most incidents in modern distributed systems (Distributed System) come from “bad input,” not “bad memory.”

Core Proxies Need Controllable Failure Paths

FL/FL2 are Cloudflare’s core proxies; all requests must pass through them. Such components should not fail with a panic, but have the following capabilities:

Ignore abnormal features;
Truncate over-limit fields;
Roll back to previous versions;
Fail-open or fail-close;
Skip the Bot module and continue processing traffic.

As long as the proxy “stays alive,” the entire network won’t be completely paralyzed.

Data Changes Are More Uncontrollable Than Code Changes

The essence of this incident:

Subtle permission changes;
ClickHouse default behavior changed;
Query results propagated to distributed systems;
Automated publishing amplified the error;
Edge proxies crashed due to uncontrolled input.

Future AI Infra (AI Infrastructure) will be even more complex: models, tokenizers, adapters, RAG indexes, and KV snapshots all require frequent updates.

In future AI infrastructure, data-plane risks will far exceed those of the code-plane.

Recovery Process Shows Engineering Maturity

During the incident, Cloudflare took several measures:

Stopped generating erroneous feature files;
Force-distributed the previous version of the file;
Rolled back Bot module configuration;
Ran Workers KV and Access outside the core proxy;
Restored traffic in stages.

Restoring hundreds of PoPs worldwide simultaneously demonstrates a high level of engineering maturity.

Lessons for Infra/AI/Cloud Native Engineers

The Cloudflare event highlights four common risks in large-scale systems:

Implicit assumptions fail;
Configuration supply chain contamination;
Automated publishing amplifies errors;
Core proxies lack graceful degradation paths.

For AI Infra practitioners, these risks are even more relevant:

Model weight updates without schema validation;
Adapter merges may be contaminated;
RAG index incremental builds are unstable;
Inference graph configuration may be broken by bad data;
Automatically rolled-out models may propagate errors network-wide.

AI engineering is replaying Cloudflare’s infrastructure dilemmas—just at greater speed and scale.

Summary of Former Cloudflare Engineer’s Views

His insights pinpoint the hardest problems in distributed systems:

The issue isn’t code, but missing contracts;
Not the language, but undefined input boundaries;
Not modules, but lack of validation in the configuration supply chain;
Not bugs, but absence of fail-safe mechanisms.

This incident proves: The real fragility in modern infrastructure lies in “behavioral boundaries,” not “memory boundaries.”

Summary

The Cloudflare November 18 outage was not a coincidence, but an inevitable result of modern internet infrastructure evolving to large-scale, highly automated stages.

Key takeaways from this event:

System assumptions must be made explicit;
Configuration pipelines must be validated;
Automated publishing needs “dead-end” mechanisms;
Core proxies must be designed with controllable failure paths;
Data-plane contracts must be stricter than code-plane contracts.

In the AI-native Infra era, these requirements will only become more stringent.

References

The Second Half of Cloud Native: The Era of AI Native Platform Engineering Has Arrived

Jimmy Song — Mon, 17 Nov 2025 11:07:40 +0000

The second half of cloud native isn’t about being replaced by AI, but being rewritten by it. The future of platform engineering will revolve around models and agents, reshaping the tech stack and developer experience.

Since I first encountered Docker and Kubernetes in 2015, I’ve followed the cloud native journey: from writing Deployments in YAML, to exploring Service Mesh and observability, and in recent years, focusing on AI Infra and AI-Native Platforms. Looking back from 2025, the years 2015–2025 can be seen as the “first half” of cloud native. Marked by KubeCon / CloudNativeCon NA 2025, the industry is collectively entering the “second half”: the era of AI-Native Platform engineering.

This article reviews the past decade of cloud native, and, combined with KubeCon NA 2025, outlines key turning points and the technical coordinates for the next ten years.

2015–2025: The “First Half” of Cloud Native

Over the past decade, cloud native technology themes have evolved through three main stages. The following flowchart illustrates the progression.

The flowchart below illustrates the progression of cloud native technology themes over the past decade:

Figure 1: Cloud Native Decade Technology Evolution Flow

The first stage focused on containerization and orchestration standardization.

Docker realized the engineering dream of “build once, run anywhere”
Kubernetes won the orchestration wars and became the de facto standard
CNCF was founded, with Prometheus, Envoy, and other projects joining
Enterprises focused on migrating applications to Kubernetes

Typical tasks during this phase involved moving Java services from VMs to containers and K8s, emphasizing understanding of Deployment, Service, and Ingress.

The second stage, 2018–2020, saw complexity shift from “deployment” to “communication” and “operations”.

Service Mesh (Istio / Linkerd / Consul) addressed east-west traffic management
The observability trio (Logs / Metrics / Traces) became default configurations
Multi-cluster and multi-region practices matured
Enterprises focused on managing large microservice systems

During this period, I spent significant time researching Istio, service mesh, and traffic management, and authored Kubernetes and Istio books. The focus shifted to system stability, observability, and reliability.

The third stage, 2021–2025, is defined by Platform Engineering and GitOps.

As microservices and tools proliferated, platform complexity began to overwhelm developers, making Platform Engineering a key industry term.

GitOps (Argo CD / Flux) drove declarative delivery processes
Internal Developer Platforms (IDP) became priorities for large enterprises
“Platform as a product” philosophy spread
FinOps, cost management, and compliance auditing became platform concerns
DevOps evolved from “tool practice” to “organizational + platform capability”

My takeaway: simply giving developers a pile of tools isn’t enough. End-to-end delivery paths and stable abstraction layers are needed so developers can focus on business, not tool integration.

The table below summarizes the main features of each “first half” stage.

Stage	Core Challenge	Key Tech Stack	Typical Issues
2015–2017 Orchestration	Migrating from VM to containers	Docker, Kubernetes, CNI	Reliable deployment, rolling upgrades
2018–2020 Mesh	Microservice scale, complex communication & observability	Istio/Linkerd, Prometheus, Jaeger	Troubleshooting, fragmented observability
2021–2025 Platform	Tool sprawl, declining developer experience	GitOps, IDP, FinOps, Policy-as-Code	Developer fatigue, platform team overload

Table 1: Cloud Native First Half Stage Features

KubeCon NA 2025: Signals of Cloud Native’s “Second Half”

The main theme of KubeCon 2025 is no longer “how to use Kubernetes well,” but how to reconstruct Kubernetes and the cloud native ecosystem into AI-Native Platforms for the AI era.

Key signals from KubeCon NA 2025 include:

CNCF released the Certified Kubernetes AI Conformance Program
Dynamic Resource Allocation (DRA) entered mainstream discussions
Model Runtime / Agent Runtime projects became conference hotspots
Vendors focused on AI SRE, AI-assisted development, AI security, and supply chain governance
Speakers like Alex Zenla openly stated that Kubernetes’ underlying structure needs rethinking

Together, these mark a clear dividing line: cloud native has officially entered its “second half.”

First Half vs Second Half: Shifting the Cloud Native Narrative

If 2015–2025 is the “first half,” then 2025–2035 is likely the “second half.” The table below compares their core differences.

It highlights changes in platform objects, goals, abstraction layers, and more.

Dimension	First Half (2015–2025)	Second Half (2025–2035, AI Native)
Core Objects	Containers, Pods, Microservices	Models, inference tasks, Agents, data pipelines
Platform Goals	Stable application delivery	Efficient, continuous AI workload & agent orchestration
Abstraction Layers	Deployment / Service / Ingress / Job	Model / Endpoint / Graph / Policy / Agent
Resource Scheduling	CPU / Memory / Node	GPU / TPU / ASIC / KV Cache / Bandwidth / Power
Engineering Focus	DevOps / GitOps / Platform Engineering 1.0	AI Native Platform Engineering / AI SRE
Security & Compliance	Image security, CVE, supply chain SBOM	Model security, data security, AI supply chain & “hallucination dependencies”
Runtime Forms	Container + VM + Serverless	Container + WASM + Nix + Agent Runtime

Table 2: Core Differences: First vs Second Half of Cloud Native

From a developer’s perspective, the most direct change is: future platforms will no longer treat “services” as first-class citizens, but will center on “models + agents.”

Example: Technical Layers of an AI Native Platform

To clarify the structure of an AI-Native Platform, the following layered diagram shows the relationships between technical levels.

The layered diagram below shows the relationships between different technical levels in an AI-Native Platform:

Figure 2: AI Native Platform Layering Diagram

Historically, cloud native focused on L0 + L2 (Kubernetes + platform engineering), but in the AI Native era, L1 (Model Runtime, Agent Runtime, heterogeneous resource scheduling) becomes the new battleground.

Key Change 1: From “Container-Centric” to “Model-Centric”

In the first half, cloud native’s main object was the application process, with containers as packaging. The second half requires handling:

Model version management and canary releases
Balancing inference performance, latency, and cost
Multi-model composition, routing, A/B testing
Relationships between models, data, features, and vector indexes

At KubeCon NA 2025, CNCF’s AI Conformance Program aims to standardize model workloads, managing them like Deployments. Platform engineering will gain new abstractions—not just “deploying services,” but “deploying model capabilities.”

Key Change 2: DRA and the Golden Window for Heterogeneous Resource Scheduling

Previously, writing a Deployment meant focusing on CPU and memory. Now, GPU inference, training, and Agent Runtime scenarios demand more than static quotas.

Dynamic Resource Allocation (DRA) brings:

Pluggable resource types (GPU/TPU/FPGA/ASIC)
Topology-aware, NUMA, and memory fragmentation scheduling
Binding inference requests to compute allocation for fine-grained QoS
Cost optimization and power control in scheduling decisions

This is the most significant “resource perspective” upgrade since Kubernetes’ inception. The scheduler is no longer just a cluster component, but the AI platform’s policy engine.

Key Change 3: Agent Runtime as the New Generation of Runtime

KubeCon showcased several representative projects:

Edera: Minimal, verifiable runtime redesign
Flox: Nix-based “uncontained” runtime environment
Golem: WASM-based large-scale agent orchestration

The consensus: AI agents aren’t suited to traditional container runtime models. Agents have these traits:

Strong statefulness: context, memory, sessions
High concurrency but fine granularity: massive lightweight tasks
Extremely sensitive to latency and cold starts
Need to resume after failure

Next-gen runtimes focus on reliably executing, managing state, and auditing “hundreds of thousands of agents,” not just “spinning up more Pods.”

Key Change 4: AI SRE and AI Security

At KubeCon NA 2025, security and operations topics were amplified by AI:

Software supply chain attacks and CVEs continue to rise
LLM-assisted coding introduces “hallucination dependencies” and “vibecoded vulnerabilities”
AI-driven artifact scanning, dependency auditing, and license analysis
“AI SRE” is now a formal product category

Traditional cloud native already emphasized security and SRE, but now must address model weights, datasets, vector stores, and agent workflows. AI-Native Platform engineering must answer:

Are code and dependencies secure?
Are models and data trustworthy?
Are agent behaviors controllable?

This will drive deep integration of Policy-as-Code, MCP, graph permission systems, and AI.

Key Change 5: Open Source Participation Becomes a Baseline

In interviews, platform engineering leaders noted:

Hiring increasingly values upstream contributions to Kubernetes and related projects
Open source involvement shortens ramp-up time
New AI Native projects (Model Runtime, Agent Runtime, Scheduler) are also open source

For career growth, contributing to AI Native open source projects will become a basic requirement for platform engineering and AI Infra roles—not just a resume bonus.

The Contours of Cloud Native’s “Second Half”

The table below summarizes the technical focus and essential differences of the “second half.”

It highlights the key coordinates of AI-Native Platform engineering.

Direction	Technical Focus	Essential Difference from First Half
AI Native Platform	Models/Agents as first-class citizens, unified abstraction & governance	Objects shift from services to models & inference
Resource Scheduling	DRA, heterogeneous compute, topology awareness, power & cost	From static quotas to dynamic, policy-driven
Runtime	Container + WASM + Nix + Agent Runtime	From “process containerization” to “execution graph containerization”
Platform Engineering	IDP + AI SRE + Security + Cost + Compliance	From toolset to “autonomous platform”
Security & Supply Chain	LLM dependencies, model weights, datasets, vector store governance	Protection expands from images to “all AI engineering assets”
Open Source & Ecosystem	AI Infra / Model Runtime / Agent Runtime upstream collaboration	Not just “using open source,” but “building the future in open source”

Table 3: Cloud Native Second Half Technical Coordinates

Summary

Over the past decade, cloud native evolved from container orchestration to platform engineering 1.0. With KubeCon NA 2025 as a milestone, the industry systematically brings AI into cloud native technology and organizational stacks:

Kubernetes is no longer just “infrastructure for microservices,” but “runtime for AI workloads”
Platform Engineering is no longer just “tool integration,” but “autonomous platforms for models and agents”
Security, SRE, runtime, scheduling, and networking will all be reimagined under AI

For me, the past ten years were about “making applications more stable in the cloud native world.” The next ten will focus on “making AI better, safer, and more controllable in the cloud native world.” This is, in my view, the opening whistle for cloud native’s “second half.”

NotebookLM: My Most Recommended AI Tool for Learning and Knowledge Organization

Jimmy Song — Mon, 17 Nov 2025 08:44:45 +0000

NotebookLM is the most tailored AI tool I’ve used for knowledge workers. It truly helps me structure massive information and dramatically boosts my learning and content creation efficiency.

As a lifelong learner who reads technical specs and researches open-source projects, I’ve always sought a tool that can “shortcut” my way through mountains of material, reduce mechanical reading, and help me quickly build a global understanding. NotebookLM has been the smoothest and most reliable experience for me over the past year.

It’s not a traditional “chat-style AI tool”—it’s more like an AI-native learning and content organization system that ingests your materials, organizes them, and presents them in various structured formats. The more I use it, the more I realize its help in learning new technologies, understanding unfamiliar fields, organizing large project documents, and building teaching materials—things that general large language models (LLM, Large Language Model) simply can’t match.

The Core Value NotebookLM Brings Me

NotebookLM has significantly improved my workflow, especially in learning new technologies, organizing documents, and content creation.

Quickly Understanding New Technologies: Feed in Complex Materials, Get a “Learnable Version”

My most frequent and indispensable scenario is learning a completely unfamiliar technology or development framework. Faced with dozens or even hundreds of pages of documentation, my typical approach is:

Add official docs, README files, design documents, and architecture diagrams into a single Notebook
Let NotebookLM generate:
- Study guides
- Briefings
- Key knowledge points
- FAQs
- Quizzes
Ultimately, I get a clearly structured “learning entry point” instead of a flood of raw materials.

The following flowchart illustrates how NotebookLM compresses complex documents into a learnable structure:

Figure 5: NotebookLM Document Structuring Flow

In the end, what I gain is an “organized knowledge system” rather than a pile of PDFs waiting to be consumed.

Generating MindMaps: Instantly Turning Large Documents into Structured Knowledge Graphs

I rely heavily on MindMaps to build the “skeleton of knowledge.” NotebookLM’s MindMap feature stands out for:

Automatically identifying relationships between topics
Interactive node expansion and collapse
Integrating multiple source documents

Although it currently only exports PNG, the logical structure itself is already an excellent “knowledge compression.”

The table below compares the auto-generation and visualization capabilities of different tools:

Tool	Auto-Generation	Multi-Doc Integration	Visualization Quality	Export Formats
NotebookLM	Strong	Strong	Good	PNG only (SVG not yet supported)
Common LLM Tools	Weak	Weak	Poor	Depends on tool
MindMap Software (Manual)	None	None	Strong	Fully supported

Table 7: Comparison of MindMap Capabilities in Mainstream Tools

NotebookLM’s greatest advantage is automation.

Generating Teaching Outlines, Training Scripts, and Book Structures: Truly Saving Me Time

NotebookLM is more than just “summarization”—it can generate formal teaching structures based on my prompts. By feeding in project docs, API references, architecture designs, case studies, videos, and blogs, and prompting it to generate:

Teaching outlines
Project training manuals
Course structures
Book chapter frameworks
Slide text
Training case descriptions

For anyone who needs to create content, conduct training, or give presentations, this feature is a huge time-saver.

Below is a typical prompt I actually use:

Based on the provided content excerpts, write a detailed training manual that systematically explains the core principles covered. The manual should use a professional and instructional tone, breaking down complex concepts into actionable steps and lessons. Ensure all content is strictly based on the source material and covers every aspect mentioned.

The training manual should include:
1. Training objectives and expected outcomes
2. Training content and structure
3. Training methods and tools
4. Training evaluation and feedback
5. Training summary and follow-up actions
6. Training cases and examples
7. Training resources and references

The results are often surprisingly good.

Multi-Format Input Capability: The Most Stable I’ve Seen

NotebookLM supports direct ingestion of various material types, with extremely stable parsing. The table below summarizes my actual experience:

Input Type	My Actual Experience
PDF	Most stable, clear structure parsing
Google Docs	Syncs instantly, very smooth
Word / PPT	Recognized normally
YouTube Video	Auto-summary + key content extraction, very useful
Website URL	Depends on site structure, high success rate
Plain Text	No issues
Images	Partial success, sufficient for screenshots

Table 8: NotebookLM Multi-Format Input Experience

By contrast, other tools often have format parsing issues, garbled text, missing content, or skipped paragraphs. NotebookLM is especially stable in “multi-format ingestion.”

My Most Common NotebookLM Workflow

The following flowchart shows my daily workflow with NotebookLM:

Figure 6: NotebookLM Daily Workflow

Essentially: let AI help me grasp the big picture → then dive deeper → then output content.

My Suggestions and Minor Regrets

NotebookLM is already excellent, but I still have some strong expectations for future improvements:

MindMap Export Formats Should Support SVG or Text-Based (Markmap)

Currently, only PNG is supported, which gets blurry when enlarged. The table below lists my expectations for future features:

Expected Feature	Purpose
SVG Export	For writing books, making slides, scalable without loss
Markmap Output	Most friendly for Markdown writers
Raw JSON	Allows custom rendering

Table 9: Expected MindMap Export Formats

I’m especially looking forward to NotebookLM supporting Markmap format export, which would be extremely friendly for users who write blogs and docs in Markdown.

Recently, Google also launched CodeWiki, similar to DeepWiki, which auto-generates image-rich Wikis for GitHub projects, but currently does not support Mermaid or Markmap.

Conversation History Should Support Long-Term Saving

Currently:

Chats are not persistently saved
Only manually “add to notes” preserves results

This causes some knowledge context to be lost. I hope to see a “Notebook conversation history” feature in the future.

Slide Generation Should Support Templates for Content Creators

Currently, Video Overview offers various visual styles, but cannot:

Upload custom PPT templates
Apply enterprise/personal branding templates

If PPT template support is added, NotebookLM could become the “video generation hub” for content creators.

Deep Research Should Launch Soon and Be Fully Open

I’m especially looking forward to this feature, as it could upgrade NotebookLM from a “knowledge organization tool” to a “research-grade tool.” I hope it will:

Reliably crawl more public web pages
Ensure citation quality
Integrate with existing Notebook materials

This is a major upgrade I personally care about.

Mobile Experience Should Be Enhanced Beyond Content Playback

Currently, the mobile experience is minimal, only allowing:

Listening to audio
Viewing Notebook Guide summaries
Simple Q&A

I hope mobile will soon support:

Editing Notebooks
Deep conversations
MindMap interaction
Content output (generating docs, outlines, etc.)

Summary

NotebookLM is truly one of the AI tools I use every single day because it achieves a critical goal:

Organizing information, structuring knowledge, so I don’t have to start from scratch with massive documents.

Whether it’s:

Learning new technologies
Reading long documents
Creating courses
Conducting training
Writing books
Drafting speeches
Summarizing content

It saves me a huge amount of time upfront, letting me focus on “understanding” and “creating.”

I’ll continue to use NotebookLM as one of my essential tools and keep an eye on its progress in Deep Research, template systems, and mobile features.

This is a tool truly designed for “knowledge workers” and deserves to be known by more people.

Helm v4: Paradigm Convergence and Plugin System Rebuild

Jimmy Song — Fri, 14 Nov 2025 11:18:30 +0000

The release of Helm 4 is not just a technical upgrade, but a deep convergence of cloud-native delivery paradigms. The rebuilt plugin system and supply chain governance capabilities make Helm once again a driving force in the Kubernetes ecosystem.

Since its first release in 2016, Helm has been one of the most important application distribution tools in the Kubernetes ecosystem. Helm v4 is not a “minor enhancement,” but a comprehensive update around delivery methods, extension mechanisms, and supply chain approaches.

This article reconstructs Helm’s historical context and focuses on why Helm 4 represents a paradigm-converging release.

Helm: From Tiller to Declarative Delivery

Below is a textual timeline showing key milestones from Helm v2 to v4, helping you understand its technical evolution:

2016: Helm v2 released, using the Tiller architecture.
2017: Chart Hub expands, major projects begin providing official Charts.
2018: Security model controversies intensify, Tiller’s permission issues become apparent.
2019: Helm v3 released, Tiller removed, OCI support introduced.
2021: GitOps becomes widespread, Server-Side Apply (SSA) becomes the mainstream delivery semantic.
2023: kstatus widely adopted for controller status assessment and health calculation.
2025: Helm v4 released, bringing SSA, WASM plugins, reproducible builds, and content hash caching.

Each major Helm release closely follows Kubernetes paradigms, driving progress in declarative delivery and ecosystem tooling.

Fundamental Changes in Helm v4

This section analyzes the core technical upgrades and paradigm shifts in Helm v4.

Delivery Paradigm Update: Default Server-Side Apply (SSA, Server-Side Apply)

In Helm v3 and earlier, Helm used a “three-way merge” model for resource delivery. Helm v4 fully switches to Server-Side Apply (SSA, Server-Side Apply), meaning the API Server determines field ownership.

This shift brings several direct results:

Full semantic alignment with kubectl apply and GitOps controllers (such as Argo, Flux)
When multiple controllers manage the same object, silent overrides are avoided and conflicts are explainable
Helm’s behavior now follows Kubernetes’ officially recommended declarative delivery paradigm

The following flowchart compares the delivery semantics of Helm v3 and v4.

Figure 3: Helm v3/v4 Delivery Semantics Comparison

Helm is now aligned with the delivery semantics of modern Kubernetes versions, improving predictability and safety in resource management.

kstatus-Driven Wait Behavior and Readiness Annotations

In Helm 3, --wait could only make fuzzy status judgments on limited resources, lacking extensibility and explainability.

Helm 4 introduces kstatus (Kubernetes Status) as the basis for health status parsing, and supports two key annotations:

helm.sh/readiness-success
helm.sh/readiness-failure

Chart authors can precisely define conditions for installation success or failure. Helm’s waiting model now offers “explainability + extensibility,” upgrading from a “templating tool” to a true “deployment orchestrator.”

Extension System Rebuild: WASM Plugin System

Helm 4 thoroughly reconstructs the plugin model, mainly including:

Typed and Structured Plugins

Arbitrary scripts are no longer allowed; plugins must follow typed and structured standards

WebAssembly Plugin Runtime (Extism)

More secure (sandbox isolation)
Cross-language support
Easy unified management in CI/CD and enterprise platforms
Predictable and testable

Post-renderer Integrated into Plugin System

Moves beyond the “external executable black box” era
Helm becomes a programmable platform, not just a template renderer

Engineering Capabilities Upgrade: Reproducible Builds, Content Hash Caching, chart API v3

Helm v4 brings the following engineering improvements:

Chart packaging is reproducible (supports signing, SBOM, SLSA, etc. for supply chain governance)
Local cache uses content hashes, avoiding version-based conflicts
chart API v3 (experimental) is stricter and more flexible
SDK logging system upgraded to Go slog (modern logging)

These capabilities enable Helm charts to enter serious software supply chain governance.

Feature Comparison (Helm v3 → v4)

The table below compares core features between Helm v3 and v4 for a quick understanding of the upgrade value.

Area	Helm 3	Helm 4
Apply Model	Three-way merge	Default SSA
Wait Behavior	Fuzzy, not extensible	kstatus + annotation
Plugin System	Script, uncontrollable	WASM, typed plugins
Post-renderer	External executable	Plugin subsystem
Build	Not reproducible	Reproducible build
Cache	name/version	Content hash
Chart API	v2	v2 + v3 (experimental)
SDK Logs	stdlib log	slog

Table 3: Helm v3 vs v4 Feature Comparison

This is a release that “repays technical debt in bulk + aligns with contemporary Kubernetes semantics.”

Why Is Helm v4 a Paradigm Convergence Event?

The release of Helm v4 is not just a feature upgrade, but a deep convergence of delivery paradigms, mainly in three aspects:

Kubernetes Delivery Semantics Unified to SSA

Previously: kubectl, GitOps, and Helm each had their own logic. Now: All unified to SSA, consistent delivery behavior, smoother ecosystem collaboration.

Plugin System Enters the Platform Era

WASM (WebAssembly) brings a secure, universal, and controllable plugin runtime. Infrastructure projects widely adopt WASM: Envoy → WASM Filters, Kubernetes → WASM CRI/OCI, and now Helm joins the platform camp.

Charts Enter Supply Chain Governance

Reproducible builds and digest verification allow Helm charts to be managed as seriously as container images, greatly enhancing supply chain security.

The entire ecosystem moves to a unified capability baseline, driving cloud-native delivery standardization.

My Helm History and Observations

As an early user from the Helm v2 era, I have experienced the following stages:

Tiller security controversies
v3 migration (state stored in secrets)
Large-scale chart consolidation in the community
OCI adoption
Today’s SSA / WASM / reproducible build

Each major Helm version upgrade is not about chasing trends, but proactively aligning with Kubernetes paradigms:

v3 aligns with K8s “no cluster-side runtime” principle
v4 aligns with SSA, kstatus, WASM, OCI, and other advances from the past five years

Helm exemplifies the evolution rhythm of infrastructure projects: not by piling on features, but by evolving in semantic alignment with the platform.

Summary

The release of Helm v4 marks a new paradigm for Kubernetes application delivery. SSA, WASM plugins, kstatus, and reproducible builds make Helm not just a templating tool, but a core for supply chain governance and platform extensibility. For cloud-native developers and platform teams, Helm v4 is a paradigm upgrade worth attention.

References

Kimi K2 Thinking: The True Awakening of China's Thinking Model

Jimmy Song — Fri, 14 Nov 2025 08:25:26 +0000

China’s large language models have finally moved from “writing like humans” to “thinking like humans.” The open-sourcing of Kimi K2 is a watershed moment for China’s AI trajectory.

The narrative around China’s large language models is shifting from “Chat-style models” to “Thinking models (Thinking Model, Thinking Model).”

Moonshot AI’s open-sourcing of Kimi K2 Thinking marks the first real landing of this transition. K2 is not just another iteration like ChatGLM or Qwen; it’s the first time a Chinese team has unified “deep reasoning + long context + tool invocation continuity” in training. This is the core of the thinking model approach and the reason why models like Claude and Gemini have led the field.

The Significance of K2’s Open Source: China Enters the Era of Thinking Models

Why is K2’s open source a turning point? Because it enables Chinese models to achieve the following capabilities for the first time:

Stable execution of 200–300 tool invocations (toolchain reasoning stability)
Deep, multi-stage reasoning chain execution (CoT Consistency, Chain-of-Thought Consistency)
256k context as a “working memory” (Working Memory, Working Memory)
Native INT4 acceleration + MoE activation sparsity scheduling

This is a completely different path from “stacking parameters → stacking benchmarks,” emphasizing reasoning ability over parameter scale.

In short:

K2 is the first time a Chinese model has entered the sequence of thinking models (Thinking Model, Thinking Model).

Dissecting K2’s Technical Approach

K2’s technical approach can be broken down into five key points, each directly impacting the model’s reasoning ability and ecosystem adaptability.

MoE Expert Division: Cognitive Division Rather Than Parameter Expansion

K2’s MoE (Mixture of Experts, Mixture of Experts) design philosophy is distinct from previous models. The core is not about activating fewer parameters or running larger models more cheaply, but about assigning different cognitive sub-skills to different experts. For example:

Mathematical reasoning expert
Planning expert
Tool invocation expert
Browser task expert
Code generation expert
Long-chain retention expert

This division aligns directly with Claude 3.5’s cognitive layering (Cognitive Layering, Cognitive Layering) approach. K2’s MoE is about “dividing thinking among the model,” not just “making computation cheaper.”

256K Context: Building the Model’s Working Memory

K2’s ultra-long context is not just a parameter showcase; it’s designed to build the model’s “thinking buffer.” It allows the entire process to retain reasoning chains, tool invocation states, multi-stage reflection, and uninterrupted long tasks (such as research or code refactoring), stably executing multi-stage agent workflows. Long-term thinking requires long-term memory support, and K2’s long context is the “memory” for sustained reasoning chains.

Intertwined Training of Tool Invocation and Reasoning Chains

K2 excels in the intertwined training of tool invocation and reasoning chains. Traditional open-source models typically follow this process:

Generate reasoning
Output JSON function call
Tool returns result
Continue reasoning

In this approach, the reasoning chain and invocation chain are separated. K2’s training allows the reasoning chain to invoke tools at any time and feed tool results back into the reasoning chain for the next stage of thinking. It supports 200–300 consecutive tool invocations without interruption, fully aligning with Claude 3.5’s Interleaved CoT + Tool Use.

Native INT4 Quantization: Ensuring Reasoning Chain Stability

K2’s INT4 (INT4, 4-bit Integer Quantization) approach is not ordinary post-quantization. Its purpose is not only to reduce memory usage and increase throughput, but more importantly, to ensure that deep reasoning chains do not break due to insufficient computing power. The biggest killer of deep thinking chains is timeout, freezing, or unstable workers. INT4 enables Chinese GPUs (non-H100) to run complete reasoning chains, which is highly significant for China’s ecosystem.

MoE + Long Context + Toolchain: Unified Training Rather Than Module Stitching

K2’s most important feature is its holistic training approach: expert division, long context-driven consistency, tool invocation trained through real execution, browser tasks and long-step task reinforcement, and INT4 entering the training loop. It’s not a “ChatLLM + Memory + RAG + Tools” patchwork, but an integrated reasoning system.

Alignment and Differences Between K2 and International Mainstream Approaches

K2 is highly aligned with international mainstream models (such as Claude, Gemini, OpenAI) in cognitive reasoning, ultra-long context, and tool invocation mechanisms, but also has unique advantages for Chinese models:

Native INT4 + adaptation to Chinese computing power is rare globally
Toolchain continuity is more stable than most open-source models
Higher degree of open source, stronger ecosystem reusability

Collaborative Value of China’s AI Infra: K2 × RLinf × Mem-alpha

A series of important open-source infrastructures have emerged in the K2 ecosystem. The table below summarizes these project types and their value to K2:

Here is a comparison table of the collaborative value of each infrastructure with K2:

Project	Type	Value to K2
RLinf	Reinforcement Learning	Used to train stronger planning/browser task capabilities
Mem-alpha	Memory Enhancement	Can be combined with K2 to form long-term memory agents
AgentDebug	Agent Error Debugging	Used to analyze K2’s toolchain errors
UI-Genie	GUI Agent Training	Can serve as an experimental field for K2’s agent capability expansion

Table 3: Collaborative Value of China’s AI Infra Ecosystem

This combination is already forming a China AI Agent Infra Stack.

Personal View: The Significance of K2’s Approach

I believe the significance of K2 lies not in the model itself, but in its technical approach:

K2 marks the first time Chinese models have shifted from “language generation competition” to “thinking ability competition.”

For the past three years, the main line of China’s open-source models has been evaluation scores, parameter scale, instruction following, and alignment data. But K2 is the first to clearly take the path of deep reasoning, tool intertwining, cognitive division, long-term task chains, and native performance optimization. This means China’s model trajectory is now synchronized with the US, rather than chasing old paths.

Key Directions to Watch in K2’s Ecosystem Over the Next Year

K2’s future ecosystem influence will depend on several key points:

Whether it opens the tool registry (Tool Registry, Tool Registry)
Whether it supports dynamic memory (Mem-alpha integration)
Whether it opens the MoE expert structure
Whether it can form a Chinese reasoning chain optimization path with vLLM / llm-d / KServe
Whether it supports fault tolerance for multi-node continuous reasoning chains

These capabilities will determine K2’s ecosystem influence and technical extensibility.

K2 Thinking Model Architecture Diagram

The following flowchart illustrates the core architecture of the K2 thinking model and its collaboration with external agents/applications:

Figure 3: K2 Thinking Model Architecture

Summary

K2 is the first time China’s model trajectory is heading in the right direction:

From “writing like humans” to “thinking like humans.”

The era of thinking models is coming, and Chinese models are finally standing on the same roadmap as the international forefront.

References

Beyond Gateway: Inference Traffic Control Practice with Gateway API Inference Extension

Jimmy Song — Fri, 14 Nov 2025 08:19:16 +0000

AI inference traffic governance is undergoing a paradigm shift. The Gateway API Inference Extension makes “model awareness” the new 主线 of traffic control.

Brief Review of Gateway API’s Current State

Kubernetes Gateway API has entered the stable v1 series and continued iterating after the 1.0 GA release, enhancing advanced traffic governance capabilities such as WebSocket, timeout and retry, Service Mesh integration, GRPCRoute, request mirroring, CORS, Retry Budget, and more. Major cloud providers and gateway implementations (such as Alibaba Cloud ACK, GKE Gateway, Envoy Gateway, NGINX Gateway Fabric) have all adopted Gateway API as the new generation north-south traffic model.

Building on this foundation, the community has proposed an extension specification specifically for AI inference traffic - the Gateway API Inference Extension. This specification is not about “reinventing an API” but rather supplementing the Gateway API core model with model-aware load balancing and traffic control capabilities for Large Language Model (LLM) and other inference scenarios.

Early documentation often mentioned the InferenceModel + InferencePool CRD combination; the latest specification has evolved to InferenceObjective + InferencePool (with optional InferencePoolImport). This article consistently uses the latest terminology.

Typical Challenges in GenAI Inference Traffic

Traditional gateway, Ingress, and Service Mesh load balancing models are essentially “request-agnostic + endpoint-agnostic”: they distribute traffic across a group of static backends through algorithms like round-robin, least requests, and hashing.

In GPU-powered Large Language Model (LLM) inference scenarios, this model exposes obvious problems:

Invisible GPU utilization and queuing

A single LLM instance simultaneously maintains KV Cache, LoRA Adapter, and Token queues. Resource consumption varies significantly across the same batch of requests. Load balancing based solely on QPS or connection count can easily lead to extreme situations where “idle GPUs have no work while busy GPUs crash.”
Lack of semantic binding between models and requests

From the business perspective, one typically only sees a POST /v1/chat/completions endpoint, making it difficult to express intentions like “high-priority model / test version / canary weight” at the routing layer.
Difficult unified management of multiple models, versions, and LoRAs

Each model service implements its own routing and A/B testing solutions, making it difficult for the platform to achieve unified governance and observability at the control plane level.

The goal of Gateway API Inference Extension is to introduce AI-specific semantics and metrics into load balancing and traffic control decisions while maintaining the existing Gateway model.

Core Concepts and Resource Model of Gateway API Inference Extension

This section introduces the overall architecture and key resource objects of the Inference Extension.

Overall Architecture

The Inference Extension uses Envoy External Processing (ext-proc) mechanism to upgrade Gateway API + ext-proc capable gateways (such as Envoy Gateway, kgateway, GKE Gateway) into Inference Gateways. Requests still follow the standard Gateway + HTTPRoute path, but before being forwarded to the backend, they pass through an “Endpoint Picker” component that selects the most suitable backend instance based on real-time metrics exposed by the model server.

The flowchart below shows the overall architecture:

Figure 3: Inference Extension Architecture Flow

InferencePool: Platform-Side “Model Service Pool”

InferencePool is the core resource introduced by the Inference Extension, used to describe a group of inference Pods and routing plugin configurations. It is similar to a Service with a selector, responsible for selecting a group of model service Pods and specifying exposed ports, while also allowing attachment of Endpoint Picker plugins (such as Prefix-Cache-Aware, LoRA-aware, etc.).

In the Gateway API model, InferencePool is treated as a type of “Backend” that can be referenced by HTTPRoute.backendRefs.

The code block below shows a simplified example of InferencePool:

apiVersion: inference.networking.x-k8s.io/v1
kind: InferencePool
metadata:
 name: vllm-llama3-chat-pool
spec:
 targetPortNumber: 8000
 selector:
 app: vllm-llama3-chat
 extensionRef:
 name: prefix-cache-aware-endpoint-picker

The meaning of the above configuration is as follows:

Select all Pods with app=vllm-llama3-chat, port 8000.
Use the plugin named prefix-cache-aware-endpoint-picker to make routing decisions based on metrics such as KV Cache hit rate, queue length, and GPU utilization.

InferenceObjective: Business-Side “Request Objective”

InferenceObjective is used to express the goal and priority of a single request, decoupled from the model service pool. One request corresponds to one InferenceObjective, and the same InferencePool can serve multiple different InferenceObjectives.

Typical fields include:

Business criticality (Critical / High / BestEffort)
Required model family / version preference
Acceptable latency / cost upper limits, etc.

The Endpoint Picker can combine InferenceObjective with backend metrics to make decisions: prioritize Critical requests when resources are constrained, and shed BestEffort requests when necessary.

InferencePoolImport: Cross-Cluster / Gateway Reuse

InferencePoolImport supports importing InferencePools defined in remote clusters into the local cluster, facilitating consistent governance of multi-cluster, multi-region inference services.

Maturity and Implementation Ecosystem of Inference Extension

The current project version is v1.1.x, overall in the Alpha stage. Official recommendation is not to use it directly in production yet; it’s more suitable for platform teams to experiment with the technology stack.

Multiple implementations and integrations already exist:

Inference Gateway implementations for Envoy Gateway / kgateway
GKE Inference Gateway: Enhanced capabilities based on GKE Gateway, including KV Cache-aware routing, LoRA reuse, priority scheduling, etc.
NGINX Gateway Fabric, cloud provider ACK, and others are also following up on related extensions

Therefore, when designing practical solutions, it should be treated as a “forward-looking solution / future main path.” For production environments, prioritize managed implementations (such as GKE Inference Gateway) or commercial products from gateway vendors.

Practice: Using Inference Extension for Inference Traffic Control

The following example uses a self-hosted LLM cluster on Kubernetes to demonstrate how to serve external traffic through an OpenAI-compatible interface and achieve:

Scheduling by request priority (real-time conversation vs. batch processing)
Multi-version model canary and rollback
Optimized routing using GPU metrics and KV Cache hit rates
Unified platform-side observability and rate limiting entry point

Deploying Gateway API and Inference Extension

The deployment process is as follows:

Install a Gateway API-compatible gateway implementation (such as Envoy Gateway, kgateway, GKE Gateway) in the cluster.
Install the Gateway API Inference Extension CRD and control plane components.
Enable the metrics endpoints and plugin protocols required by Inference Extension on the model server side (such as vLLM, Triton, TGI).

Defining InferencePool: Abstracting LLM Pods as an “Inference Pool”

The code block below shows a typical InferencePool configuration:

apiVersion: inference.networking.x-k8s.io/v1
kind: InferencePool
metadata:
 name: chat-llama3-pool
spec:
 targetPortNumber: 8000
 selector:
 app: chat-llama3
 extensionRef:
 name: prefix-cache-aware
 # Optional: Plugin configuration ConfigMap / CR

Key points:

selector selects LLM Pods, targetPortNumber specifies the inference service port.
extensionRef binds the Endpoint Picker plugin, implementing KV Cache prefix-aware routing, selecting replicas with lighter load based on metrics like queue_length/gpu_utilization, and triggering load shedding when necessary.

Defining InferenceObjective: Connecting “Business Intent” to Routing Decisions

The code block below shows a sample InferenceObjective configuration (fields can be adjusted according to the actual version and implementation):

apiVersion: inference.networking.x-k8s.io/v1
kind: InferenceObjective
metadata:
 name: chat-critical
spec:
 criticality: Critical  # Real-time conversation
 preferredModel: llama3-70b
 fallbackModel: llama3-8b
---
apiVersion: inference.networking.x-k8s.io/v1
kind: InferenceObjective
metadata:
 name: chat-batch
spec:
 criticality: BestEffort  # Batch analysis, log summarization
 preferredModel: llama3-8b

The Endpoint Picker can combine InferenceObjective with InferencePool metrics to make the following decisions:

When GPU is constrained and queues are too long, prioritize chat-critical requests and discard chat-batch if necessary.
For Critical requests, prioritize the large model; if the target pool is unavailable, fall back to the small model pool.

Importing Business Traffic to InferencePool via HTTPRoute

The code block below shows the HTTPRoute configuration on the Gateway API side:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: llm-route
spec:
 parentRefs:
 - name: public-gateway
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /v1/chat/completions
 backendRefs:
 - group: inference.networking.x-k8s.io
 kind: InferencePool
 name: chat-llama3-pool

Some implementations (such as GKE Inference Gateway) can route based on the model field in the request body, mapping OpenAI-style model names to different InferencePool / InferenceObjective combinations.

Fine-Grained Inference Traffic Control Using Metrics

Inference Extension provides platform teams with a unified metrics system, including kv_cache_hits, gpu_utilization, request_queue_length, per-request inference duration, token count, and more.

Based on these metrics, multi-level traffic control strategies can be built:

Priority + Capacity

Set priority and capacity upper limits for different InferenceObjectives, automatically guaranteeing critical business when resources are constrained.
Rate Limiting by Cost / Token

Aggregate token / latency metrics exposed by Inference Extension to Prometheus, then add cost-based rate limiting logic at the gateway / API Gateway level (such as total tokens per minute per user / application). The specification itself doesn’t mandate “Token-level rate limiting,” but provides observability and hooks for easy policy extension.
Prefix Cache Aware Routing

For requests with shared context (such as RAG, template generation), enable the Prefix Cache Aware plugin to route requests with the same prefix to the same replica, maximizing KV Cache hit rates and significantly reducing TTFT.
Auto-Scaling Integration

Use metrics output by Inference Extension as HPA input to achieve “model-aware” auto-scaling, rather than relying solely on CPU / memory.

Relationship and Trade-offs with Traditional Gateway / Service Mesh

Control Plane: Continue using Gateway API as the unified north-south / east-west traffic modeling specification. Service Mesh can still perform fine-grained circuit breaking, retry, mTLS, etc. within the cluster.
Data Plane: Inference Extension pushes “model-aware routing” down to the ext-proc path implemented by the Gateway, avoiding redundant business-side wheel reinvention.
Adoption Strategy: In the current Alpha stage, prioritize productionized implementations (such as GKE Inference Gateway, commercial gateway vendor Inference Gateways), start with “bypass pilots” outside the critical path, and gradually migrate existing AI gateway routing rules to the Inference Extension model.

Summary

Combining official documentation and community implementations, we can see:

Gateway API has become the Kubernetes standard north-south traffic model and continues to enhance in the 1.x series.
Gateway API Inference Extension introduces GPU metrics, KV Cache, LoRA, and other inference semantics into load balancing decisions through InferencePool, InferenceObjective, and Endpoint Picker.
The project is still in Alpha stage, but has achieved experimental or productionized adoption in implementations such as GKE, kgateway, and NGINX Gateway Fabric. It is one of the important future directions for inference traffic control.

Previous descriptions of Inference Extension as “built-in Token rate limiting CRD, AIInferencePolicy, and other objects” are no longer accurate and should all be replaced with the design based on InferencePool / InferenceObjective + metrics-driven approach.

References

TRAE SOLO vs VS Code: Rethinking Coding Tools from the Perspective of AI Engineering Entities

Jimmy Song — Fri, 14 Nov 2025 07:14:39 +0000

Coding tools are evolving from “AI assistants” into true engineering entities. How can we reinterpret the roles of TRAE SOLO and VS Code from a pipeline perspective?

Recently, TRAE International Edition SOLO mode has been fully opened to overseas users. It claims to be a “responsive coding agent” and is now available for official trial, with token-based rate limiting.

I’ve used early versions of TRAE (without SOLO access), and also tried Qoder and Kiro. The AI coding field is flourishing, each tool with its own strengths.

This article compares TRAE SOLO and VS Code (with Copilot, Plan/Agent mode, and Agent HQ) from the perspective of AI engineering entities, combining personal experience to outline their differences in engineering automation, collaboration, and governance.

Three Engineering Role Abstractions: End-to-End Executor, Contextual Collaborator, and Expert Orchestrator

From an engineering perspective, current mainstream AI coding tools can be abstracted into three roles:

End-to-End Executor:

Focuses on “requirement to deployment” workflows, capable of autonomous planning, task breakdown, coding, testing, previewing, and even deployment. Officially called “AI-Powered Context Engineer.”
User experience: like a “full-chain executor”—give it a requirement, and it handles the project, even if it’s slow or imperfect.

Contextual Collaborator:

VS Code is a powerful editor. Copilot has evolved from line-level completion to Chat, Plan agent, Agent mode, supporting multi-step tasks and codebase analysis.
It doesn’t take over the whole project, but efficiently handles local tasks under your guidance, acting as an automated unit for specific segments.

Expert Orchestrator / Specialist Engine:

GitHub’s Agent HQ is a “central platform for AI coding agents,” a unified control plane that can connect to OpenAI, Anthropic, Google, xAI, etc., run agents in parallel, and compare results.
Functions as an “expert orchestrator” for key steps—planning, review, refactoring, or decision-making—providing high-quality output without taking over the entire project.

These three roles correspond to the structure in “AI Engineering Entity (AIEE)”:

Single end-to-end executor (TRAE SOLO)
Contextual collaborator residing in the IDE (VS Code + Copilot)
Specialist orchestrator platform for multi-entity scheduling (Agent HQ)

Quick Product Status Check

To avoid memory bias, let’s clarify some key facts.

TRAE / TRAE SOLO

TRAE claims to be a “10x AI Engineer,” able to independently understand requirements and execute development tasks.
SOLO mode is GA for international users, emphasizing full-chain automation, available directly but with token limits.
Underlying open-source Trae Agent CLI can execute multi-step engineering tasks in real codebases.
TraeIDE’s official page shows built-in Claude 3.5/3.7, DeepSeek, etc., but the community notes slow integration of new models like Claude Sonnet 4.5.

So, “TRAE does not support Claude” is now inaccurate—at least officially, Claude models are included. Which model is used in SOLO mode and whether it’s exposed to users remains unclear; the experience still needs improvement.

VS Code + Copilot + Agent HQ

Copilot in VS Code now features Chat, Plan agent, Todo/multi-step execution:
- Plan mode analyzes codebases, generates execution plans, splits into Todos, then implementation agents execute step by step.
- Agent mode provides a more automated “multi-step companion programmer” experience.
GitHub launched Agent HQ at Universe 2025, integrating Copilot and third-party agents (Anthropic, OpenAI, Google, xAI, Cognition, etc.) into a unified control plane, supporting parallel runs and result comparison.

In short:

TRAE is like “embedding an engineering entity into the IDE.”
VS Code + Copilot is “adding a set of engineering entities to a mature IDE.”
Agent HQ is positioned as “headquarters for multiple engineering entities.”

Reconstructing Comparison Dimensions with the AI Engineering Entity Framework

In “AI Engineering Entity (AIEE),” the definition is:

AI has evolved from editor auto-completion to a formal node in the software supply chain, able to receive tasks, produce reviewable artifacts (PR/diff/report), pass tests/gates, and be replaced if it fails. It’s no longer just an “enhanced human developer,” but a “functional engineering unit” in the pipeline.

Based on this, we can reconstruct key comparison dimensions for TRAE and VS Code:

Existence as an Independent Functional Unit
Can it autonomously plan, implement, and produce PRs/reports from natural language requirements, without continuous human intervention?
Context Modeling Capability
Can it model across files, directories, terminal output, and browser content to form a stable engineering context?
Position in the Pipeline
Is it an enhancement layer within the IDE, or a formal node in CI/CD and code review flows?
Reviewability and Replaceability
Are its outputs standardized (PR, diff, report) and suitable for regular pipeline review and rollback?
Multi-Agent Collaboration Capability
Does it natively support multi-agent collaboration, or is it focused on single-agent enhancement?

TRAE SOLO vs VS Code: Engineering Entity Comparison Table

The following table summarizes the main differences from the engineering entity perspective. Note: VS Code includes Copilot Chat + Plan/Agent mode by default and can mount the Agent HQ ecosystem.

You can interpret this table as: “If AI is treated as an engineering entity in the pipeline, what roles do TRAE and VS Code play?”

Table:

Dimension	TRAE SOLO	VS Code + Copilot / Agent
Engineering Role	Single strong entity, directly handles end-to-end tasks from idea to deployment	IDE + multiple entities (Plan, Implementation, Review), IDE itself is the engineering base
Task Granularity	Project/feature level: from PRD-style description to full project scaffold, implementation, testing, preview	Function/file level mainly; Plan mode can scale to feature/subsystem level
Context Modeling	Emphasizes “context engineering”: reads codebase, terminal output, browser content as unified input for SOLO	Mainly codebase; Plan Agent generates plans based on code analysis, Agent mode schedules by plan
Automation	Can proactively modify files, run commands, tests, start local services, forming a complete loop	Plan/Agent can run commands, modify files, run tests, but is more dependent on your current project/workflow
Human Intervention	More “post-review”: let it run first, then review and fine-tune	More “in-process collaboration”: frequent intervention in planning, implementation, and review, with control points at each step
Output Form	Code changes, test results, preview; sometimes PRs/docs	Code completion, refactoring, PR comments, CodeQL reports, Plan/Todo lists
Multi-Agent	Core is the SOLO agent; other capabilities (like Trae Agent CLI) are extensions	Copilot itself is an agent; Agent HQ allows parallel competition among multiple agents
Model Transparency	Product exposes specific models poorly; users can’t tell which model is used	GitHub clearly marks Copilot’s model family; Agent HQ shows agent sources directly
Performance	Strong automation but slow; complex projects may stall at “thinking” stage; hard token limits	Stable response in familiar projects; mostly local changes, overall latency is controllable
Privacy & Compliance	Official and third-party reviews mention extensive telemetry/data collection; enterprise adoption needs extra evaluation	Copilot for Enterprise has clear data isolation/compliance, suitable for most enterprise governance needs

Table 3: TRAE SOLO vs VS Code Engineering Entity Comparison

From the table:

If you want “an AI engineering entity that takes full responsibility from requirement to deployment,” TRAE SOLO fits that role.
If you want “a stable engineering base + a set of pluggable entities,” VS Code + Copilot + Agent HQ fits better.

Workflow Comparison: Two Engineering Entity Pipelines

To clarify their engineering flows, the following diagram illustrates typical workflows for TRAE SOLO and VS Code.

Before the diagram, here’s an introductory sentence:
The following Mermaid diagram visually compares the engineering pipelines of TRAE SOLO and VS Code, highlighting their respective collaboration models.

Figure 3: TRAE SOLO vs VS Code Engineering Entity Pipeline Comparison

This diagram shows two typical collaboration models:

TRAE SOLO attempts to encapsulate “context aggregation → planning → implementation → testing → preview/deployment” within a single engineering entity, with user intervention only at requirement input and output review.
VS Code + Copilot + Agent HQ uses the IDE as runtime, with Plan/Implementation/Review agents corresponding to different roles. Agent HQ supports parallel agent competition, allowing developers to select the best solution.

Model Transparency, Speed, and Predictability

Based on personal experience, here are the model transparency and speed issues from the engineering entity perspective:

Model Transparency

TRAE currently exposes “which model is called” poorly; switching MAX mode only suggests “stronger model or higher quota,” but no clear feedback.
Community feedback notes slow integration of new models; some strong models (like Claude series) are available elsewhere but not yet in TRAE.

This means:

TRAE is hard to use as a “precisely configurable engineering unit,” more like a black box, making model change management in CI/CD or production pipelines difficult.
VS Code + Copilot + Agent HQ is stronger in standardization; GitHub clearly marks Copilot’s model family, Agent HQ uses agent source as the abstraction boundary.

Speed and Predictability

TRAE SOLO’s “slowness” comes from executing more steps (reading files, analyzing, planning, testing) and insufficient engineering process visualization. The UI shows “Thinking…” prompts, making it hard to tell if it’s stuck or planning.
VS Code’s Plan mode explicitly lists plans and Todos; Agent mode emphasizes “execution by plan,” letting users clearly see the entity’s work status, improving predictability.

Agent HQ Positioning: Single Entity vs Multi-Entity Headquarters

From a platform perspective, GitHub and TRAE differ as follows:

Agent HQ’s core idea: future development will rely on multiple specialized agents collaborating in parallel. GitHub is building “agent headquarters,” not a single engineering agent. Developers can schedule agents in a unified control plane, integrating with existing GitHub Flow (Issue, PR, Review, CI/CD).
TRAE is more like “proprietary IDE + agent + full-stack context engineering,” delivering an integrated experience.

In terms of engineering entity organization:

GitHub is building “infrastructure and governance for multi-entity engineering systems.”
TRAE is building “vertically integrated engineering entity + private runtime.”

They’re not mutually exclusive, representing “broad platform + multi-entity scheduling” vs “single strong entity + proprietary toolchain.”

Subjective Experience and Engineering Framework Integration

Translating personal experience into engineering language:

VS Code is more accustomed to a “single IDE + multiple views” experience; TRAE splits IDE and SOLO modes, requiring mental switching.
TRAE’s engineering entity capabilities surpass ordinary completion tools, able to take on tasks, but model transparency and context quality are unstable, and governance needs improvement.
VS Code doesn’t take over the whole project, but local work is stable; Plan, Agent, and Review combinations enable multi-entity collaboration.

According to the “AI Engineering Entity (AIEE)” framework:

TRAE SOLO is a single AI engineering entity (AIEE) capable of handling complete engineering tasks, but still has clear shortcomings in model transparency, engineering governance, and enterprise-level controllability.
VS Code + Copilot + Agent HQ is an infrastructure platform for multiple engineering entities, less aggressive in end-to-end outsourcing in the short term, but clearer in engineering consistency, model replaceability, and organizational governance.

Summary

This article systematically compares TRAE SOLO and VS Code (with Copilot, Agent HQ) from the perspective of AI engineering entities, focusing on automation, collaboration, and model transparency. TRAE SOLO is better suited for individual developers or small teams seeking end-to-end automation, while VS Code + Copilot + Agent HQ provides stronger infrastructure for multi-entity collaboration, enterprise governance, and engineering consistency. In the future, AI engineering entities will become formal nodes in the software development pipeline, and tool selection should be based on engineering needs and governance requirements.

References

Closed-Source Flagships Accelerate, Open-Source Ecosystem Forced to 'Synchronize'

Jimmy Song — Fri, 14 Nov 2025 04:21:07 +0000

Closed-source models are accelerating, while open-source ecosystems are forced to catch up. What engineers truly need to focus on is infrastructure and controllability—not just the surface-level “Twin” phenomenon.

Recently, I came across an email titled: “Every Big AI Model Now Has an Open-Source Twin”. Literally translated, it means “Every major closed-source model now has an open-source sibling.”

From the perspective of media or venture capital, this is an easy story to tell: a closed-source flagship model is released, the community quickly produces an open-source counterpart, and the narrative becomes “open source is catching up with closed source”—the ecosystem is thriving, innovation is accelerating, and the future looks promising.

But from the viewpoint of someone deeply involved in infrastructure, cloud native, and architecture, this narrative has several issues:

It equates “synchronized pace” with “matched capabilities.”
It overlooks the real factors that determine the ceiling: data, compute, and engineering systems.
It blurs a key fact: the open-source ecosystem is fundamentally in a reactive state, not leading in parallel.

This article breaks down the “Open-Source Twin” narrative from an engineering and infrastructure perspective, and shares the core issues I personally care about.

From “There’s a Twin” to “Forced Synchronization”: How the Narrative Changed

Let’s first outline the phenomenon.

In the past two years, the industry has repeatedly seen the following pattern:

Big tech releases a closed-source flagship model (e.g., GPT-5 series, Claude 4/4.5, Gemini 2.5).
Soon after, a batch of open-source counterparts emerge (e.g., Qwen, GLM, Yi, K2), aligning on parameter scale and benchmark metrics.
Media and community start using terms like “open-source twin,” “replacement,” and “counterpart.”

At this level, it’s easy to draw optimistic conclusions: open source has established full benchmarking capabilities; no matter how fast closed source runs, the community can keep up.

But the more critical question is: Who sets the pace, who defines the rules, and who bears the real cost?

The current structure is clear:

Pace is set by closed-source giants: They decide when to boost inference, extend context, push multimodality, or specialize reasoning (like the R1 series).
Open-source ecosystem passively initiates response mechanisms: Each closed-source upgrade triggers a new round of “open-source benchmarking.”

In other words, the current model isn’t parallel innovation or mutual stimulation—it’s closed source constantly shifting gears at the front, with open source adjusting to avoid falling out of sight.

From an engineering perspective, “Every Big AI Model Has an Open-Source Twin” is more accurately:

Every Big AI Model Now Forces an Open-Source Response.

What Exactly Is Open Source “Synchronizing” With?

To understand the “synchronized response” phenomenon, we need to break it down into three categories.

Before listing them, let’s add some context: every closed-source flagship update isn’t just “more parameters, higher scores”—it’s constantly rewriting constraints, including inference cost, interaction patterns, context length, multimodal consistency, and explainability.

In this context, open source isn’t just synchronizing “scores,” but increasingly complex objective functions.

Synchronizing the “Imagination Boundary” of Capability Ceilings

Closed-source models essentially expand “what people think a model should be able to do,” such as:

From pure text to text + image + audio + video.
From single-turn Q&A to engineering-level reasoning, coding, debugging, fixing, and refactoring.
From thousands of tokens of context to hundreds of thousands or more.
From “black box output” to having chains of thought, reasoning traces, and verifiable outputs.

Open-source models then align their goals: “We also need long context, multimodality, coding ability, and agent workflow support.”

Synchronizing the “Expectation Value” of Interfaces and Usage Patterns

Once developers and enterprise users are educated by closed-source models:

How low can interaction latency go?
How long can context be extended without breaking?
How smooth can multimodal input be?
How “smart” can the reasoning process get?

Their expectations for any open-source model are recalibrated.

Thus, open source must:

Continuously optimize inference frameworks (e.g., vLLM, SGLang, TGI) to narrow the latency gap.
Make serving experiences closer to closed-source, such as OpenAI API compatibility and better SDKs.
Forcefully catch up on multimodality and long context, even if training costs are high.

Synchronizing “Surface Metrics,” Not “Complete Capabilities”

From a benchmark perspective, open source can indeed reach 80–90% on public test sets:

MMLU, GSM8K, HumanEval.
Common reasoning, reading comprehension, code generation metrics.

But these metrics only reflect surface capabilities, not:

Robustness to long-tail problems.
Stability in complex, multi-step scenarios.
Reliability and controllability in large-scale production systems.
“Engineering health” over long-term evolution.

This is why I’m skeptical of the “Twin” term: it uses superficial metric similarity to mask deep structural differences.

Why “Open-Source Twin” Sounds Good but Misses the Core Contradiction

From an infrastructure and engineering perspective, the real issue isn’t “can open source copy a similar architecture,” but:

Who can sustainably manage data, compute, scheduling systems, and engineering teams to build a long-term model production pipeline.

There are three key contradictions here.

Data Is Unavailable, Training Recipes Can’t Be Fully Reproduced

Open source can replicate general network structures and optimization tricks, but can’t access:

Closed-source data sources and cleaning standards.
Filtering strategies, detoxification, alignment details.
Large-scale synthetic data generation and selection methods.

Result: Even if you match parameter scale and training steps, the effect may not truly align.

Many open-source projects have to use rougher data, limited compute budgets, and more conservative training strategies, ending up with a “usable but not truly stable” state.

Compute Gap Is Structural, Not Solved by One-Time Funding

Training flagship models requires compute that’s not just hundreds or thousands of GPUs—it’s a structural, long-term investment.

In the open-source camp, those approaching this scale usually have:

Backing from large companies or national labs.
Funding from real business budgets, not community donations.
Compute supply that can be planned long-term, not just a one-off “burn.”

Reality:

Entities truly capable of “flagship open-source models” are essentially “institutions,” not loose personal communities.
Most “open-source twins” are backed by enterprises, with product goals and commercial interests.

So, “open source vs closed source” is more like “many big companies vs a few giants,” not “community vs company.”

“Reproducing” Architecture ≠ “Leading” Architecture

Many open-source models look architecturally similar to closed-source:

Transformer variants, MoE variants.
Minor tweaks at the decision layer.
Some inference optimizations.

But in terms of industry power, those truly pushing these architectures to production scale and validating feasibility are still on the closed-source side. Open source mainly:

Validates closed-source approaches on weaker compute.
Explores “smaller, cheaper” approximations.
Prunes and adapts for specific scenarios.

So, “Twin” is more a marketing term than an engineering one.

My Perspective: What Really Matters in This Game

As an engineer in cloud native, service mesh, and distributed systems, my default thinking when looking at AI infrastructure is:

Treat “models” as just one component in the system, and focus on the underlying infrastructure, scheduling systems, and engineering pipelines.

From this angle, the real concern behind “every closed-source model has an open-source twin” is:

Figure 3: Mermaid Diagram

Can Open-Source Models Stand Firm in Production Over Time?

The focus isn’t whether it can run a demo, but:

Is there a clear upgrade cadence?
Are rollback and compatibility strategies robust?
Is there a sound evolution path for model weights, inference frameworks, and configurations?
Is the entire stack observable and debuggable?

If these infrastructure layers aren’t mature, the so-called “Twin” is just “something that looks similar, but don’t ask if it can support your production workloads.”

Has Training and Inference Infrastructure Formed a Replicable “Engineering Paradigm”?

The real value in open source is whether it can form a unified, teachable, and transferable engineering paradigm, such as:

Training pipeline: data preparation → preprocessing → training → evaluation → alignment → deployment.
Inference infrastructure: how vLLM / SGLang / TGI maintain consistent performance across different GPU topologies.
Scheduling and resource management: how to manage large-scale inference loads on Kubernetes and cloud-native infrastructure.

If these can be established, “open-source twin” isn’t just “we also have a model,” but a reusable, transparent, and learnable engineering system.

The True Value of Open Source: Controllability and Bargaining Power, Not Absolute Performance

Realistically, closed-source flagships will continue to lead in overall capability for the foreseeable future: larger scale, more complex training, better data, richer scenario tuning.

For enterprises and developers, the key value of open source isn’t “I want to fully replace closed source,” but:

Maintaining controllability over technical direction.
Gaining bargaining power, avoiding vendor lock-in.
Building your own model stack in privacy-sensitive or compliance-heavy scenarios.

From this perspective, the “Twin” term should be soberly rewritten as:

In many scenarios, open source can provide a controllable, more flexible alternative path—but it’s not a mirror of closed source, it’s a separate engineering decision space.

Practical Advice for Engineers and Teams: Don’t Worship “Twins,” See the Structure Clearly

Before the final summary, here are my actionable views on this topic.

Premise: If you’re an engineer, architect, or technical leader, your real decision isn’t “choose open source or closed source,” but:

Under your business constraints, how do you combine closed-source APIs, open-source weights, and self-built infrastructure to create an evolvable, observable, and portable system.

With this in mind, “Every Big AI Model Has an Open-Source Twin” breaks down into several sober judgments:

When you see an “open-source twin,” first ask: can it run stably in your production environment over time, not just pass benchmarks?
What you really need to understand: is there a clear story behind its training/inference infrastructure, not just a weight download link?
Reframe “open source vs closed source” as “where do I need closed source (capability/cost), and where do I need open source (controllability/compliance)?”
If you’re working on infrastructure and platform layers, focus on:
- How to run different models in a unified scheduling, monitoring, and logging system.
- How to treat large models as observable, governable services on Kubernetes/cloud-native stacks, not mysterious black boxes.

Summary

Closed-source flagships keep accelerating, shifting gears, and adding dimensions, while open-source ecosystems are forced to develop increasingly mature synchronized response mechanisms. What truly determines the gap is data, compute, and engineering infrastructure—not just a single model release.

Personally, my focus will remain on:

The evolution of inference infrastructure (like vLLM, SGLang, TGI).
Training and scheduling: how to stably manage model lifecycles in cloud-native environments.
Engineering paradigm accumulation: moving from “can run” to “reproducible, maintainable, and evolvable.”

Key Takeaways from Ingress NGINX Retirement: Managing Technical Debt in Cloud Native Migration

Jimmy Song — Thu, 13 Nov 2025 01:43:05 +0000

The evolution of cloud native infrastructure inevitably faces the reality of technical debt and governance. The retirement of Ingress NGINX is a profound reminder about standardization and sustainability.

Kubernetes officially announced: Ingress NGINX will be completely discontinued in March 2026. This is not just a typical project sunset, but a landmark event in the evolution of the Kubernetes networking model. It signals the inevitable shift of the tech stack from “flexible but fragile” to “controllable and governable.”

As someone who has long promoted Kubernetes and cloud native practices, I have witnessed both the golden age of Ingress NGINX and the gradual accumulation of its technical debt. Here are the clear insights this event has brought me.

Technical Debt Will Eventually Backfire, Especially for Infrastructure Components

The core issue with Ingress NGINX is not a decline in users, but that “maintenance costs permanently exceed the pace of contributions.” High flexibility leads to a huge attack surface, years of complex configuration legacy, and a shortage of community maintainers, ultimately making the project unsustainable.

Once infrastructure components can no longer be securely updated, they cease to be assets and become liabilities.

My Conclusion

The threshold for future infrastructure will be higher, with stricter requirements for security and maintainability. The model of individual hero maintainers will continue to fail.

Kubernetes Officially Enters the Gateway API Era

Before introducing Gateway API (Gateway API, Gateway Application Programming Interface), it’s important to review the design of Ingress. Ingress was once praised for its simplicity, but now it cannot meet modern needs for traffic management, scalability, security policies, and multi-team collaboration.

Gateway API is designed with a more modern philosophy:

Governance model across roles (Infra / Dev / Ops)
Strong CRD (Custom Resource Definition) extensibility
Pluggable implementation
Significantly improved observability and lifecycle management

This means: The entire ecosystem is moving from “controller differentiation” to “API standardization” at the traffic layer.

Most Users Are Unprepared for the Complexity of the Underlying Network Stack

Long-term community observation shows that most users treat Ingress NGINX as a black box. Now, migrating from Ingress to Gateway API or other Ingress controllers represents a “hidden migration wave” for many clusters.

This announcement highlights two points:

When a “default component” in a complex system stops being updated, it brings widespread invisible risks
The cloud native ecosystem needs long-term, sustainable supply chain governance

Security Is the Final Straw

The official announcement repeatedly emphasizes that security risks and vulnerabilities can no longer be continuously fixed. This once again proves: Flexibility and security are always a tradeoff, and the closer a component is to the data plane, the less compromise is acceptable.

The “Individual Maintainer Bottleneck” in Cloud Native Will Become More Pronounced

Ingress NGINX has long relied on just one or two maintainers, and ultimately had to retire. This exposes a long-standing issue in the open source world: Critical projects are heavily relied upon, but contributions are insufficient.

The future of infrastructure is clear:

Large companies will be more willing to invest in core open source infrastructure
Individual maintainers cannot support critical foundational components
The boundary between commercialization and open source will continue to tighten

My Personal Takeaway: Gateway API, L7 Traffic Management, and the Integration with AI Native Infra

The retirement of Ingress NGINX points to an underlying trend: Unified and extensible APIs will become the dominant paradigm for cloud native infrastructure.

The AI-Native infrastructure I’m researching—such as inference routing, model gateways, AI Gateway, and Agent Orchestrator—will follow a similar path: from early flexible hacks to mature, standardized, and governed APIs.

Summary

Ingress NGINX is arguably one of the most important control planes in Kubernetes history. Its retirement is not a failure, but an inevitable result of the system advancing to the next stage.

For me, this is a strong reminder:

Technical debt cannot be avoided
Infrastructure must be built for the long term
Standardized APIs are the future
Sustainable open source requires collective investment
The convergence of AI and cloud native will follow the same evolutionary trajectory