Jimmy Song – Jimmy Song's Blog

Core Model Overview

Jimmy Song — Tue, 10 Feb 2026 13:56:12 +0000

The Yin-Yang - Five Elements - Yun - Qi Model views AI Infrastructure as an organic whole, revealing its operational mechanisms from four dimensions. Each layer focuses on different fundamental questions:

Four-Layer Model

Layer	Name	Focus Question
Yin-Yang	State Layer	The system’s internal unity of opposites tension structure, revealing how dual elements like performance vs. constraints, innovation vs. governance coexist
Five Elements	Role Layer	Five basic role elements in the system and their collaborative relationships, breaking down complex infrastructure into data, models, compute, platforms, and hardware
Yun	Time Layer	The development stage the system is in and its cyclical patterns, describing the evolution cycle from exploration to platformization, then scaling and rebalancing
Qi	Flow Layer	The effective “field” of flow within the system, characterizing the conduction and feedback of signals and resources, reflecting the overall smoothness of operation

Table 3: Four-Layer Model

Model Interactions

The four-layer model is not isolated but an interconnected organic whole:

The tension of Yin-Yang permeates the dynamic balance of Five Elements
The development of Five Elements roles is constrained by their Yun stage
The flow of Qi connects the above elements into a self-adaptive cyclic system

The overview diagram below illustrates each layer of the model and their interactions:

Figure 3: AI Infrastructure ‘Yin-Yang - Five Elements - Yun - Qi’ Model Overview. The Yin-Yang layer embodies the system’s internal tension and unity of opposites, the Five Elements layer defines core role elements, the Yun layer describes system stage cycles, and Qi as a flow element permeates and drives the entire system.

Model Application Value

This four-layer model provides a unique perspective for the design, operations, and governance of AI Infrastructure:

Holistic Cognitive Framework: Transcend the limitations of single technical metrics to grasp system state as a whole
Dynamic Balance Thinking: Understand unity of opposites relationships and avoid extremes
Evolutionary Stage Awareness: Grasp the system’s development stage and act in accordance with the situation
Flow Insights: Focus on energy flow within the system to anticipate problems

Next, we will delve into the connotation, engineering mapping, and mechanism of each layer.

The Yin-Yang Layer: Dynamic Balance of System States

Jimmy Song — Tue, 10 Feb 2026 13:56:33 +0000

Yin-Yang is originally a fundamental concept in Chinese philosophy, representing two opposing yet interdependent forces present in all things in the universe. Everything in the world can be classified as either Yin or Yang, and their continuous movement and change generate the various transformations we observe. In the context of systems, Yin-Yang represents the unity of opposites through tension—a pair of attributes or tendencies that pull against yet depend on each other.

Three Typical Pairs of Yin-Yang Tensions

In AI infrastructure, we identify three typical pairs of Yin-Yang tensions:

Expansion ↔ Constraint

Expansion ↔ Constraint: The tension between growth trends and limiting forces.

Yang (Expansion): System expansion speed, such as continuously adding tasks and scaling resources
Yin (Constraint): Limiting forces, such as cost controls, regulatory constraints, and hardware limits

System expansion speed and constraint intensity always coexist. For example, continuously adding tasks and scaling resources in GPU clusters (the Yang of expansion) is constrained by costs, regulations, or hardware limits (the Yin of constraint).

Imbalance manifestations:

Pursuing expansion without regard for constraints → Resource contention and crashes
Excessive constraint → Stifling system vitality

Innovation ↔ Governance

Innovation ↔ Governance: The tension between creative capability and control requirements.

Yang (Innovation): Technical innovation, introduction of new features
Yin (Governance): Security reviews, rule-making

The faster technical innovation progresses, the more easily governance gaps are exposed. For example, introducing new Agent features (innovation, Yang) may outpace security reviews and rule-making (governance, Yin), leading to potential risks.

Imbalance manifestations:

Innovation outpaces governance → Potential security risks
Excessively strict governance → Slowing innovation momentum

Speed ↔ Stability

Speed ↔ Stability: The tension between performance advancement and reliable operation.

Yang (Speed): Performance improvements, increased throughput
Yin (Stability): Reliable operation, system stability

When we pursue speed improvements single-mindedly, the cost to stability will eventually manifest. For example, pushing GPU utilization to the limit during model training (speed, Yang) easily leads to more frequent failures or delays (decline in stability, Yin).

Imbalance manifestations:

Extreme pursuit of speed → Decline in stability
Excessive conservatism → Performance waste

The Art of Yin–Yang Balance

The Yin–Yang poles described above are not simple trade-offs where you choose one and sacrifice the other, but rather inherent relationships of unity of opposites in systems. Both Yin and Yang sides are opposed yet complementary, neither can be dispensed with:

Expansion without constraints is difficult to sustain, constraints without expansion lose meaning

As the ancient saying goes, “One Yin and one Yang constitute the Way” (一阴一阳之谓道). Balancing Yin and Yang is the “Way” of healthy system operation. For architects, the key lies in:

Insight into dominant tensions: Determine which pair of tensions is currently dominant
Introducing the opposite: Introduce the complementary side at the right time to restore balance
Dynamic adjustment: Dynamically transform based on changes in system environment and stage

Practical Cases

Case: GPU Cluster Expansion

When the cluster is in a state of rapid expansion (Yang exuberant, Yin deficient):

✓ Add scheduling policies and resource quotas (supplement Yin)
✓ Establish cost control mechanisms (supplement Yin)
✗ Do not pursue expansion speed single-mindedly

Case: Agent Feature Innovation

When introducing new Agent features:

✓ Simultaneously establish monitoring and sandboxing mechanisms (supplement Yin)
✓ Improve security review processes (supplement Yin)
✗ Do not let innovation outpace governance

Case: Model Training Performance Optimization

When optimizing model training performance:

✓ Simultaneously strengthen fault tolerance mechanisms and testing (supplement the Yin of stability)
✓ Set performance baselines and rollback mechanisms (supplement Yin)
✗ Do not infinitely compress fault tolerance time

Dynamic Transformation of Yin–Yang States

It’s important to note that Yin–Yang states are not static and unchanging, but dynamically transform with system environment and stage.

The same capability may transform from an advantage to a risk at different stages

For example, a “rapid development” strategy that drives rapid iteration during the startup stage, if applied without restraint during the scaling stage, can instead become a major threat to stability.

The analysis of the Yin–Yang layer reminds us to constantly pay attention to the ebb and flow of these opposing forces, and to keep the system in a state of elastic tension through adjustments, rather than snapping or becoming slack and ineffective.

Five Elements Layer: Classification and Collaboration of System Roles

Jimmy Song — Tue, 10 Feb 2026 13:56:06 +0000

Five Elements (Wǔxíng, Five Elements or Five Phases) theory divides everything in the world into five basic elements: Wood, Fire, Earth, Metal, Water. Each element represents a fundamental attribute or functional role, with the five elements generating and overcoming each other in an endless cycle.

In AI infrastructure, we use “Five Elements” to characterize the system’s five core elements and their responsibilities:

Engineering Mapping of Five Elements

Five Elements	Symbol	Meaning	Engineering Correspondence
Water	🌊	Flow and containment	Data flow and quality: data pipelines, data assets, and quality control
Wood	🌲	Growth and creation	Model growth and capability expansion: model architecture iteration, parameter scale expansion
Fire	🔥	Energy and execution	Compute conversion and work efficiency: GPU/TPU computing, job scheduling efficiency
Earth	🏔️	Support and stability	Platform support and orchestration governance: distributed coordination, middleware, scheduling systems
Metal	⚙️	Strength and standardization	Hardware constraints and physical boundaries: GPU/CPU performance, storage capacity, network bandwidth

Table 5: Engineering Mapping of Five Elements

Water – Data Flow and Quality

Corresponds to data pipelines, data assets, and quality control in the system.

Water symbolizes flow and containment, analogous to the circulation and nourishing role of data in the system, including:

Training data acquisition
Real-time data input
Feedback signal transmission
Data cleaning and quality assurance

Wood – Model Growth and Capability Expansion

Corresponds to the evolution and growth of machine learning models and algorithms.

Wood represents growth and creation, mapped to:

Model architecture iteration
Parameter scale expansion
Cultivation of new capabilities
Algorithm optimization and improvement

Fire – Compute Conversion and Work Efficiency

Corresponds to computing processes and the utilization of compute resources.

Fire symbolizes energy and execution, reflected as:

Using GPU/TPU and other compute resources for calculation
Converting electrical energy into model training and inference work
Parallel computing capability
Job scheduling efficiency

Earth – Platform Support and Orchestration Governance

Corresponds to the support and governance capabilities of the platform layer.

Earth represents support and stability, analogous to:

Infrastructure platform support for upper-layer applications
Distributed system coordination and orchestration
Middleware services
Scheduling systems and policy management
Permission systems, service quality assurance

Metal – Hardware Constraints and Physical Boundaries

Corresponds to underlying hardware and system hard limits.

Metal represents strength and standardization, mapped to:

GPU/CPU hardware performance
Storage capacity
Network bandwidth
Physical conditions and hard rules (power consumption, safety specifications, etc.)

Five Elements Generation Relationships

The Five Elements form a positive cycle through “generation” relationships:

Data (Water) spawns model growth (Wood), model requirements stimulate compute investment (Fire), compute development drives platform thickening (Earth), platform capabilities utilize hardware to push the boundaries of (Metal), and hardware progress in turn supports greater data acquisition (Water)

Figure 7: Five Elements generation relationship diagram. Water generates Wood, Wood generates Fire, Fire generates Earth, Earth generates Metal, Metal generates Water, representing the mutually reinforcing cycle between data, models, compute, platforms, and hardware.

Five Elements Overcoming Relationships

At the same time, overcoming relationships also exist among the Five Elements, meaning when one element is too strong or imbalanced, it will suppress or weaken another element:

Wood overcomes Earth: Excessive model expansion increases the burden on the platform (Earth), potentially even crushing the existing architecture
Earth overcomes Water: Overly heavy platforms and rules will hinder the free flow of data (Water)
Water overcomes Fire: Data bottlenecks will limit the performance of compute
Fire overcomes Metal: Excessive compute demand may break through hardware (Metal) limits
Metal overcomes Wood: Strict hardware and rule limitations will curb the expansion of models (Wood)

Figure 8: Five Elements generation and overcoming relationship diagram. Dashed arrows indicate overcoming relationships, reflecting the system’s internal checks and balances mechanism: any element becoming excessively strong will constrain another element.

Figure 9: Five Elements generation and overcoming relationship diagram. Dashed arrows indicate overcoming relationships, reflecting the system’s internal checks and balances mechanism: any element becoming excessively strong will constrain another element.

Figure 3: Five Elements generation and overcoming relationship diagram. Dashed arrows indicate overcoming relationships, reflecting the system’s internal checks and balances mechanism: any element becoming excessively strong will constrain another element.

Five Elements Balance Diagnosis

Through the Five Elements model, engineering teams can systematically check the role completeness and balance of infrastructure.

Common Imbalance Patterns

Imbalance Pattern	Manifestation	Consequence	Solution
Strong Wood, Weak Water	Focus on model algorithm iteration, neglect data quality	Model performance hits bottlenecks	Strengthen data pipelines and quality control
Strong Metal, Weak Earth	Stack hardware, insufficient platform governance capability	Poor resource utilization, lack of vitality	Improve platform governance and scheduling
Vigorous Fire, Broken Wood	Large compute investment, models can’t keep up	Resource waste	Optimize model architecture, improve compute utilization efficiency

Table 6: Common Imbalance Patterns

Balance Principles

Successful large-scale systems require coordinated cooperation of all five elements

Let each of the five elements fulfill its duties in their respective roles
Maintain generation as primary, overcoming as secondary
Prevent any side from excessive expansion or shrinkage
Regularly check the balance state of Five Elements

Only by letting the five elements fulfill their respective roles and mutually promote each other, while preventing any side from excessive expansion or shrinkage, can the entire system maintain robustness and evolutionary capability.

The Yun Layer: Stages and Cycles of System Evolution

Jimmy Song — Tue, 10 Feb 2026 13:56:38 +0000

Yun (运) here refers to the developmental stages and temporal rhythms experienced by a system, which can be understood as the lifecycle cycles or “fortune” of infrastructure.

Large-scale infrastructure is not static but evolves cyclically through the Exploration Period, Platform Period, Scale Period, and Rebalancing Period, with each stage having its primary contradictions and tasks.

Below are the four evolutionary stages.

Exploration Period (Initial Stage)

Characteristics: High variance, low structure, rapid trial and error

At this stage, new technologies and requirements emerge constantly, system architecture is loose, and diverse experiments coexist.

Primary Tasks:

Explore effective paths
Rapidly validate model and functional directions
Collect data and preliminary stability signals

Five Elements Characteristics: Wood and Fire in Command

Model innovation (Wood) and computing experimentation (Fire) are core drivers
Expansion (Yang) outweighs constraints (Yin)

Architecture Strategy:

✓ Tolerate some chaos
✓ Encourage innovation and iteration
✓ Focus on collecting data and preliminary stability signals
✗ Don’t prematurely introduce heavy processes and restrictions

Platform Period (Growth Stage)

Characteristics: Standardization emerges, interfaces and processes converge

After exploration, the system enters a stage of integration and regulation, beginning to establish unified platforms, standard interfaces, and governance processes, consolidating scattered results into platform capabilities.

Primary Tasks:

Establish unified platforms
Define standard interfaces
Consolidate governance processes

Five Elements Characteristics: Fire Generates Earth

Successful practices in computing and functionality (Fire) give rise to platform support requirements (Earth)
Governance and standards gradually strengthen

Architecture Strategy:

✓ Extract common requirements
✓ Build support platforms (Yin increases)
✓ Lay the foundation for next-stage scaling
✗ Don’t remain in disordered exploration

Scale Period (Mature Stage)

Characteristics: Efficiency, throughput, and cost become the main battlefield

The system is deployed at scale, and focus shifts to optimizing efficiency and costs, improving throughput and reliability.

Primary Tasks:

Optimize efficiency
Improve throughput
Reduce costs
Ensure reliability

Five Elements Characteristics: Heavy Earth Breaks Wood

Platforms (Earth) and hard constraints begin to dominate
Overly idealistic model expansion (Wood) will encounter setbacks from realistic conditions

Architecture Strategy:

✓ Strengthen monitoring and automated operations
✓ Control overly strong “Yang” through governance means
✓ Ensure robust system operation
✗ Don’t continue with startup-era casual practices

Rebalancing/Substitution Period (Renewal Stage)

Characteristics: Old structures are corrected or replaced by new structures

When the previous stage’s patterns reach their limits, the system either enters self-correction by introducing new elements to rebalance, or gets disrupted and replaced by a new paradigm.

Primary Tasks:

Introduce new elements to rebalance
Or accept substitution by a new paradigm

Five Elements Characteristics: Metal and Water Rise Again

Suppressed hardware/rule innovations (Metal) and new data potentials (Water) rise again
Driving system transformation

Architecture Strategy:

✓ Be forward-looking, dare to break through
✓ Transition smoothly, avoid severe volatility
✗ Don’t cling to the status quo

Evolutionary Cycle

The above stages form a cyclical pattern, where the endpoint of each stage is also the starting point of the next ↻.

Figure 3: The “Yun” cycle of AI infrastructure evolution. Systems start from the exploration period, undergo platform period standardization, enter the scale period for efficiency optimization, and ultimately move toward a new cycle of rebalancing or substitution.*

The Art of Following the Momentum

A mature infrastructure organization should be able to determine its current stage based on internal and external signals and adjust its strategy accordingly.

If stage transitions are ignored or excessively rushed, the system will experience disturbances or even crises

Error Examples

Erroneous Behavior	Manifestation	Consequence
Pulling Up Seedlings to Help Them Grow	Managing systems still in exploration period as scaled systems, prematurely suppressing change	Stifling innovation
Going Against the Momentum	Remaining in disordered exploration when it’s time to enter the platform period	Missing the window for structured growth and creating hidden risks
Clinging to the Status Quo	Unwilling to change when rebalancing period is needed	System rigidity and aging

Table 3: Development Stage Characteristics

Stage Assessment Checklist

Through the “Yun” layer perspective, teams can examine the current macro stage:

Are we validating new concepts or expanding our achievements?
What is the system’s primary contradiction?
When might the next stage arrive?
Does our strategy align with the current stage?

Example Questions:

Are we in the exploration period?
- If yes → Focus on rapid trial and error and validation
- If no → Consider whether to enter the platform period
Does our system need standardization?
- If yes → Enter platform period, establish platforms and standards
- If no → Continue exploration

Qi Layer: Effective System Flow and Pressure Fields

Jimmy Song — Tue, 10 Feb 2026 13:56:17 +0000

Qi (气) in Chinese culture refers to the energy and flow field that permeates all things. In AI infrastructure, we borrow the concept of “Qi” to describe the effective flow and pressure distribution within systems.

This includes the circulation of data, tasks, and signals throughout the system, as well as how various explicit or implicit system pressures accumulate, propagate, and release.

The Essence of Qi: Overall State of Affairs

Unlike traditional single-point metric monitoring, the concept of “Qi” reminds us to focus on the overall state of affairs:

Signals are not isolated events, but rather gather and flow like a field

For example:

A sudden spike in GPU utilization may not be abnormal
But if multiple metrics (job queue length, response latency, memory usage, etc.) show a simultaneous trend of increase and persistence → this indicates a change in the “Qi field”
This signals the system entering a high-pressure state

This signal field manifests as the gathering and stretching of Qi, indicating the accumulation of some form of system tension.

Two States of Qi

Qi Flow: System Active

When all elements coordinate well, data and instructions flow smoothly, producing value efficiently:

Processing rates across all stages are basically matched
No long-term backlogs or idle resources
Timely system responses
Balanced resource utilization

Qi Stagnation: System Pathological

If a bottleneck or imbalance occurs somewhere, Qi’s flow is obstructed, causing local pressure to surge:

Jobs queue for long periods
CPU/GPU long-term idle or 100% utilization
Serious message queue backlog
Frequent anomaly alerts

Ultimately, this may trigger failures or performance collapse at weak points.

Qi’s Flow Path

To intuitively understand Qi’s flow path, we can view the system as a closely connected network:

Figure 3: Diagram of system ‘Qi’ flow path. Data (Water) Qi enters Model (Wood), triggering Computing Power (Fire) operation, coordinated via Platform (Earth), executed on Hardware (Metal), producing results that feed back to the data layer, forming a closed loop.

Qi’s Cycle:

Data (Water) Qi enters Model (Wood)
Drives Computing Power (Fire) to operate
Coordinated via Platform (Earth)
Executes computation on Hardware (Metal)
Outputs results, producing new data or signals
Feeds back into the data pool (Water)
Cycle repeats

Two Forms of Qi

Healthy Flow

Qi circulates ceaselessly among the five elements, maintaining system functionality:

If every step flows smoothly → system operates smoothly
If any step is obstructed → Qi flow slows or even reverses, damaging system performance and stability

Pressure Propagation

Qi refers not only to healthy flow, but also to pressure propagation:

Example: Data Inflow Surge

Data inflow surges but model processing capacity cannot keep up
Unprocessed data continuously accumulates
Manifests as excessive pressure in the data layer (Water)
Leading to suppression of computing power performance (Fire weakens)

Example: Hardware Resource Exhaustion

Hardware (Metal) resources exhausted
Computing requests cannot be satisfied
Obstructed Qi transforms into queuing pressure
Feeds back to platform (Earth) scheduling layer and user experience

Application of Qi Layer in Operations

Through the lens of “Qi”, operations and architecture teams can more sensitively detect sub-optimal system states:

Qi State	Manifestation	Warning Significance
Stagnation Emerging	Latency jitter gradually worsening	System entering sub-stable state, needs 疏导
Flow Obstruction	Request failure rate rising, retries increasing	某环节阻塞，needs investigation
Qi Scattering	Metrics fluctuating severely, irregular	System severely imbalanced, needs overall adjustment
Qi Deficiency	Resource utilization long-term low	Configuration unreasonable, needs optimization

Table 5: Qi State and Warning Significance

Qi Disorder Precedes Major Incidents

Latency jitter gradually worsening → signals system entering sub-stable state
If no measures are taken to resolve (scaling resources, optimizing algorithms, or rate limiting) → may evolve to complete failure
Agent task interaction rhythm (Qi) slows or stops → may indicate poor communication between agents or deadlock

Strategies for Guiding Qi Flow

Maintaining smooth Qi flow requires building resilience:

Architecture Level

Peak shaving and valley filling mechanisms: Absorb 突发流量
Message queue backpressure protection: Prevent pressure backflow
Elastic buffer design: Reserve margin to handle impacts

Strategy Level

Slack capacity: Maintain certain redundancy
Elastic scaling strategies: Dynamically adjust resources
Rate limiting and degradation mechanisms: Protect core functionality

Agent System Special Attention

Monitor task queues and communication latency
Ensure information flow (Qi) between agents is unobstructed
Introduce coordinator agents or reduce concurrency when necessary to smooth Qi flow

Qi Layer Monitoring Practices

Establish system-wide observability:

Monitoring Dimension	Focus	Tool Examples
Traffic Distribution	Request flow across stages	Distributed Tracing
Queue Backlog	Queue length trends	Message Queue Monitoring
Resource Utilization	CPU/GPU/Memory/Storage	Prometheus + Grafana
Latency Distribution	P50/P95/P99 latency	APM Tools
Anomaly Trends	Error rate, retry rate changes	Log Aggregation Analysis

Table 6: Qi Layer Monitoring Dimensions

The Qi layer provides an effective liquidity metric, helping us pulse-check whether the system’s “blood and Qi” are abundant and flowing smoothly

Summary

Qi’s operation can be understood as whether the system’s “meridians” are unobstructed:

Qi flow means system active: Data and instructions flow smoothly, producing value efficiently
Qi stagnation means system pathological: Flow obstructed, local pressure surges, ultimately triggering failures

Just as in Traditional Chinese Medicine’s four examination methods, by observing “Qi’s” operation, we can predict the trajectory of system problems and apply targeted remedies.

Dynamic Relationship Modeling: Five Elements Flow Under Yin-Yang Balance

Jimmy Song — Tue, 10 Feb 2026 13:55:47 +0000

Yin-Yang × Five Elements: Intrinsic Tension of Elements

Each Five Elements component contains both Yin and Yang aspects, manifesting with different polarities in different contexts:

Figure 3: Yin-Yang states of Five Elements. Each element includes Yin (potential, static, introverted) and Yang (explicit, dynamic, extroverted) aspects, with transformation possible between them depending on context.

Yin-Yang Attributes of the Five Elements:

Five Elements	Yin State	Yang State
Water (Data)	Potential data reserves, implicit patterns (static storage of historical data)	Instant data flow, real-time feedback
Wood (Model)	Dormant capabilities (unactivated parameters, backup algorithms)	Explicit expansion (model architecture updates, parameter surge)
Fire (Compute)	Stored energy (idle compute, waiting for scheduling)	High-load operation
Earth (Platform)	Static support (stable operation, non-intervention)	Proactive scheduling and expanded governance
Metal (Hardware)	Implicit constraints (unused capacity)	Explicit limits (resource hard caps maxed out)

Table 7: Dynamic Model Overview

Signs of Yin-Yang Imbalance:

Fire Excessively Yin: GPU compute idle for long periods while tasks backlog → Poor scheduling
Fire Excessively Yang: GPUs at 24-hour full load with no elasticity → Hidden crash risk
Earth Excessively Yang: Too many platform rules → Stifling innovation
Earth Excessively Yin: Lack of platform control → Leading to chaos

Five Elements × Qi: Dynamic Network of Flow

The Five Elements framework provides tools to decompose systems, but system components are not static puzzles—rather, they connect into a dynamic network through the flow of Qi.

Generating Relationships: Qi flows smoothly, forming positive feedback loops
Controlling Relationships: Qi stagnates at certain links or reverse effects strengthen

Dynamic Relationship Principles:

Generating primarily, Controlling secondarily—main energy flows transmit successfully through each link, while balancing forces intervene moderately only to prevent extreme situations.

Yun × Yin-Yang Five Elements: Boundary Conditions for Stage Evolution

The stage-based nature of Yun provides a perspective of boundary conditions evolving over time for the aforementioned Yin-Yang Five Elements dynamics.

Each stage strengthens or weakens certain elements and tensions:

Stage	Main Characteristics	Five Elements Characteristics	Yin-Yang Characteristics
Exploration Stage	High variance, low structure, rapid trial and error	Wood and Fire dominant	Expansion (Yang) outweighs Constraints (Yin)
Platform Stage	Standardization emerges, interfaces and processes converge	Fire generates Earth	Governance (Yin increasing) gradually strengthens
Scale Stage	Efficiency, throughput, cost become main battlegrounds	Earth dominates Wood	Stability (Yin) takes precedence
Rebalancing Stage	Old structures corrected or replaced by new structures	Metal and Water resurge	Transformation (Yang) rises again

Table 8: Typical Interaction Scenarios

Dynamic Stage Transitions:

The Yun layer tells us when to shift focus:

As stages change, the system needs to “allocate interests”
Previously dominant elements may become excessive and need convergence
Previously minor elements need strengthening to address shortcomings

Examples:

In Platform Stage/Scale Stage → Must strengthen governance (Earth’s Yang) and hardware optimization (Metal’s Yang)
To curb the 野蛮 growth tendencies left over from early stages (excessive Wood-Fire Qi)
In Rebalancing Stage → May need to reactivate suppressed innovation potential (Water-Wood Qi)

Comprehensive Analysis Case: GPU Scheduling Scenario

Let’s see how to apply the four-layer model to analyze a real GPU scheduling problem.

Problem Scenario: Cluster experiences task queues under high load

Layer	Diagnosis	Findings
Qi Layer	Observe Qi flow state	Compute Fire Qi is obstructed
Five Elements Layer	Locate elements	Data input too intense (Water Yang excessive) but scheduling (Platform Earth) strategy cannot keep up
Yin-Yang Layer	Analyze tensions	Scheduling strategy blindly pursues maximizing utilization (excessively Yang) while lacking elastic buffers (Yin)
Yun Layer	Assess stage	This is an emerging business that just passed exploration stage and hasn’t perfected scheduling—Platform Stage

Table 9: Four-Layer Diagnostic Analysis

Solutions

Based on four-layer collaborative diagnosis, develop comprehensive solutions:

Qi Layer: Unblock Qi flow
- Expand resources or optimize algorithms
Five Elements Layer: Balance elements
- Strengthen platform scheduling capabilities (Earth)
Yin-Yang Layer: Restore balance
- Introduce elastic buffer mechanisms (supplement Yin)
- Avoid blindly pursuing high utilization
Yun Layer: Follow the trend
- Accelerate introduction of standardized scheduling and resource governance (Earth’s Yun is approaching)

Value of Dynamic Modeling

Through the multi-level dynamic modeling above, we can:

Explain complex scenarios more comprehensively: No longer limited to single perspectives
Locate root causes of problems: Find fundamental causes rather than surface phenomena
Point improvement directions: Obtain systematic solutions
Predict system evolution: Prepare in advance for stage transitions

Practical Recommendations

In daily architecture design and operations, you can establish these thinking habits:

When encountering problems: Analyze layer by layer from a four-layer perspective
When making decisions: Consider impacts on all four layers
When conducting post-mortems: Check whether warning signals from the four-layer model were ignored

The value of a system lies not in pursuing the extreme of a single performance indicator without limit, but in balancing all elements to achieve long-term coordinated development

Engineering Practice Guide: Architecture Decisions Guided by Theory

Jimmy Song — Tue, 10 Feb 2026 13:55:59 +0000

The theoretical models mentioned above are not 停留在停留在 the conceptual level, but directly provide guidance for the engineering practice of AI infrastructure. In specific scenarios such as GPU scheduling, Agent runtime, and platform governance, we can follow the principles below to apply the Yin-Yang Five Elements Qi Movement model.

Balance Yin and Yang, Avoid Extremes

Consider both propelling forces and restraining forces when making architecture decisions.

GPU Cluster Scaling:

✓ Satisfy business growth (expanding Yang)
✓ Set quota and priority policies (constraining Yin)
✓ Prevent resource abuse

Agent Runtime Design:

✓ Give agents more autonomy (innovation, Yang)
✓ Introduce monitoring and sandboxing mechanisms (governance, Yin)
✓ Prevent loss of control

Practice Checklist:

After every major adjustment, ask yourself: Have I introduced corresponding counter-forces to stabilize the system?

Complete the Five Elements, Identify and Fill Weaknesses

Regularly review whether the five types of elements in the system are balanced.

GPU Infrastructure Check:

Do data pipelines keep up with computing power improvements? (Water and Fire matching)
Does model optimization fully utilize hardware? (Wood and Metal matching)
Can the scheduling platform handle peak loads? (Earth supporting Fire)
Has hardware resources become a bottleneck? (Metal not holding back)

Agent Platform Check:

Is there high-quality knowledge base or real-time data support? (Water)
Is there strong model capability? (Wood)
Is there sufficient computing resources? (Fire)
Is there a good orchestration framework? (Earth)
Is there a reliable environment and interfaces? (Metal)

Practice Strategy:

Once a bottleneck or overload is discovered in a certain link, decisively invest resources to fill the weakness or reduce the burden on the overloaded part

Problem Discovered	Solution
Insufficient data quality (“Water” weak)	Prioritize data governance
Long-term low hardware utilization (Metal strong, Fire weak)	Optimize algorithms or scheduling to better utilize hardware

Table 1: Problem Discovery and Solutions

Follow the Trend, Align with the Movement

Develop reasonable strategies based on the stage of the system.

Strategies for Different Stages:

Stage	Should Do	Should Not Do
Exploration Phase	Rapid trial and error, validate value	Prematurely introduce heavy processes and constraints
Platform Phase	Standardized management, MLOps tools	Remain in disordered exploration
Scale Phase	Strengthen governance and efficiency optimization	Still use the casual practices of the startup period
Rebalancing Phase	Architecture innovation, introduce new technologies	Refuse to move forward

Table 2: Strategies for Different Stages

Regular Assessment: At each quarter or important milestone, assess:

Which stage are we currently in?
What is the main contradiction in this stage?
When might the next stage arrive?
Prepare in advance for the transition

Practice Cases:

An AI training cluster after validating the concept → Should consider entering standardized management (transitioning from exploration phase to platform phase)
When system scale expansion encounters bottlenecks → Consider whether to enter the rebalancing phase and break through through architecture innovation

Observe Qi Field, Optimize Flow

Establish global observability of the system, focusing on trends and correlations rather than single-point metrics.

Monitoring Methods:

Distributed tracing
Metric correlation analysis
Full-link monitoring

Signals of Qi Disorder:

Signal	Possible Cause
Frequent occurrence of various abnormal logs	Global investigation needed
A metric’s periodic fluctuations becoming increasingly intense	The system may be approaching a limit internally

Table 3: Signals of Qi Disorder

Strategies to Keep Qi Flowing Smoothly:

Architecture Level:

Peak clipping and valley filling mechanisms
Message queue backpressure protection

Strategy Level:

Slack capacity
Elastic scaling strategies

Agent System Special Attention:

Monitor task queues and communication latency
Ensure smooth information flow (Qi) between agents
Introduce coordinator agents or reduce concurrency when necessary

Dynamic Adjustment, Continuous Rebalancing

Integrate the Yin-Yang Five Elements Qi Movement model into the team’s continuous improvement process.

Core Questions in Architecture Reviews or Incident Retrospectives:

Is the current main contradiction more inclined toward expansion or constraint, speed or stability?
Is any Five Elements element overloaded (Yang excess) or missing (Yin deficiency)?
Is System Qi congested somewhere?
Do our strategies align with the current stage?

Continuous Improvement Process:

Problem Discovery → Four-Layer Model Diagnosis → Strategy Formulation → Implementation Adjustment → Effect Evaluation → Continuous Optimization

Practice Case: Large-Scale GPU Training Cluster Optimization

Background: A team encountered stability issues while operating a large-scale GPU training cluster.

Four-Layer Model Diagnosis:

Layer	Diagnosis	Findings
Yin-Yang Layer	Speed vs Stability	Continuously compressing fault tolerance and testing time in pursuit of efficiency (speed Yang), leading to frequent online failures (stability Yin damaged)
Five Elements Layer	Five Elements Check	Data pipeline latency gradually increasing (Water weaker than Fire)
Movement Layer	Stage Judgment	System has moved from barbaric growth period to maturity period
Qi Layer	Qi Flow State	Qi stagnation phenomenon obvious

Table 4: Monitoring Methods

Comprehensive Solution:

Yin-Yang Balance:
- Suspend performance optimization
- Invest time to strengthen fault tolerance mechanisms and testing (supplement stability Yin)
Five Elements Completion:
- Add data preprocessing nodes and caching (strengthen Water)
Movement Adjustment:
- Change mindset, shift focus from feature expansion to optimization and governance
Qi Flow Regulation:
- Build full-link tracing system
- Monitor the time of each link from training job submission to completion
- Identify Qi stagnation points and clear them

Result: While maintaining high utilization, the cluster’s stability was greatly improved, and no serious downtime occurred again.

Scenario Application Quick Reference Table

Scenario	Yin-Yang Focus	Five Elements Check	Movement Judgment	Qi Flow Monitoring
GPU Scheduling	Utilization vs Elasticity	Fire - Earth - Metal Balance	Scale Phase Efficiency Optimization	Task queues, resource utilization curves
Agent Runtime	Autonomy vs Governance	Water - Wood - Fire Coordination	Exploration Phase Rapid Iteration	Communication latency, task interaction rhythm
Platform Governance	Innovation Risk Control vs Process Efficiency	Earth - Metal Constraints	Platform Phase Standardization	Rule execution rate, change frequency
Cost Optimization	Performance vs Cost	Fire - Metal Matching	Scale Phase Refinement	Resource waste, idle time

Table 5: Signals of Qi Disorder

Summary

Through the Yin-Yang Five Elements Qi Movement model, we can in practice:

Avoid Extremes: Not blindly pursuing single metrics
Systematic Thinking: Analyzing problems from multiple dimensions
Follow the Trend: Adjust strategies based on stages
Predict Problems: Early warning of risks through Qi field changes
Continuous Improvement: Establish systematic optimization processes

The value of this system lies in: combining Eastern wisdom with engineering practice to provide a unique and effective thinking framework for complex AI infrastructure

System Diagnosis Principles: Criteria for Health Status

Jimmy Song — Tue, 10 Feb 2026 13:56:28 +0000

To maintain the long-term healthy evolution of AI infrastructure, post-mortem summaries are far from sufficient. We need a set of system diagnosis principles to detect hidden risks early and correct deviations.

Based on the Yin-Yang Five Elements Yun model, diagnosis can be conducted from the following five dimensions:

Five-Dimensional Diagnosis Framework

Figure 5: Five-Dimensional Diagnosis Framework Diagram

Five Elements Balance Check

Assess the current status of five aspects: Data (Water), Models (Wood), Compute (Fire), Platform (Earth), and Hardware (Metal).

Diagnosis Method

Checklist:

Can data pipelines keep up with demands? (Water)
Are model capabilities fully utilized? (Wood)
Are compute resources effectively used? (Fire)
Can the platform support current load? (Earth)
Is hardware becoming a bottleneck? (Metal)

Identify Problems

Problem Type	Manifestation	Solution
Short Board	One element significantly weaker than others	Prioritize strengthening that element
Overload	One element consumes excessive resources or frequently becomes a bottleneck	Introduce limits or expand other elements to share pressure

Table 19: Problem Types and Solutions

Typical Symptoms

Water Level Too Low: Data pipelines always lag behind training needs → Replenish data processing capacity
Metal Overload: Hardware often runs at full capacity or even triggers limit alarms → Expand capacity or impose constraints on upper layers

Most failures do not stem from missing components, but from long-term role imbalance

Qi Flow Smoothness Check

Analyze whether Qi flows smoothly through the system via full-link monitoring.

Diagnosis Method

Key Metrics:

Latency distribution of key processes
Queue backlogs
Resource utilization curves

Qi Smooth vs. Qi Not Smooth

State	Characteristics
Qi Smooth	Processing rates across stages basically match, without long-term backlogs or idle resources
Qi Not Smooth	One stage remains a bottleneck for long periods, or large amounts of resources sit idle

Table 20: Qi Flow: Smooth vs Obstructed

Diagnosis Points

Distinguish temporary fluctuations from persistent trends: brief peaks don’t necessarily indicate Qi blockage, but persistent deviations must be addressed

Tool Support:

Dashboards and automated alerts
Timely capture of “stagnant Qi” locations
Further investigation of causes (which Five Elements imbalance corresponds)

Yin-Yang Dynamics Check

Assess whether current strategy and state are Yang Excess Yin Deficiency or Yin Excess Yang Deficiency.

Diagnosis Method

Qualitative Analysis:

Look at whether recent architecture decisions overly favor one extreme
Have you been continuously expanding and adding new features while ignoring stability?
Or conversely, multiple layers of approval and strict constraints but lack innovation momentum?

Quantitative Metrics:

Metric	Yang Excess	Yin Excess
Change Frequency	Extremely high	Extremely low
Incident Rate	Frequent	Extremely low but no change
Release Rhythm	Continuous	Long-term stagnation

Table 21: Yin-Yang Status

Balance Strategy

State	Symptoms	Solution
Yang Excess Yin Deficiency	Frequent changes with frequent incidents	Pause releases, focus on addressing hazards (replenish Yin)
Yin Excess Yang Deficiency	Long-term no change and stagnation	Introduce challenges and innovation (add Yang)

Table 22: Balance Strategies

Yun Alignment Check

Determine whether the organization’s actions match the system’s current stage, preventing counter-Yun operation.

Diagnosis Method

Combine Business Development and Technical Maturity:

Error Pattern	Manifestation	Consequences
Premature Standardization	Spending 大量精力 on process management and cost optimization for emerging projects	These are typically scale stage concerns, but the project is still in exploration stage
Counter-Yun Exploration	Frequently changing underlying architecture for widely used platforms without rigorous testing	Inconsistent with scaling stage

Table 23: Error Patterns

Stage-Strategy Reference Table

Stage	Should Focus On	Should Not Do
Exploration Stage	Diversity, flexibility, rapid trial and error	Premature pursuit of efficiency
Platform Stage	Standardization, process norms	Frequent arbitrary changes
Scale Stage	Optimization, stability, efficiency	Still growing wildly
Rebalancing Stage	Transformation, breakthrough, innovation	Clinging to the past

Table 24: Stage-Strategy Mapping

Checklist:

Which stage are we currently in?
Do our actions match the stage?
Do we need to adjust strategy?

When discovering actions don’t match the stage, immediately adjust strategy to avoid working at cross-purposes

Yang Runaway Warning

Pay special attention to whether there are signs of Yang state runaway in the system.

What is Yang Runaway?

Exponential explosion or collapse risk caused by unconstrained positive feedback.

Typical Scenarios

Scenario	Mechanism	Risk
Service Call Volume Surge	Bug or abuse → Resource strain → Queuing and retry storms → Further increase in calls	Resource exhaustion
Training Task Self-Replication	Tasks unlimitedly self-replicate to accelerate → Cluster resource exhaustion	System collapse

Table 25: Typical Scenarios

Diagnosis Signals

A metric shows exponential explosive growth
Lack of slowing mechanisms
Formation of vicious cycles

Response Strategy

Strategy	Means	Effect
Establish Hard Limits	Metal’s constraints	Immediate shutdown
Introduce Negative Feedback	Earth’s governance (rate limiting, quotas)	Braking and deceleration
Break Positive Feedback Chain	Activate emergency plan	Pull back to steady state

Table 26: Response Strategies

When discovering a metric showing exponential explosive growth without slowing mechanisms, intervene immediately

Diagnosis Implementation Process

Regular Diagnosis Mechanism

Recommend establishing a periodic diagnosis process:

Figure 6: Regular Diagnosis Mechanism Flowchart

Diagnosis Meeting Agenda

Fixed Session of Weekly Operations Review Meeting:

Check Five Elements scores for each module
Browse global Qi flow diagram
Analyze Yin-Yang dynamics
Discuss current Yun

This systematic examination makes hidden risks 无处遁形，thus achieving prevention before problems occur

Diagnosis Action Matrix

Diagnosis Result	Action Recommendation
Five Elements: One Element Too Weak	Concentrate resources to strengthen the weakness
Five Elements: One Element Overloaded	Expand capacity or introduce constraints
Qi Stagnation at One Stage	Clear bottlenecks, optimize processes
Yang Excess Yin Deficiency	Strengthen governance and stability mechanisms
Yin Excess Yang Deficiency	Activate innovation and boost vitality
Counter-Yun Operation	Adjust strategy and go with the flow
Yang Runaway Warning	Immediate intervention, break positive feedback

Table 27: Diagnosis Action Matrix

Summary

Through the above diagnosis principles, architects and operations teams can periodically take the pulse of infrastructure like TCM pulse diagnosis.

When diagnosis indicates imbalance in some aspect, immediately prescribe remedy based on the theory: replenish what needs replenishing, purge what needs purging.

Long-term adherence will keep the system on a healthy evolutionary trajectory.

Conclusion and Outlook

Jimmy Song — Tue, 10 Feb 2026 13:56:22 +0000

This paper systematically presents the four-layer model of “Yin-Yang - Five Elements - Yun - Qi” for AI infrastructure, providing a comprehensive cognitive map from theory to practice.

Review of Theoretical Model

Through four dimensions, we have constructed a global framework for understanding AI infrastructure:

Layer	Core Value	Key Insights
Yin-Yang	Understanding the tension and balance within systems	Expansion and constraint, innovation and governance, speed and stability—these three are opposites yet unified, all indispensable
Five Elements	Organizing the fundamental role elements of systems	Data, models, computing power, platforms, hardware—these five generate and restrain each other in endless cycles
Yun	Grasping the periodic patterns of system evolution	Exploration phase, platform phase, scale phase, rebalancing phase—act in accordance with the trends
Qi	Insight into the flow state of system operation	When Qi flows, the system is active; when Qi stagnates, the system becomes pathological

Table 1: Layer, Core Value, and Key Insights

More importantly, we have demonstrated how this theory combining Eastern wisdom with engineering practice can provide insights and guidance for real-world problems such as GPU scheduling, Agent Runtime, and platform governance.

Core Value of the Model

Holistic View

Traditional fragmented perspectives often see trees but not the forest, making it difficult to provide timely warnings of systemic risks

The Yin-Yang Five Elements Qi-Yun model, with its holistic view, helps architects:

Break free from the constraints of pure technical metrics
Grasp the principal contradictions and driving forces of system evolution
Extract meaningful patterns from complex signals

Dynamic View

The value of a system lies not in pursuing the extreme of a single performance indicator without limit, but in balancing all elements to achieve long-term coordinated development

The model’s dynamic view reminds us:

Yin-Yang dynamics transform dynamically with environment and stage
The same capability may shift from advantage to risk at different stages
Strategies need timely adjustment as Yun changes

Balance View

The core philosophy of the model is balance rather than extreme:

Not pursuing the limit of a single metric
But pursuing system coordination and sustainability
Finding dynamic balance points within unity of opposites

Practical Application Value

During Architecture Design

Consider the completeness and balance of the Five Elements
Reserve Yin-Yang constraint mechanisms
Design evolution paths that align with Yun trends
Plan channels for Qi flow

During Operations and Governance

Regularly check Five Elements balance
Monitor Qi circulation status
Assess Yin-Yang dynamic changes
Determine Yun phase transitions
Provide early warning of Yang loss-of-control risks

During Decision Review

Analyze root causes from the four-layer model perspective
Check whether basic principles of any layer were violated
Develop systematic solutions
Establish long-term improvement mechanisms

Insights for Architects

In an era of flourishing large models and autonomous agents, infrastructure has become unprecedentedly complex and active.

Cognitive Upgrade

From “managing machines and applications” to “managing intelligence and knowledge”:

Not only focus on application logic itself
But more on how knowledge and intelligence integrate into systems
View models as dynamically evolving components

Mindset Shift

From single-metric optimization to system balance:

Not pursuing the extreme of a single element
But pursuing overall coordination and sustainability
Finding dynamic balance within unity of opposites

Capability Development

From technical expert to systems philosopher:

While mastering technical tools
Cultivate systems thinking and philosophical reflection
Apply holistic frameworks like Yin-Yang and Five Elements

Limitations of the Model

It must be noted that this theory is not a panacea:

Not a Rigid Formula

Its value lies not in providing a rigid formula, but in guiding us to return to reality and think about problems from a more comprehensive perspective

The model provides a thinking framework, not standard answers
Specific applications need to consider actual scenarios
Architects ultimately must make judgments based on specific context

Requires Continuous Validation

Theory needs continuous validation and refinement in practice
Different scenarios may require adjustment and extension
Feedback and improvement in practice are encouraged

Supplement, Not Replace

The model is a tool to assist decision-making
Cannot replace professional judgment and experience
Should be used in combination with other methodologies

Future Outlook

Theory Development

This model has significant room for development:

Quantitative Metrics: Develop more precise quantitative indicators to make the theory more actionable
Tool Support: Develop analysis tools and automated diagnostic systems based on the model
Case Accumulation: Collect more practical cases to validate and enrich the theory
Cross-Domain Application: Explore applications of the model in other complex system domains

Practice Promotion

We hope this framework can help:

CTOs, infrastructure architects, and platform R&D teams
When facing increasingly complex AI infrastructure
Make wiser decisions

Ultimate Vision

Standing with sword in the midst of waves of change, embracing both the Yang of innovation and the Yin of governance, riding the system’s Qi above the currents

Conclusion

AI infrastructure stands at the starting point of a new era. We need not only technological innovation but also conceptual innovation.

The Yin-Yang Five Elements Qi-Yun model offers a unique perspective—combining Eastern philosophical wisdom with modern engineering practice—helping us find simplicity in complexity, stability in change, and unity in opposition.

We hope this model becomes a powerful tool for your thinking about AI infrastructure, helping you find your own “Way” in the balance and evolution of systems.

What Is AI-Native Infrastructure?

Jimmy Song — Sun, 18 Jan 2026 05:43:57 +0000

The essence of AI-native infrastructure is to make model behavior, compute scarcity, and uncertainty governable system boundaries.

AI-native infrastructure is not a simple checklist of technologies, but rather a new operating order designed for a world where “models become actors, compute becomes scarce, and uncertainty is the system default.”

The core of AI-native infrastructure is not faster inference or cheaper GPUs, but providing governable, measurable, and evolvable system boundaries for model behavior, compute scarcity, and uncertainty—making AI systems deliverable, governable, and evolvable in production environments.

Why We Need a More Rigorous Definition

The term “AI-native infrastructure/architecture” is being adopted by an increasing number of vendors, but its meaning is often oversimplified as “data centers better suited for AI” or “more complete AI platform delivery.”

In practice, different vendors emphasize different aspects of AI-native infrastructure:

Cisco emphasizes delivering AI-native infrastructure across edge/cloud/data center domains, highlighting delivery paths where “open & disaggregated” and “fully integrated systems” coexist (e.g., Cisco Validated Designs).
HPE emphasizes an open, full-stack AI-native architecture for the entire AI lifecycle, model development, and deployment.
NVIDIA explicitly proposes an AI-native infrastructure tier to support inference context reuse for long-context and agentic workloads.

For CTOs/CEOs, a definition that can guide strategy and organizational design must meet two criteria:

Clarify how the first-principles constraints of infrastructure have changed in the AI era
Converge “AI-native” from a marketing adjective into verifiable architectural properties and operating mechanisms

Authoritative One-Sentence Definition

AI-native infrastructure is:

An infrastructure system and operating mechanism premised on “models/agents as execution subjects, compute as scarce assets, and uncertainty as the norm,” which closes the loop on “intent (API/Agent) → execution (Runtime) → resource consumption (Accelerator/Network/Storage) → economic and risk outcomes” through compute governance.

This definition contains two layers of meaning:

Infrastructure: Not just a software/hardware stack, but also includes scaled delivery and systemic capabilities (consistent with vendors’ emphasis on “full-stack integration/reference architectures/lifecycle delivery”).
Operating Model: It inevitably rewrites organizational and operational methods, not just a technical upgrade—budget, risk, and release rhythm are strongly bound to the same governance loop.

A More Precise Layered Model: Paradigm Above, Refactoring Below

AI-native infrastructure should not be framed as a replacement of cloud-native infrastructure, but as a combined model of higher-level paradigm + downward refactoring.

Layer 0 — Cloud-Native Substrate (retained): Kubernetes, container runtime, CNI/CSI, and observability primitives remain the execution substrate.
Layer 1 — Resource Model Rewrite (core breakpoint): Scheduling semantics shift from CPU/memory-centric to GPU/token/context-centric; context and token become first-class resources. GPU virtualization, slicing, sharing, and oversubscription (such as HAMi) belong here.
Layer 2 — Runtime and Control Plane Shift: Workload form shifts from request/response services to agentic loops, async multi-step workflows, and fused inference/training/evaluation execution paths.
Layer 3 — Governance and Scheduling First: Scheduling becomes the primary control plane; deployment becomes a secondary concern under budget, risk, and policy constraints.

In other words: Cloud Native is the substrate, AI Native is the semantic override and resource-model shift on top of that substrate.

Three Premises

The core premises of AI-native infrastructure are as follows. The diagram below illustrates the correspondence between these three premises and governance boundaries.

Figure 1: Three constitutional premises of AI-native infrastructure

Model-as-Actor: Models/agents become “execution subjects”
Compute-as-Scarcity: Compute (accelerators, interconnects, power consumption, bandwidth) becomes the core scarce asset
Uncertainty-by-Default: Behavior and resource consumption are highly uncertain (especially in agentic and long-context scenarios)

These three points collectively determine: the core task of AI-native infrastructure is not to “make systems more elegant,” but to make systems controllable, sustainable, and capable of scaled delivery under uncertain behavior.

Boundaries: What AI-Native Infrastructure Manages and What It Doesn’t

In practical engineering, defining boundaries helps focus resources and capability development. The table below summarizes what AI-native infrastructure focuses on versus what it doesn’t:

Not focused on:

Prompt design and business-level agent logic
Individual model capabilities and training secrets
Application-layer product features themselves

Focused on:

Compute Governance: Quotas, budgets, isolation/sharing, topology and interconnects, preemption and priorities, throughput/latency versus cost tradeoffs
Execution Form Engineering: Unified operation, scheduling, and observability for training/fine-tuning/inference/batch processing/agentic workflows
Closed-Loop Mechanisms: How intent is constrained, measured, and mapped to controllable resource consumption and economic/risk outcomes

Verifiable Architectural Properties: Three Planes + One Loop

To facilitate understanding, the following sections introduce the core architectural properties of AI-native infrastructure.

The diagram below shows the visualization of the three planes and the closed loop, facilitating rapid boundary alignment during reviews.

Figure 2: Three Planes and One Loop reference architecture

Three Planes:

Intent Plane: APIs, MCP, Agent workflows, policy expressions
Execution Plane: Training/inference/serving/runtime (including tool calls and state management)
Governance Plane: Accelerator orchestration, isolation/sharing, quotas/budgets, SLO and cost control, risk policies

The Loop:

Only with an “intent → consumption → cost/risk outcome” closed loop can it be called AI-native.

This is also why NVIDIA elevates the sharing and reuse of “new state assets” like inference context to an independent AI-native infrastructure layer: essentially bringing the resource consequences of agentic/long-context into governable system boundaries.

AI-Native vs Cloud Native: Where the Differences Lie

Cloud Native focuses on delivering services in distributed environments with portability, elasticity, observability, and automation. Its governance objects are primarily service/instance/request.

AI-native infrastructure addresses a different set of structural problems:

Execution unit shift: From service request/response to agent action/decision/side effect
Resource constraint shift: From elastic CPU/memory to hard GPU/throughput/token constraints and cost ceilings
Reliability pattern shift: From “reliable delivery of deterministic systems” to “controllable operation of non-deterministic systems”

Therefore, AI-native is not a parallel track that replaces cloud-native, but a reinterpretation of cloud-native under AI constraints: the substrate remains, while resource, runtime, and governance semantics are rewritten.

Bringing It to Engineering: What Capabilities AI-Native Infrastructure Must Have

To avoid “right concept, misaligned execution,” the following minimum closed-loop capabilities are listed.

Resource Model: Making GPU, Context, and Token First-Class Resources

Cloud native abstracts CPU/memory into schedulable resources; AI-native must further bring the following resources under governance:

GPU/Accelerator Resources: Scheduled and governed by partitioning, sharing, isolation, and preemption
Context Resources: Context windows, retrieval paths, cache hits, KV/inference state asset reuse, etc., which directly affect tokens and costs
Token/Throughput: Become measurable capacity and cost carriers (can enter budgets, SLOs, and product strategies)

When tokens become “capacity units,” the platform is no longer just running services, but operating an “AI factory.”

Budgets and Policies: Binding “Cost/Risk” to Organizational Decisions

AI systems cannot operate with a “ship and done” approach. Budgets and policies must become the control plane:

Trigger rate limiting/degradation when budgets are exceeded
Trigger stricter verification or disable high-risk tools when risk increases
Version releases and experiments are constrained by “budget/risk headroom” (institutionalizing release rhythm)

The key is infrastructure solidifying organizational rules into executable policies.

Observability and Audit: Making Model Behavior Accountable and Observable

Traditional observability focuses on latency/error/traffic; AI-native must add at least three types of signals:

Behavior Signals: Which tools the model called, which systems it read/wrote, what actions it took, what side effects it caused
Cost Signals: Tokens, GPU time, cache hits, queue wait, interconnect bottlenecks
Quality and Safety Signals: Output quality, violation/over-privilege risks, rollback frequency and reasons

Without “behavior observability,” governance cannot be implemented.

Risk Governance: Bringing High-Risk Capabilities Under Continuous Assessment and Control

When model capabilities approach thresholds that can “cause serious harm,” organizations need a systematic risk governance framework, not relying on single-point prompts or manual reviews.

Can be split into two layers:

System-Level Trustworthiness Goals: Organizational-level requirements for security, transparency, explainability, and accountability
Frontier Capability Readiness Assessment: Tiered assessment of high-risk capabilities, launch thresholds, and mitigation measures

The value lies in: transforming “safety/risk” from concepts into executable launch thresholds and operational policies.

Takeaways / Checklist

The following checklist can be used to determine whether an organization has entered the AI-native stage:

Do we treat models as “agents that act,” not as replaceable APIs?
Do we bring compute and budgets into business SLAs and decision processes?
Do we treat uncertainty as the default premise, not as an exception?
Do we have audit, rollback, and accountability for model behavior?
Do we have cross-team AI governance mechanisms, not single-point engineering optimizations?
Can we explain the system’s operating boundaries, cost boundaries, and risk boundaries?

Summary

The essence of AI-native infrastructure lies in: taking models as behavior subjects, compute as scarce assets, and uncertainty as the norm, achieving deliverable, governable, and evolvable AI systems through governance and closed-loop mechanisms. Only by engineering these capabilities can organizations truly step into the AI-native stage.

References

AI-Native Infrastructure One-Page Reference Architecture: Three Planes + One Loop

Jimmy Song — Sun, 18 Jan 2026 00:00:00 +0800

The true value of architecture is enabling organizational consensus on complex systems within five minutes, not creating another new technology stack.

Industry-leading vendors emphasize different aspects: Cisco focuses more on AI-native infrastructure and reference designs such as Cisco Validated Designs; HPE emphasizes an open, full-stack AI-native architecture across the AI full lifecycle; NVIDIA explicitly proposes adding a new AI-native infrastructure tier for inference context reuse in long-context and agentic workloads. This chapter converges these perspectives into a verifiable architecture framework: Three Planes + One Loop.

Note

The “architecture” in this chapter serves as a review framework, not a component checklist. The goal is to unify organizational language and review boundaries, not to reinvent the technology stack.

One-Page Architecture Overview

Below is the detailed reference architecture diagram for the three planes and closed loop of AI-native infrastructure, helping readers quickly establish an overall understanding:

Figure 1: Three planes detailed architecture diagram

This diagram can be understood as: The new control plane (Intent) of AI-native infrastructure must be constrained by the Governance Plane and produce measurable resource consequences in the Execution Plane.

This is also the common ground behind different vendor narratives: Cisco uses reference designs and delivery frameworks to make infrastructure capabilities scalable and replicable; HPE uses open/full-stack to cover lifecycle delivery; NVIDIA elevates the reuse of “context state assets” to an independent infrastructure layer. All three point to the same issue: incorporating AI resource consequences into governable system boundaries.

Core Capabilities of the Three Planes

This section details the core capabilities of each of the three planes to help clarify focus areas during architecture reviews.

Intent Plane

The Intent Plane is responsible for expressing “what I want,” including the following capabilities:

Inference/Training APIs (entry points and contracts)
MCP/Tool calling protocols and tool catalog (standardizing tool access as “declaratable capability boundaries”)
Agent/Workflow (breaking down tasks into executable steps)
Policy as Intent: priorities, budgets, quotas, compliance/security constraints (front-loaded in the form of “intent”)

Key Point: The Intent Plane is not the starting point itself; the real starting point is—whether intent can be translated into executable and governable plans. Otherwise, Agents/MCP will only amplify uncertainty: more tools, longer chains, larger state spaces, and more uncontrollable resource consumption.

During architecture reviews, focus on these questions:

Is intent declarable (contract) and rejectable (admission)?
Does intent carry budget/priority/compliance constraints (policy as intent)?
Is the translation from intent to execution traceable?

Execution Plane

The Execution Plane is responsible for landing intent into “actual execution,” mainly including:

Training, fine-tuning, inference serving, batch processing, agentic runtime
“State and Context” services: cache/KV/vector/context memory, etc., for carrying inference context, retrieval results, and session state
Full-chain observability hooks: token metering, GPU time, video memory, network traffic, storage I/O, queue wait times, etc.

An industry trend worth emphasizing: as long-context and agentic workloads become widespread, “context” itself becomes a critical state asset and may even rise to become an independent infrastructure layer. NVIDIA explicitly proposes inference context memory storage in the Rubin platform, establishing an AI-native infrastructure tier to provide shared, low-latency inference context at the pod level to support reuse (for long-context and agentic workloads).

Review points focus on three things:

Is execution measurable: Can attribution be done across token/GPU/network/storage dimensions?
Is state governable: What are the lifecycle, reuse boundaries, and isolation strategies for context and cache?
Is observability closed-loop oriented: Observability is not for “seeing,” but for “enabling governance to correct deviations.”

Governance Plane

The Governance Plane is the “core differentiator” of AI-native infrastructure, responsible for transforming resource scarcity and uncertainty into a controllable system:

Budget/quotas/billing: governing consumption across teams, tenants, projects, models, and agent tasks
Isolation and sharing strategies: same-card sharing, video memory isolation, preemption, priorities, fairness
Topology-aware scheduling: incorporating GPU, interconnect, network, and storage topology into placement (especially in training and high-throughput inference)
Risk and compliance control: audits, policy enforcement points, sensitive data and access control
Integration with FinOps/SRE/SecOps: incorporating cost, reliability, and risk into a single operational mechanism

From a vendor narrative perspective, this layer typically corresponds to “reference architecture + full-stack delivery”: Cisco emphasizes accelerating and scaling delivery in AI infrastructure through “fully integrated systems + Cisco Validated Designs”; HPE emphasizes end-to-end delivery with “open, full-stack AI-native architecture” to support model development and deployment.

The baseline question for Governance Plane reviews is: Can you make explainable resource allocation and degradation decisions under budget/risk constraints?

Closed-Loop Mechanism Explained

This section introduces the core workflow of the closed-loop mechanism to help understand the essential difference between AI-native and AI-ready.

Significance of the Closed-Loop Mechanism

The closed loop is the most confusing yet critical dividing line between AI-native and “AI-ready.”

The minimal implementation of the closed loop includes four steps:

Admission: Bind intent with policy at the entry point (budget, priority, compliance)
Translation: Translate intent into executable plans (select runtime, resource specifications, topology preferences)
Metering: End-to-end metering and attribution across tokens/GPU/network/storage
Enforcement: Budget triggers degradation/rate limiting/preemption; risk triggers isolation/audits; SLO triggers scaling/routing

In other words: The closed loop is not a “monitoring dashboard,” but a “governance-driven real-time correction mechanism.” If there is no closed loop for “intent → consumption → cost/risk outcomes,” systems can easily spin out of control across cost, risk, quality, and other dimensions.

This is also why “AI-native” is often accompanied by changes in operating model: when system execution speed and resource consumption are amplified by models/agents, organizations must front-load governance mechanisms and institutionalize them. LF Networking also explicitly points out: becoming AI-native is not just a technical migration, but a redefinition of the operating model.

Practical Usage of the One-Page Architecture

In subsequent chapters, this “one-page architecture” can be repeatedly reused as a review template:

Discussing MCP/Agent: Position them in the Intent Plane and constrain with the closed loop (admission/translation) to avoid “intent proliferation”
Discussing runtime and platforms: Place in the Execution Plane, focusing on observable, attributable, governable state assets (context/cache/KV/vector)
Discussing GPUs, scheduling, costs: Ground in the Governance Plane, using budget/isolation/topology/metering as leverage points
Discussing enterprise implementation: Use the closed loop to examine if it’s “truly AI-native” (whether cost/risk outcomes can be written back as executable policies)

If you can only remember one sentence: The determination of AI-native is not in “how many AI components are used,” but in “whether there exists an executable governance closed loop that constrains intent to controllable resource consequences and economic/risk outcomes.”

Summary

The one-page reference architecture provides a unified systems language and review framework for AI-native infrastructure. Through the three planes of Intent, Execution, and Governance, combined with the closed-loop mechanism, organizations can achieve efficient collaboration in architecture design, resource governance, and risk control. Looking ahead, as AI-native capabilities continue to mature, the governance closed loop will become a core competitive advantage for enterprises implementing AI.

References

Why Start with Compute Governance, Not API Design

Jimmy Song — Sun, 18 Jan 2026 05:45:13 +0000

Compute and governance boundaries are the true foundation of AI-native infrastructure architecture.

The previous chapter presented a “Three Planes + One Closed Loop” reference architecture. This chapter focuses on a core CTO/CEO-level question:

How should AI-native infrastructure be layered? What belongs in the “control plane” of APIs/Agents, what belongs in the “execution plane” of runtime, and what must be pushed down to the “governance plane (compute and economic constraints)”?

This question is critical because over the past year, many platform companies “pivoting to AI” have fallen into a common trap: treating AI as an API morphology change rather than a system constraint change. When your system shifts from “serving requests” to “model behavior” (multi-step Agent actions with side effects), what truly determines system boundaries is often not the elegance of API design, but rather: whether compute, context, and economic constraints are institutionalized as enforceable governance boundaries.

The core argument of this chapter can be summarized as:

AI-native infrastructure must be designed starting from “Consequence” rather than stacking capabilities from “Intent”; the control plane is responsible for expressing intent, but the governance plane is responsible for bounding consequences.

The Purpose of Layering: Engineering the Binding Between “Intent” and “Resource Consequences”

In AI-native infrastructure, mechanisms like MCP, Agents, and Tool Calling enhance system capabilities while also introducing higher risks. These risks are not abstract “uncontrollability,” but rather engineering “unbudgetable consequences”:

Path explosion in behavior, long contexts, and multi-round reasoning bring long-tail resource consumption;
The same “intent” can lead to orders-of-magnitude differences in tokens, GPU time, and network/storage pressure;
Without governance closed loops, systems will move toward “cost and risk runaway” while becoming “more capable.”

Therefore, the fundamental purpose of layering is not abstract aesthetics, but achieving a hard constraint goal:

Ensure each layer can translate upper-layer “intent” into executable plans and produce measurable, attributable, and constrainable resource consequences.

In other words, layering is not about making architecture diagrams clearer, but about encoding “who expresses intent, who executes, and who bears consequences” into system structure.

AI-Native Infrastructure Five-Layer Structure and “Three Planes” Mapping

To help understand the layering logic, the diagram below refines the “Three Planes” architecture from the previous chapter, proposing a more actionable “five-layer structure”:

Figure 1: Layered governance relationship from intent to consequence

Top two layers = Intent Plane
Middle two layers = Execution Plane
Bottom layer = Governance Plane

Below is a detailed expansion of the five-layer architecture, showing the primary responsibilities and typical capabilities of each layer:

Figure 2: Five-layer architecture diagram

It is important to note that MCP belongs to Layer 4 (Intent and Orchestration Layer), not Layer 1. The reason is that MCP primarily defines “how capabilities are exposed to models/Agents and how they are invoked,” addressing control plane consistency and composability, but does not directly take responsibility for “how the resource consequences of capability invocations are metered, constrained, and attributed.”

MCP/Agent is the “New Control Plane,” But Must Be Constrained by the Governance Layer

MCP/Agent is called the “new control plane” because it moves system “decisions” from static code to dynamic processes:

“Tool catalogs + schemas + invocations” form a composable capability surface;
Agents complete tasks by selecting tools, invoking tools, and iterating reasoning;
“Policy” is no longer just in code branches but expressed as routing, priorities, budgets, and compliance intent.

However, it is crucial to emphasize an infrastructure stance, which is also the foundation of this chapter:

MCP/Agent can express intent, but the key to AI-native is: intent must be translated into governable execution plans and metered and constrained within economically viable boundaries.

This statement aims to correct two common misconceptions:

Control plane is not the starting point: Treating MCP/Agent as “the entry point for AI platform upgrades” easily leads systems down a “capability-first” path;
Governance plane is the baseline: When compute and tokens become capacity units, any unconstrained “intent expression” will leak as cost, latency, or risk.

Therefore, system layering should be clear: Layer 4 is responsible for “expression,” Layers 1/2/3 are responsible for “fulfillment and bearing consequences,” and the governance loop is responsible for “correction.”

“Context” Is Rising to a New Infrastructure Layer

In traditional cloud-native systems, request states are mostly short-lived, relying more on application-layer state management. Infrastructure typically only handles “computation and networking” without needing to understand the economic value of “request context.”

AI-native infrastructure is different. Long-context, multi-turn dialogue, and multi-agent reasoning mean inference state often survives across requests and directly determines throughput and cost. In particular, KV cache and context reuse are evolving from “performance optimization techniques” to “platform capacity structures.”

This can be summarized as an infrastructure law:

When a state asset (context/state) becomes a determinant variable of system cost and throughput, it rises from application detail to infrastructure layer.

This trend is gradually appearing in the industry: inference context and KV reuse are explicitly elevated to “infrastructure layer” capability development directions. Future expansion will include distributed KV, parameter caching, inference routing state, Agent memory, and a series of “state assets.”

The Foundation of AI-Native Infrastructure: Reference Designs and Delivery Systems

AI-native infrastructure is far more than “buying a few GPUs.” Compared to traditional internet services, AI workloads have three characteristics that make the “foundation” more engineered and productized:

Stronger topology dependencies: Network fabric, interconnects, storage tiers, and GPU affinity determine available throughput;
Harder scarcity constraints: GPU and token throughput boundaries are less “elastic” than CPU/memory;
Higher delivery complexity: Multi-cluster, multi-tenant, multi-model/multi-framework coexistence means only “replicable delivery” can scale.

Therefore, AI Infra is not just a component list, but must include “scalable delivery and repeatable operation” system capabilities:

Reference Designs (validated designs)

Codify “correct topology and ratios” into reusable solutions.

Automated Delivery

Institutionalize deployment, upgrade, scaling, rollback, and capacity planning.

Governance Implementation

Make budgeting, isolation, metering, and auditing default capabilities rather than after-the-fact patches.

From a CTO/CEO perspective, this means: what you purchase is not “hardware” but a “delivery system for predictable capacity.”

“Layered Responsibility Boundaries” from a CTO/CEO Perspective

To facilitate internal alignment on “who is responsible for what and what is the cost of failure,” the table below maps “technical layers” to “organizational responsibilities,” avoiding the scenario where platform teams only build control planes while no one bears consequence boundaries.

Layer	Typical Capabilities	Primary Owner (Recommended)	Cost of Failure
Layer 5 Business Interface	SLA, product experience, business goals	Product / Business	Customer experience and revenue impact
Layer 4 Intent/Orchestration (MCP/Agent)	Capability catalogs, workflow, policy expression	App / Platform / AI Eng	Behavior runaway, tool abuse
Layer 3 Execution (Runtime)	Serving, batching, routing, caching policies	AI Platform / Infra	Insufficient throughput, latency jitter
Layer 2 Context/State	KV/cache/context tier	Infra + AI Platform	Token cost spike, throughput collapse
Layer 1 Compute/Governance	Quotas, isolation, topology scheduling, metering	Infra / FinOps / SRE	Budget explosion, resource contention, incident spillover

Table 1: AI-Native Infrastructure Layer and Organizational Responsibility Mapping

As you can see, the organizational challenge of AI-native is not in “whether we have agents,” but in “whether inter-layer closed loops are established”. When model-driven amplification of consequences occurs, organizations must institutionalize governance mechanisms as platform capabilities: executable budgets, explainable consequences, attributable anomalies, and rewritable policies. This is the true meaning of “starting from compute governance” rather than “starting from API design.”

Conclusion

The layered design of AI-native infrastructure centers on engineering the binding between “intent” and “resource consequences.” The control plane is responsible for expressing intent, while the governance plane is responsible for bounding consequences. Only by institutionalizing governance mechanisms as platform capabilities can we ensure cost, risk, and capacity remain controllable while enhancing capabilities. As context, state assets, and other new variables become infrastructure, AI Infra delivery systems will continue to evolve, becoming the foundation for sustainable enterprise innovation.

References

Operating and Governing AI-Native Infrastructure: Metrics, Budget, Isolation, Sharing, SLO to Cost

Jimmy Song — Sun, 18 Jan 2026 04:17:45 +0000

The key to governing AI-native infrastructure lies in how to institutionalize the closed-loop management of costs and risks arising from uncertainty.

In the cloud-native era, system operations were typically considered “basically deterministic”: request paths were predictable, resource curves were relatively stable, and scaling could respond promptly to load changes. However, entering the AI era, this assumption no longer holds—uncertainty has become the norm.

This chapter aims to provide CTOs/CEOs with key conclusions for architecture reviews:

The starting point of AI-native infrastructure is to treat uncertainty as the default input; The goal is to achieve closed-loop governance of the resource consequences (cost, risk, experience) arising from uncertainty.

This is also why “becoming AI-native” in organizational contexts increasingly points to the reshaping of operational methods and governance models: when system consequences are amplified, governance must be institutionalized.

What is an “Uncertain System”

In this handbook, “uncertainty” does not refer to randomness in the probabilistic sense, but to three types of phenomena in engineering practice:

Unpredictable behavior: execution paths change dynamically with model inference, especially evident in agentic processes (Agent intelligent workflows).
Unpredictable resource consumption: tokens, KV cache, tool calls, I/O, and network overhead exhibit long-tail and burst characteristics.
Non-linear consequences: the same “intent” can produce cost and risk outcomes differing by orders of magnitude.

Therefore, the infrastructure problem of AI-native infrastructure has shifted from “how to make the system more elegant” to:

How to ensure the system maintains economic viability, controllability, and recoverability when worst-case scenarios occur.

During architecture reviews, if you cannot answer “what is the worst case, where are the upper bounds, and how to degrade/rollback when triggered,” you are still reviewing the inertia extension of deterministic systems, not true AI-native systems.

Major Sources of Uncertainty

The following table summarizes common sources of uncertainty in AI-native infrastructure and their specific manifestations, facilitating quick reference for CTOs/CEOs.

Type	Manifestations	Impact Areas
Behavior Uncertainty	Agent task decomposition path changes, tool selection and call sequence changes, failure retry and reflection	Cost, Risk, Resilience
Demand Uncertainty	Concurrency and burst, long-tail requests, multi-tenant interference (noisy neighbor)	Resource pools, Experience, Isolation
State Uncertainty	Context reuse across requests, KV cache migration and sharing	Performance, Cost, Governance
Infrastructure Uncertainty	High sensitivity to network/storage/interconnect, congestion and jitter amplified into tail latency	Experience, Cost, Stability

Table 1: Sources and Manifestations of Uncertainty in AI-Native Infrastructure

Behavior Uncertainty

Behavior uncertainty is mainly reflected in changes to agent task decomposition paths, dynamic adjustment of tool selection and call sequences, and path explosion caused by failure retry, reflection, and multi-round planning. Tools and contexts are combined through standard interfaces (such as MCP protocol integration), significantly expanding system capability surfaces while making branch space a governance challenge.

More critically, tool calls are not “free external functions” - they occupy context windows and consume token budgets, amplifying cost and tail latency pressures. Therefore, behavior uncertainty is not merely “feature flexibility” at the product layer, but “cost and risk elasticity” at the platform layer, which must be budgeted, capped, and made auditable.

Demand Uncertainty

Demand uncertainty includes concurrency and burst (peaks), long-tail requests (ultra-long contexts, complex reasoning), and mutual interference under multi-tenancy (noisy neighbor). This drives capacity planning from “average capacity” to “tail capacity + governance strategies.”

In AI-native infrastructure, experience and cost are often determined not by average requests, but by the combination of tail requests: a small number of long-chain, long-context, tool-intensive requests can overwhelm shared resource pools. Therefore, demand uncertainty requires answering: which requests deserve guarantees, which must be throttled, and which should be isolated.

State/Context Uncertainty

State uncertainty is the most underestimated category in the AI era: context is a state asset, and it often exists across requests. When inference state / KV cache is elevated to a reusable, shareable, migratable system capability, it is no longer an application detail but a decisive variable for throughput and unit cost. NVIDIA in public materials identifies Inference Context Memory Storage as a new infrastructure layer, pointing to state reuse and sharing requirements for long-context and agentic workloads.

The conclusion is: “context/state” has changed from optional optimization to a critical infrastructure asset that must be meterable, allocable, and governable.

Infrastructure Uncertainty

AI workloads are far more sensitive to network, interconnect, and storage than traditional microservice workloads. Congestion, packet loss, and I/O jitter are amplified into tail latency and job completion time instability, creating “non-linear consequences” for experience and cost.

This type of uncertainty usually cannot be solved through “component selection” but requires end-to-end path engineering constraints: from topology, bandwidth, and queuing, to transport protocols, isolation strategies, and congestion control—all must be incorporated into the governance plane, not just the operations plane.

How Uncertainty Amplifies Across Layers

The diagram below illustrates the closed-loop relationship between metrics, budgets, and isolation strategies, emphasizing that governance must be rewritable.

Figure 1: SLO to cost feedback loop

The flowchart below demonstrates the cross-layer amplification path of uncertainty in AI-native infrastructure:

Figure 2: Uncertainty amplification across layers

Typical phenomena include:

Agent branch explosion: more tools and composable paths make tail costs increasingly uncontrollable.
Context inflation: long contexts and multi-round reasoning make KV cache a performance bottleneck and cost black hole.
Resource contention distortion: GPU/network contention under multi-tenancy makes “average performance” meaningless—tails must be governed.

Therefore, the core of AI-native is not “making execution stronger,” but enabling you to stably answer three questions:

Where are the upper bounds (budgets, steps, call counts, state occupancy)
What to do when crossing boundaries (degradation, rollback, isolation, blocking)
How results are rewritten (policy iteration and cost correction)

Engineering Response of AI-Native Infrastructure

Enterprises can refer to the following five “hard standards” during reviews—missing any one means inability to achieve closed-loop governance of uncertainty.

Admission: Ingress Admission Control

Implement tiered admission for requests with ultra-long contexts, oversized tool graphs, or ultra-high budgets
Bind “budget, priority, compliance” as part of intent (policy as intent)
Clearly communicate rejection reasons and explain why requests are denied

Key Point

The responsibility of admission is not “to allow features,” but to write consequence constraints into the contract.

Translation: Intent Translation to Governable Execution Plans

Select runtime, routing/batching strategies, and caching strategies for requests
“Cap” agent workflows: maximum steps, maximum tool calls, maximum tokens
Include fallback paths: deterministic alternatives, cached answers, manual/rule-based fallbacks

Key Point

Upgrade from “prompt-driven execution” to “plan-driven execution”—plans must be understandable and constrainable by the governance plane.

Metering: End-to-End Metering and Attribution

Meter tokens, GPU time, KV cache footprint, I/O, and network for each request/agent task
Attribute by tenant, project, model, and tool to form cost and quality metrics
Separately label “tail overhead” so long-tail costs no longer hide in averages

Key Point

No ledger means no budget; no attribution means no governance, let alone ROI.

Enforcement: Budget and Degradation Mechanisms

Budget triggers: rate limiting, degradation, preemption, queuing (by priority and tenant isolation)
Risk triggers: isolation

Conclusion

The core of AI-native infrastructure governance lies in front-loading uncertainty, layered metering, policy feedback, and institutionalized constraints to form a closed loop of cost and risk. Only with engineering mechanisms such as Admission, Translation, Metering, and Enforcement can systems achieve economically viable, controllable, and recoverable operations under normalized uncertainty.

References

Organization and Culture: How the Operating Model Changes

Jimmy Song — Sun, 18 Jan 2026 04:18:02 +0000

The compute governance closed loop is the foundational safeguard for sustainable innovation in AI-native organizations.

Perspective

API/Agent/MCP solve “how intent is expressed,” while compute governance addresses “whether the resource consequences of intent are economically viable and risk-controllable.” In the AI era, the latter becomes a prerequisite for the former. API-first without governance only amplifies costs and uncertainty, pushing organizations into the trap of “functional but unsustainable.”

The FinOps Foundation states directly in “Scaling Kubernetes for AI/ML Workloads with FinOps” that Kubernetes elasticity can easily evolve into a runaway cost problem. Therefore, FinOps should not be just cost reporting, but must become a shared operating model where every scaling decision simultaneously answers two questions: Are performance SLOs met?, and Is it affordable?.

The Challenge of API-first “Implicit Assumptions” in the AI Era

The diagram below shows the boundary relationships and accountability chains between platform, ML, and security teams.

Figure 1: Organizational boundaries and accountability chain

The intuitive path of API-first is: first make the interfaces and workflows work, then gradually optimize performance and cost through engineering. In AI-native infrastructure, this path often fails because it relies on three implicit assumptions that no longer hold in the AI era.

Assumption 1: Resources are not the core scarcity

Traditional software bets scarcity on engineering efficiency, throughput, and stability; whereas AI-native infrastructure scarcity comes primarily from asset boundaries like GPU/interconnect/power consumption. Scarcity is no longer “slow to scale,” but “hard to scale and expensive,” constrained by both supply chain and datacenter conditions.

Assumption 2: Request costs are predictable

Traditional request cost distribution is relatively stable; AI requests are inherently long-tailed: branching in agentic tasks, inflation of long contexts, and chain amplification of tool calls all make tokens and GPU time into random variables that cannot be linearly extrapolated. You think you’re scaling “QPS,” but actually you’re scaling “total cost of tail probability events.”

Assumption 3: State is ephemeral and discardable

The cloud-native era emphasized stateless scaling with externalized state; but on the inference side, inference state/context reuse often determines whether unit costs are controllable. NVIDIA describes this in Rubin’s ICMS (Inference Context Memory Storage) as the “context storage challenge brought by new inference paradigms”: KV cache needs reuse across sessions/services, sequence length growth causes linear KV cache inflation, forcing persistence and shared access, forming a “new context tier,” and proving with TPS and energy efficiency gains that this is not a nice-to-have, but a threshold for scalability.

Conclusion

In AI-native infrastructure, state and compute governance have become prerequisites for “whether it can run,” not post-optimization items.

The Nature of Compute Governance: What is Being Governed

“Compute governance” is often misunderstood as “managing GPUs,” but what truly needs governance is the resource consequences of intent. More precisely, it’s governing the combined effects of four types of objects:

Token Economics

Each request/task’s token consumption, context inflation, implicit token tax from tool definitions and intermediate results, ultimately directly mapping to cost and latency.

Accelerator Time

GPU time, memory footprint, batching strategies, and the impact of routing and cache hits on effective throughput. The key is not “whether there are GPUs,” but “whether output per unit GPU time is controllable.”

Interconnect and Storage (Fabric & Storage)

Network and storage pressures from training all-reduce, inference KV/cache sharing, and cross-service data movement. AI performance and cost are often amplified by fabric, not by APIs.

Organizational Budget and Risk (Budget & Risk)

Multi-tenant isolation, fairness, audit, compliance, and accountability. These determine whether the system can scale to multiple teams/business lines, not just scaling demos to more instances.

The FinOps Foundation also emphasizes: AI/ML cost drivers are not just GPUs; storage (checkpoints/embeddings/artifacts), network (distributed training/cross-AZ), and additional licensing and marketplace fees often “quietly exceed compute.” Therefore, governance objects must cover end-to-end, not just stare at inference bills.

MCP/Agent: Amplification Effects Under Governance Gaps

MCP/Agent expand the “capability surface,” but simultaneously make cost curves steeper, especially showing exponential amplification when governance is missing:

More tools, more branches: Planning space expands, tail probability rises, cost volatility becomes uncontrollable.
Tool definitions and intermediate results consume context: Directly consuming context window and tokens, translating to cost and latency.
Stronger tool usage triggers more external I/O: External system calls, network round trips, and data movement all enter the overall cost function.

Anthropic explicitly states in “Code execution with MCP” that direct tool calls increase cost and latency due to tool definitions and intermediate results consuming context window; when tool numbers rise to hundreds or thousands, this becomes a scalability bottleneck, thus proposing code execution forms to improve efficiency and reduce token consumption.

Conclusion

In the MCP/Agent era, governance is not about suppressing innovation, but making innovation sustainable within budget boundaries. Without governance, agents are not productivity tools, but cost amplifiers.

Minimal Implementation Path for “Compute Governance First”

You don’t have to bind to any vendor, but you must implement a “minimum viable governance stack.” The goal is not perfection, but giving the system controllable boundary conditions from day one.

Admission and Budget (Admission + Budget)

Set budgets and priorities for workload types (training/inference/agent tasks).
Include budget, max steps, max tokens, max tool calls in policy-as-intent, and enforce at the entry point.

Practice Recommendation

FinOps’ core view is: embed FinOps early into architecture, making every scaling decision simultaneously answer “performance” and “affordability,” otherwise bills only get attention when incidents occur.

End-to-End Metering and Attribution (Metering + Attribution)

At minimum achieve one traceable chain: request/agent → tokens → GPU time/memory → network/storage → cost attribution (tenant/project/model/tool).
Without attribution, there is no governance; without governance, enterprise scaling is impossible, because costs and responsibilities cannot align, and organizations will internally waste energy on “who consumed the budget.”

Isolation and Sharing (Isolation + Sharing)

Sharing for improving utilization; isolation for reducing risk. Both must exist simultaneously, not either/or.
CNCF’s Cloud Native AI report notes: GPU virtualization and sharing (like MIG, MPS, DRA, etc.) can improve utilization and reduce costs, but requires careful orchestration and management, and demands collaboration between AI and cloud-native engineering teams.
The key to governance is not choosing sharing or isolation, but making it an executable policy: who shares under what conditions, who isolates under what conditions.

Topology and Network as First-Class Citizens (Topology + Fabric First)

AI training and high-throughput inference are highly sensitive to network characteristics.
Cisco’s AI-ready infrastructure design guides and related CVD/Design Zone emphasize: building high-performance, lossless Ethernet fabric for AI/ML workloads, and delivering reference architectures and deployment guides through validated designs.
This means topology is not “the datacenter team’s business,” but a core variable determining whether JCT, tail latency, and capacity models hold.

Context/State Becomes a Governance Object (Context as a Governed Asset)

When long-context and agentic become mainstream, KV cache and inference context reuse will directly determine unit costs.
NVIDIA’s ICMS defines this as a “new context tier” for solving KV cache reuse and shared access, emphasizing TPS/energy efficiency gains.
In this era, treating context as a temporary variable is actively relinquishing cost control.

Anti-Pattern Checklist

The following anti-patterns are not “engineering inelegance,” but will trigger organizational loss of control, worth vigilance.

API-first, treating governance as post-optimization

Result: System launches first, only to discover unit costs and tail latency are uncontrollable, can only “hard brake” through feature limiting/rate limiting, ultimately locking the product roadmap.
Contrast: FinOps points out elasticity easily becomes runaway costs, must advance cost governance into architecture decisions.

Treating MCP/Agent as capability accelerators, not cost amplifiers

Result: More tools make it “smarter,” but token and external call costs rise exponentially, engineering teams forced to fight systemic amplification with “more complex prompts and rules.”
Contrast: Anthropic notes tool definitions and intermediate results consume context, increase cost and latency, proposing more efficient execution forms as the scalability path.

Only buying GPUs, without sharing/isolation and orchestration

Result: Low utilization, severe contention, budget explosion, organizations internally blame each other “who’s grabbing resources, who’s burning money.”
Contrast: CNCF Cloud Native AI report emphasizes sharing/virtualization improves utilization, but must match orchestration and collaboration mechanisms.

Ignoring network and topology, treating AI as ordinary microservices

Result: Training JCT and inference tail latency amplified by network, capacity planning and cost models fail, more scaling makes it more unstable.
Contrast: Cisco in AI-ready network design and validated designs makes requirements like lossless Ethernet fabric critical foundations for AI/ML.

Summary

The first-principle entry point for AI-native is the compute governance closed loop: budget and admission, metering and attribution, sharing and isolation, topology and network, context assetization. API/Agent/MCP remain important, but must be constrained by this closed loop, otherwise the system can only oscillate between “smarter” and “more bankrupt.”

References

Migration Roadmap: From Cloud Native to AI Native

Jimmy Song — Sun, 18 Jan 2026 04:20:11 +0000

Migration is not “rebuilding the platform,” but using governance loops and organizational contracts to transform uncertainty into controllable engineering capabilities.

The previous five chapters have established: AI-native infrastructure is Uncertainty-by-Default. Therefore, the architectural starting point must be compute governance loops, not “connect a model and call migration complete.” Otherwise, systems easily spiral out of control in three dimensions: cost (runaway cost), risk (unauthorized actions/side effects), and tail performance (P95/P99 and queue tail behavior).

This explains why the FinOps Foundation emphasizes: running AI/ML on Kubernetes, “elasticity” easily evolves into uncontrollable cost overflow. FinOps must be incorporated into architecture and organization upfront as a shared operating model, not as an after-the-fact reconciliation exercise.

This article presents an actionable migration roadmap, covering both technical evolution paths and organizational implementation approaches. You don’t need to “rebuild an AI platform” all at once, but you must establish working governance loops at each stage: budget/admission, metering/attribution, sharing/isolation, topology/networking, and context assetization.

The North Star: From Platform Delivery to Governance Loops

The diagram below shows the migration path from bypass pilot to AI-first refactoring.

Figure 1: AI-native migration roadmap

Cloud-native migration typically centers on “capability delivery”: CI/CD, self-service platforms, service governance, and auto-scaling. Its default assumptions: systems are deterministic, costs grow linearly with requests, and scaling doesn’t significantly alter system boundaries.

AI-native migration must center on “governance loops”, focusing on cost, risk, tail performance, and state assets. Its default assumptions are precisely the opposite: systems are inherently uncertain, and the “actions and consequences” of inference/agents drive costs and risks into nonlinear territory.

North Star Definition

AI-Native Migration = Establish an AI Landing Zone + Compute Governance Loop + Context Tier, and ensure all agents/APIs/runtimes operate within this loop.

Elevating “Landing Zone” to North Star level here isn’t chasing trends—it’s because it naturally serves an organizational-level task: delineating responsibility boundaries between platform teams and workload teams. Major cloud providers universally use Landing Zones to host “shared governance baselines” (networking, identity, policies, auditing, quota/budget), while business teams iteratively build applications within controlled boundaries. For AI, this boundary is the carrier of the governance loop.

Migration Prerequisites: Build Three Foundations First, Then Scale Applications

You can run PoCs and build applications in parallel, but if these three foundations are missing, any “application explosion” can easily transform into platform firefighting and financial disputes.

Foundation A: FinOps / Quotas as Control Plane (Finance and Quotas as Control Plane)

The first migration step is not “launch the first agent,” but incorporating budgets, alerts, showback/chargeback, and quotas into the infrastructure control plane:

Budgets and alerts are not just financial reports, but triggers for runtime policies (rate limiting, degradation, queuing, preemption).
showback/chargeback is not just accounting, but binding “cost consequences” to organizational decisions and product boundaries.
Quotas are not static limits, but evolvable governance instruments (dynamic budgets and priorities by tenant/team/use-case).

Migration Threshold

If you cannot attribute the primary consumption of each agent/job to team/project/model/use-case (at minimum covering tokens, GPU time, KV footprint, key network/storage), you haven’t reached the “scale” starting line. Piloting is acceptable, but expansion is not advisable.

Foundation B: Resource Governance (GPU Sharing/Isolation and Orchestration Capabilities)

The “elasticity” of AI-native infrastructure is constrained by how scarce compute is governed. Treating GPUs as ordinary resources typically results in low utilization and uncontrolled contention. Therefore, you need viable combinations of sharing/isolation and orchestration capabilities:

Sharing/partitioning: MIG/MPS/vGPU paths transform “exclusive” into “pooled.”
Scheduling upgrades: Introduce explicit modeling of topology, queues, fairness, preemption, and cost tiers.
Orchestration loop: Solidify isolation, preemption, and priority policies into executable rules.

The key is not which partitioning technology you choose, but whether you can elevate GPUs from “machine assets” to first-class governance resources and incorporate them into budget and admission systems.

Foundation C: Fabric as a First-Class Constraint (Network/Interconnect as First-Class Constraint)

Training and high-throughput inference are extremely sensitive to congestion, packet loss, and tail latency. Ignoring networking and topology leads to “seemingly sporadic but actually structural” problems:

Training JCT is amplified by tail behavior, invalidating capacity planning;
Inference P99 and queue tails are amplified, making SLOs difficult to honor.

Therefore, you need to build reusable AI-ready network baselines: capacity assumptions, lossless strategies, isolation domain partitioning, measurement and acceptance criteria. Networking is not “optimize later,” but baseline engineering that must land in Days 31–60.

Migration Path Selection: Layered by Organizational Risk and Technical Debt

Migration isn’t “pick one path and see it through,” but mapping organizations with different risk appetites and debt structures to different starting approaches and exit criteria. Paths can advance in parallel, but each needs defined applicable conditions and exit criteria.

Path 1: Bypass Pilot / Skunkworks

Applicable when cloud-native platforms are running stably, but AI demand is just emerging, organizational uncertainty is high, and governance mechanisms are not yet mature.

The approach is establishing an “AI minimum closed-loop sandbox” alongside the existing platform. The goal is not “feature completeness,” but “making the loop work”:

Independent GPU pool (or at least independent queue) + basic admission and budget
Minimal token/GPU metering and attribution
Controlled inference/agent entry points (max context / max steps / max tool calls)
“Failure-acceptable” SLOs and cost caps (define boundaries first, then discuss experience)

Exit criteria:

Cost curve is explainable (at minimum attributable to team/use-case)
GPU utilization and isolation strategies form reusable templates
Pilot capabilities can be absorbed into platform capabilities (enter Path 2)

Path 2: Domain-Isolated Platform

Applicable when AI has entered multi-team, multi-tenant stages, requiring “pilot assets” to be solidified into platform capabilities to prevent cost and risk from spreading across domains.

The approach is building an AI Landing Zone, where the platform team centrally manages shared governance capabilities, and workload teams iteratively build applications within controlled boundaries.

Platform-side essential modules (recommend organizing by “governance loop”):

Identity/Policy: Unified identity, policy distribution, and auditing (policy-as-intent)
Network/Fabric baseline: AI-ready network baseline and automated acceptance
Compute governance: Quotas, budgets, preemption, fairness, isolation/sharing
Observability & Chargeback: End-to-end metering, alerts, showback/chargeback
Runtime catalog: “Golden paths” and templated delivery for inference/training runtimes

Exit criteria: Platform provides “replicable AI workload landing approaches” and can scale use case count under budget constraints, rather than relying on manual firefighting to maintain stability.

Path 3: AI-First Refactor (AI Factory / Replatform)

Applicable when AI is core business, requiring infrastructure to be treated as a “production line” rather than a “cluster,” and optimization objectives to switch from “shipping features” to “throughput/unit cost/energy efficiency.”

The approach centers on “state assets + unit cost” refactoring:

Context/state of inference/agents is explicitly governed and reused (no longer application-level tricks)
Introduce Context Tier architectural assumptions: long context and agentic inference require inference state / KV cache to be reusable across nodes and sessions
Drive platform evolution with “unit token cost, tail latency, throughput/energy efficiency,” not “number of new components”

Exit criteria: Can consistently make engineering decisions using “unit cost and tail performance,” and treat context reuse as a platform capability rather than application team trick caching.

90-Day Actionable Plan: AI Landing Zone + Minimum Governance Loop

The goal is to establish “AI Landing Zone + minimum governance loop” within 90 days, forming a replicable template. The key is not covering all scenarios, but connecting the admission—metering—enforcement—feedback loop.

Day 0–30: Establish the Ledger (Cost & Usage Ledger)

First, define attribution dimensions, establish budgets/alerts and baseline reports, and implement quotas/usage controls.

Attribution dimensions: tenant/team/project/model/use-case/tool
Establish budgets and alerts, baseline reports (cost + business value metrics)
Implement quotas and usage controls (at minimum covering GPU quotas and key service quotas)

Deliverables:

Cost and usage dashboard (weekly-level, traceable)
“Admission Policy v0” (max context / max steps / max budget)

Day 31–60: Establish Resource Governance (GPU Governance + Scheduling)

This phase requires evaluating GPU sharing/isolation strategies, introducing topology/networking constraints, and forming two golden paths for inference and training.

GPU sharing/isolation strategy: MIG/MPS/vGPU/DRA path evaluation and PoC (executable strategy as acceptance criteria)
Introduce topology/networking constraints, form AI-ready network baseline and capacity assumptions (including acceptance criteria)
Form two templated delivery paths for inference/training

Deliverables:

Workload templates (1 each for inference and training)
Scheduling and isolation strategies (whitelisted, auditable)

Day 61–90: Establish the Loop (Enforcement + Feedback)

The final phase requires executing budget policies, migrating pilot use cases to the landing zone, and solidifying organizational interfaces.

Execute budgets: rate limiting/queuing/preemption/degradation strategies, linked to SLOs
Migrate pilot use cases to landing zone (or service landing zone capabilities)
Solidify “organizational interface”: platform team vs workload team responsibility boundaries (forming executable contracts)

Deliverables:

“AI Platform Runbook v1” (including oncall, changes, cost auditing)
Two replicable use case landing paths (new use cases <= 30 minutes to golden path)

Operating Model: The “Contract” Between Platform Teams and Workload Teams

Migration success depends on establishing clear, executable “organizational contracts.” The contract essence: who is responsible for “capability provision,” who is responsible for “behavioral consequences.”

Platform teams provide (must be stable)

Landing zone, network baseline, identity and policies, budget/quota systems, metering/attribution, GPU governance capabilities, runtime golden paths

Workload teams own (must be self-service)

Model selection, prompt/agent logic, tool integration, SLO definition, business value measurement, use case risk classification and rollback paths

This is also why the FinOps Framework emphasizes operating model (personas, capabilities, maturity) rather than just tools: without “contracts,” budgets are difficult to execute; if budgets cannot execute, loops cannot form.

Migration Anti-Patterns

Below are common migration anti-patterns and their consequences:

Anti-Pattern	Typical Consequences
Build only API/Agent platform, without ledger and budget	runaway cost (most common, and difficult to remediate afterwards)
Treat GPUs as ordinary resources, without sharing/isolation and scheduling upgrades	Low utilization + uncontrolled contention, platform forced to allocate compute via “administrative means”
Ignore networking and topology	Tail latency and training JCT amplified, capacity planning fails, SLOs difficult to honor
Context not assetized (only “tricky caching” within applications)	Unit cost out of control in long context/agentic era, reuse capabilities difficult to solidify as platform capabilities

Table 1: Common Migration Anti-Patterns and Consequences

Summary

The core of AI-native migration is not a “migration checklist,” but under uncertainty premises, incorporating cost, risk, and tail performance into a unified governance loop, using Landing Zone to carry organizational contracts, and using Context Tier to implement state reuse infrastructure capabilities. Only in this way can platform and business maintain controllability and efficiency during scaled evolution.

References

Glossary

Jimmy Song — Sun, 18 Jan 2026 05:19:24 +0000

Unified terminology is the first step toward organizational consensus.

Conclusion first: in the context of AI-native infrastructure, key terms must remain consistent; otherwise, both governance and communication lose focus.

The following glossary serves to align cross-team terminology.

Core Terms

AI Native Infrastructure

An infrastructure system premised on “models/agents as execution entities, compute as scarce assets, and uncertainty as the norm,” closed-looping “intent → execution → resource consumption → economic and risk outcomes” through compute governance.

Model-as-Actor

Models/agents become “execution entities” with action capabilities, capable of invoking tools, modifying system state, and producing side effects, thus requiring governance and audit.

Compute-as-Scarcity

Compute (GPU, interconnects, power consumption, bandwidth) becomes the core scarce asset, with expansion constrained by supply chain and data center conditions, and costs that cannot be made elastic.

Uncertainty-by-Default

Behavior and resource consumption are highly uncertain (especially in agentic and long-context scenarios), requiring verification and fallback mechanisms.

Intent Plane

The API, Agent, and policy expression layer responsible for expressing “what I want,” including priorities, budgets, compliance, and other policies.

Execution Plane

The training/inference/serving/runtime layer responsible for translating intent into actual execution, including state management, tool invocation, model routing, and so on.

Governance Plane

The quota/budget, isolation/sharing, and cost control layer responsible for bounding resource consequences, including topology-aware scheduling, SLO and risk policies.

The Loop

Possessing a closed loop of “intent → consumption → cost/risk outcomes,” comprising four steps: Admission, Translation, Metering, and Enforcement.

Compute Governance

Governing the resource consequences of intent, including four categories of objects: token economics, accelerator time, interconnect and storage, and organizational budgets and risks.

FinOps / Financial Operations

Embedding cost governance early into architecture so that every scaling decision simultaneously answers “whether performance meets requirements” and “whether it is affordable.”

Agent

An execution entity that completes tasks by selecting tools, invoking tools, and iterating reasoning, with uncertain behavioral paths and resource consumption.

MCP / Model Context Protocol

A protocol that standardizes tool access as “declaratable capability boundaries,” defining how capabilities are exposed to models/agents and how they are invoked.

Operating Model

Institutional design for organization and operational methods, including responsibility boundaries, collaboration mechanisms, and decision-making processes, answering “who is responsible for what and what are the costs of failure.”

References

Executive Checklist (10 Questions)

Jimmy Song — Sun, 18 Jan 2026 05:21:59 +0000

The following 10 questions assess whether an organization possesses the strategic and execution readiness for AI-native infrastructure. The diagram below categorizes these questions into three domains: strategy, governance, and execution, facilitating executive discussions.

Figure 1: Executive checklist structure diagram

Can you clearly define the unit cost for each major AI workload (e.g., per 1M tokens, per agent task, per batch job)?
Do you have budget/quota mechanisms that can constrain team/project/tenant compute consumption within controllable bounds?
Can you make explicit policy trade-offs between “performance (throughput/latency)—cost—risk” (rather than relying on verbal constraints)?
Can your platform handle uncertainty: spikes, long-tail effects, and resource fluctuations caused by agent path explosions?
Are agent/MCP “intents” mapped to actionable and billable/auditable resource consequences?
Do you have clear resource isolation and sharing strategies (same-card sharing, memory isolation, preemption, prioritization) to improve utilization?
Can you achieve cross-layer observability: end-to-end tracing from request/agent → runtime → GPU/network/storage → cost?
Does your infrastructure support rapid adoption of new hardware/interconnect/topology changes (heterogeneity and evolution are the norm)?
Has the organization established “AI SRE/ModelOps + FinOps” collaboration mechanisms and accountability boundaries (who owns cost and reliability)?
When you say “we are AI-native,” can you provide three planes + one closed loop architecture diagram and governance strategy on a single page?

References

HAMi Community Evolution: When AI Writes Code, What Makes an Open Source Community Valuable?

Jimmy Song — Wed, 15 Jul 2026 09:17:07 +0000

This deck uses HAMi’s growth story to explore what makes an open source community valuable in the AI-coding era. HAMi is a GPU virtualization and sharing middleware for Kubernetes, evolving from its 2021 open-source debut to the 2024 CNCF Sandbox and 2026 CNCF Incubating milestones. The core thesis: AI reduces the cost of producing code, but it does not reduce the cost of building consensus — the moat of an open source community is shifting from code production toward consensus and trust.

Use the controls or keyboard shortcuts below to navigate the embedded interactive slides.

Slide: HAMi Community Evolution

Key Topics

A journey from code to community — 2021 open source → 2024 CNCF Sandbox → 2026 CNCF Incubating
Community growth is the real infrastructure — 3,709 stars · 129 contributors · 16 releases · 609 forks
Organizations and geography — 863 organizations across 44 countries
AI changed software development — “Before AI coding” vs. the “AI coding era”
The maintainer’s new role — technical direction, community trust, coordination; future maintainers are consensus builders
Governance must evolve with AI — HAMi PR #2019 on disclosing AI-generated contributions

Cloud Native Community China Is Officially Established

Jimmy Song — Sat, 11 Jul 2026 10:00:00 +0800

On July 11, 2026, Cloud Native Community China was officially established. It is the CNCF-recognized cloud native community for China, hosted on the Open Community Groups platform.

China has no shortage of cloud native developers, contributors, and end-user companies, yet we’ve never had an official, unified, continuously-synced community entry point. This launch fills exactly that gap.

About Cloud Native Community China

CNCF has communities across many countries and regions worldwide. Each one connects its local cloud native developers, open source projects, end-user companies, and in-person events. Cloud Native Community China sits alongside these as a peer, together forming CNCF’s global community map.

Cloud native activity in China used to be scattered across city-level meetups, project SIGs, and individual company blogs. It was hard to know where the next worthwhile talk was happening, or what peers were paying attention to. The point of an official community site is to wire all those scattered nodes into one network.

What the Site Is For

The newly launched cncf-china community page does three core things.

Aggregating China’s cloud native events

Meetups, online talks, KubeCon meetups, project SIG meetings, and training sessions all get published here in one place. You no longer have to dig through separate channels to find what’s happening.

Syncing community updates

Project progress, releases, governance updates, and volunteer calls-to-action get pushed out through the community site, giving anyone following cloud native a stable information source.

Receiving notifications and staying connected

Once you join, you can subscribe to notifications and get new events and updates delivered first. For anyone who wants to contribute, meet peers, or keep up with the latest in cloud native, this is the most direct entry point today.

Founding Members

A community’s launch always rests on a group of people willing to carry it at the start. The founding members of Cloud Native Community China come from maintainers of active Chinese cloud native open source projects, end-user company representatives, and community organizers who have run local meetups for years. Most have been deeply involved in the CNCF ecosystem for a long time, across areas like Kubernetes, Istio, HAMi, observability, and AI infrastructure.

For the full, up-to-date founding member roster, check the cncf-china community page. To become a contributor or co-organizer, you can reach them through the community page as well.

Join the Community

If you’re a developer, SRE, or architect in the cloud native space, or your company runs on Kubernetes, Istio, or serverless, joining this community gives you at least three things:

Never miss an event: All official events are published in one place; just subscribe to notifications.
Meet peers and project maintainers: A community is fundamentally a human network, and a lot of collaboration starts at a single meetup.
Find a path into open source: If you want to contribute to CNCF projects but don’t know where to start, the community has ready-made guidance.

Visit Cloud Native Community China, join the community on the page, and subscribe to notifications. Once you’re in, you’ll receive pushes about cloud native events, updates, and community news in China.

Cloud native adoption in China is already large in scale, but the community’s organizational density has room to grow. Establishing Cloud Native Community China is the first step in wiring that momentum into CNCF’s global network. You’re welcome to join.

After HAMi Became a CNCF Incubating Project: Open Source Is Moving from Code to Consensus

Jimmy Song — Wed, 08 Jul 2026 13:51:44 +0000

After HAMi became a CNCF Incubating project, I want to talk about something overlooked: AI is shifting the scarce resource of open source communities from code to consensus.

On July 2, 2026, HAMi officially became a CNCF Incubating project (see the announcement). For an open source project, this means more than recognition of its technical capability; it means that community governance, ecosystem building, and real-world adoption have all entered a new phase.

But if you only read HAMi’s growth as “a GPU virtualization project succeeded,” you might miss the more important shift.

My time building the HAMi community has left me with one increasingly strong feeling: AI is changing how open source communities produce. As AI coding drives the cost of producing code lower and lower, the core of competition in open source will no longer be who wrote the most code, but who can build stronger technical consensus, attract more contributors, and form a sustainable ecosystem network.

This article is my attempt to lay out that argument clearly.

Figure 9: Congratulations to HAMi for becoming a CNCF Incubating project

HAMi’s Growth Shows That a Project’s Real Asset Is Its Community

Let’s start with the data. Here is where HAMi stands today:

Metric	Value
GitHub Stars	3,700+
Contributors	Nearly 500, from 27 countries
Participating organizations	Multiple, and growing
Release cadence	Once every three months

Table 3: HAMi community key metrics (as of July 2026)

Figure 10: HAMi community contributors map, contributors from 27 countries

I’m not listing these numbers to show off “growth metrics.” I’m making a different point: an open source project is shifting from “software maintained by a team” into “a technical community of people gathered around a shared goal.”

That distinction matters. Software can be forked, rewritten, or generated overnight by AI. But a community with a shared goal, trust, and rhythm cannot be forked. That is the irreplaceable asset of an open source project.

From a governance-maturity perspective, HAMi has passed through three milestones:

Figure 11: Evolution of HAMi’s governance maturity

Note the middle stretch: from entering Sandbox in August 2024 to reaching Incubating in July 2026, roughly two years. During those two years the code certainly grew, but what actually convinced the CNCF Technical Oversight Committee (TOC) was the diversification of the community, the formalization of governance, and real production adoption. None of that is something you produce by writing code.

When HAMi Open-Sourced in 2021, There Was No AI Coding

HAMi was first open-sourced in 2021. Back then, most developers did not see AI coding the way we do today. Whether an open source project survived depended on developers genuinely investing their time, discussing problems in issues, submitting code through pull requests, and building trust through code review.

Today, the environment has changed.

In a recent HAMi community livestream (Mastering HAMi DRA, Yang Shouren, HAMi Community Livestream Episode 2), someone asked the maintainers a question: “How much of HAMi’s code now comes from AI assistance?”

The answer from Yang Shouren stuck with me: about half of the code in the HAMi community today is already AI-assisted.

Half. And that share is still rising.

This immediately raises a sharp question: if AI can write more and more code, what is the value of an open source community? Could one person plus a few AI agents just fork a “new HAMi”?

My answer is no, because what AI lowers is the cost of producing code, not the cost of building technical consensus.

In the AI Era, Code Is No Longer Scarce; Consensus Is

Let me sharpen that point.

An AI agent can already do a lot today: write code, fix bugs, add tests, generate docs, produce migration scripts. These capabilities are getting stronger fast. But there are a few things in a community that AI cannot replace today, and in my judgment will not replace soon.

First, deciding which problems are worth solving.

Take HAMi. Why does GPU sharing matter? Not because “slicing cards finely” is a cool technique. It matters because once AI infrastructure scales, reality looks like this: GPU costs are enormous, heterogeneous hardware keeps multiplying, and Kubernetes’ native resource model is no longer enough.

The community has to first agree that “this problem is worth investing in” before anyone writes any code. That agreement is a human-to-human matter, supported by real scenarios, real costs, and real pain. AI can solve a problem you have already defined, but “which problem is worth defining” is decided by community consensus.

Second, choosing a technical path.

In GPU virtualization there are many paths: MIG, MPS, time-slicing, vGPU, DRA. Each has trade-offs, and choosing wrong can cost you two years of detours.

Code can be generated, but architectural choice is fundamentally a value judgment. HAMi’s decision on Ascend 910C to move from hardware SR-IOV to userspace HAMi-core was not about someone writing a better piece of code; it was about the maintainers holding to a judgment that “hardware partitioning is too coarse, software partitioning is more flexible.” That kind of judgment is ground out through repeated discussion, failure, and validation in the community, not prompted out.

Third, trust.

Users don’t choose HAMi because of “how much AI-generated code is in this repo.” They care about: who maintains it? Who reviews it? Are there real production cases? Does the community respond when something breaks?

Each of these is a relationship between people, a product of community organization, not a product of code quality.

Put these three together, and the production model of open source communities in the AI era is shifting:

Figure 12: How the open source production model is changing in the AI era

In the past, code came first and the community sedimented out of the code; in the future, consensus comes first, AI rapidly turns consensus into code, and code flows back to test the consensus. The center of gravity of the scarce resource moves from “code” on the left to “consensus” on the right.

In the AI Era, Open Source Governance Itself Has to Level Up

Since AI has become a new category of contributor, a community’s governance rules have to keep up.

My advice is: don’t treat AI merely as a tool, treat it as a new type of contributor. It used to be “developers write code, humans review”; in the future it will be “humans set intent, AI generates code, the community reviews, and shared knowledge is distilled.” There is an extra layer in the middle, and an extra layer of governance complexity.

HAMi is already responding to this. Its CONTRIBUTING.md is explicit:

If you are using any kind of AI assistance to contribute to HAMi, it must be disclosed in the pull request.

In other words, if you use AI to help with a contribution, you must declare it in the PR. But the community also knows that a norm without a gate is not enough (see the discussion in Issue #1998), and there is already ongoing discussion about how to give that norm real enforcement.

This is actually a problem every AI-era open source project will run into. I’d break it into a few questions:

Must AI-generated code always be declared?
Which model, and what context, did the contributor use?
How do you ensure the security of AI code, avoiding injection and licensing risks?
What process should maintainers use to review an AI diff they may not be able to fully trace themselves?

Whoever figures out and operationalizes these rules first will keep contribution quality stable in the AI era. HAMi’s exploration here is worth a look for every open source project.

What CNCF Incubating Really Means

Back to the promotion itself.

I’d lean against treating it as an “honor.” What CNCF Incubating really validates is not code quality, but whether a project has the capacity to become infrastructure. It examines a whole package: technical maturity, community governance, production adoption, and ecosystem building.

HAMi’s case, in one sentence, is not “a Chinese team built a GPU project.” It is this:

An AI Infrastructure community, jointly shaped by developers from around the world, is taking shape.

The first is a product story; the second is an ecosystem story. The Incubating recognition from the CNCF is recognizing the latter, because competition over infrastructure is never competition between individual products; it is competition between ecosystem networks.

Summary

The open source competition of the next decade will not be just a competition of code, but a competition of communities.

Once AI gives everyone near-infinite capacity to produce code, the truly scarce capabilities will be three: finding the right problem, building technical consensus, and organizing developers worldwide to solve a problem together.

HAMi’s path to CNCF Incubating is just one snapshot of how open source communities are evolving in the AI era. Code will keep getting cheaper, and consensus will keep getting more expensive. Whoever understands this inversion will be the one who can build open source communities with real depth in the AI era.

Join the HAMi Community

Add me on WeChat (jimmysong) or follow HAMi on GitHub to join the community focused on GPU virtualization and heterogeneous compute scheduling. Let’s talk about open source governance in the AI era.

Olares and HAMi: A New Inflection Point for Desktop AI Workstations

Jimmy Song — Wed, 24 Jun 2026 22:30:00 +0800

HAMi used to save cards in the cluster. Now it decides how good a desktop AI workstation feels.

Figure 1: Olares and HAMi: a new inflection point for desktop AI workstations

An Old Friend Mentions a Name

A few days ago an old friend came by to chat. We used to run the cloud-native community scene together back home, so we go way back. She recently joined a company called Olares, and somewhere in the conversation she dropped this: their project had integrated HAMi.

I run the HAMi community day to day, so whenever I hear someone using it for something, I want to take a closer look. I went and dug through the Olares repo and website, and my first reaction was, huh, this is actually interesting.

Isn’t this exactly the kind of local AI workstation I’d been eyeing forever but never pulled the trigger on? I wrote in My Personal AI Stack that for someone like me who mainly works with my head, subscribing to models beats maintaining a high-end GPU. But how far can “one machine running the full AI stack” really go, and who has actually built it, I’ve wanted to see with my own eyes.

What Is This Thing, Really

Olares isn’t a “NAS with a GPU bolted on,” and it isn’t a “mini PC with Ollama installed.” It’s more like a personal cloud built on Kubernetes: local models, apps, identity, remote access, dev environment, storage, even GPU governance, all packed into one machine as a desktop cloud OS.

HAMi isn’t filler here. It’s the layer that turns “one card” into “a resource pool you can share, isolate, and schedule.”

How It Differs from the AI Mini-PC Crowd

There’s no shortage of things calling themselves AI workstations, but most are just beefier mini PCs. Olares is different. It’s genuinely designed as a cloud.

The company positions this machine outright as a 24/7 personal AI cloud, not a PC you sit in front of. That single framing explains almost everything that follows: why it has to use Kubernetes, why it cares so much about network traversal and remote access, why GPU governance suddenly matters here.

Two other details tell you a lot: it supports Thunderbolt 5 external eGPUs, and two machines can cluster up. In other words, it was never a sealed box. It starts as a single node and grows upward.

Stuffing a Whole Cloud Runtime into One Box

Architecturally, what Olares does is compress an entire cloud runtime into a single machine: auth and authorization, app lifecycle, tunnels and traversal, secrets, observability middleware, not a layer skipped.

This isn’t “a few AI apps preinstalled.” It’s a complete cloud runtime stuffed into a desktop device.

The diagram below is my own re-drawn layering based on its public docs, with the HAMi layer pulled out explicitly because it’s the point of this whole piece.

Figure 2: Olares system layered architecture, with HAMi as the GPU resource plane

One Detail That Caught My Eye

One thing jumped out while I was reading: Olares’s docs haven’t kept up with its own product.

Its 2025 architecture page still says nvshare, noting that GPUs are limited to one card per node. But flip through the release notes and 1.12 already integrates the HAMi scheduler, with exclusive, time-slicing, and memory-slicing modes all there; 1.12.2 adds multi-GPU; 1.12.5 supports DGX Spark outright and rolls automatic scheduling across all three modes.

The product is outrunning its docs. That alone tells you something: it’s shifting from a “personal cloud OS” toward a “local AI cloud OS.” That’s also why I wanted to write a whole piece on it.

What HAMi Actually Does in There

HAMi is a CNCF Sandbox project, positioned as a heterogeneous AI compute virtualization middleware. On the path there’s a Webhook, a scheduler, a Device Plugin, and HAMi-core handling in-container resource control. I summed it up in one line in Kubernetes Is Becoming the GPU Control Plane of the AI Era: it turns GPU slicing from “a hardware capability” into “a control-plane capability.”

On Olares, that’s the real thing behind the GPU mode toggle in settings. On the surface it’s a UI option; underneath, it’s a policy switch on the resource plane.

The most critical piece is almost certainly HAMi-core, which in one sentence: intercepts CUDA calls inside the container and does memory virtualization, compute throttling, and utilization monitoring. It doesn’t slice with hardware. It manages with software.

The diagram below puts that injection path and the three GPU modes side by side.

Figure 3: HAMi-core injection path and Olares’s three GPU modes

The evidence lines up too. In Olares’s GitHub issues you can see HAMi’s libvgpu.so crashing under WSL2, the discussion mentions it getting injected into every process via /etc/ld.so.preload, and the Olares team itself says this implementation “drew heavy inspiration” from the HAMi project.

One thing I should be clear about, though: whether Olares uses upstream HAMi-core directly or maintains a forked version, there’s no single official answer in public. I won’t fill that in for them.

The three modes make sense in order. Full-card exclusive is for heavy loads; time-slicing lets lightweight services take turns, and Olares even swaps inactive models into memory first, with roughly 5% switching overhead; memory-slicing cuts the memory into fixed quotas so multiple apps run together. On something like DGX Spark, where CPU and GPU share memory, it defaults to memory-slicing because there’s no traditional memory paging in and out to begin with.

Why This Matters for HAMi

HAMi used to live mostly in big clusters: multi-tenant, inference serving, mixed training-and-inference, heterogeneous cards. NIO running it for sharing across 80 nodes and 600 cards is the textbook example. That line is well-trodden.

What feels new about Olares is that it moves the same problem onto a single machine.

Figure 4: HAMi’s value narrative shifts from cluster efficiency to edge product experience

What you actually run at once on Olares is never just one Ollama: a local model, a chat UI, a research agent, an image or video pipeline, plus a pile of OCR, speech-to-text, and automation tools. In a scenario like that, without a GPU scheduling layer, the GPU always degenerates into “whoever starts first grabs it.”

So for HAMi this matters. It goes from “a tool that saves cards for platform engineers” to “something that directly decides whether an ordinary user has a good time.” In the cluster it manages utilization and cost; on Olares it manages concurrent experience, model switching, and whether this machine is actually pleasant to use. HAMi doesn’t only talk to platform engineers anymore.

Where Edge AI Goes from Here

My sense is, edge AI won’t settle at “a stronger local model box.” It’ll grow into a full stack of “control plane plus model plane plus application plane.”

NVIDIA pushing DGX Spark and RTX Spark onto the desktop has already moved the story from “run models locally” to “run agents locally.” Olares’s CUDA-plus-x86 is one path, DGX Spark’s unified memory is another, and below that there’s the DIY mini-PC-plus-eGPU route you assemble yourself. Interestingly, Olares lists eGPU and dual-node as supported paths itself, so the line between appliance and DIY isn’t that sharp.

But my guess is, the one that breaks out won’t be the one with the most ferocious specs. It’ll be the one that fuses resource governance, dev experience, app ecosystem, security, and remote access into one closed loop. If desktop AI workstations really become a category, what people compete on isn’t GPU and memory, it’s the control plane.

I went ahead and added Olares to the AI Native Landscape I maintain, so I can keep watching how it grows.

Conclusion

In one line: Olares is a desktop AI OS with Kubernetes at its core, and HAMi is the layer that turns its single-machine GPU into a shareable, isolatable, schedulable resource pool.

From cluster to desktop, HAMi’s story is expanding from “saving cards” to “making edge AI usable.” If desktop AI workstations become a real category, what people compete on is the control plane, not single-card performance.

References

Every Nation Begins with Textiles

Jimmy Song — Sat, 20 Jun 2026 05:43:26 +0000

In a bamboo-weaving workshop in Anji, holding a strip of bamboo split hair-thin, I realized for the first time: a bolt of cloth, a machine, a country, all begin with this single thread in the hand.

Figure 1: Bamboo strip

Introduction

This is something I’d been thinking about for a long time, without ever quite sorting it out.

In February this year, our company offsite took us to Anji, in Zhejiang province. It’s famous bamboo country in China, the mountains covered in moso bamboo, and the locals have been weaving with it for generations. Bamboo weaving is a national intangible cultural heritage. The offsite included a hands-on session for us to try it ourselves.

I hadn’t expected much from this kind of “experiential activity.” But once I actually sat down, picked up a bamboo strip, and listened to the old master explain how to lift, press, and raise the threads, my mind started to wander. Up and down, lift and press, the warp and weft interlacing in his hands until a pattern slowly emerged. The motion was deeply repetitive, a fixed rhythm, a clear logic.

And in that moment an almost absurd thought popped into my head: this is just 0 and 1.

A warp thread raised is 1, pressed down is 0. Unfold a piece of cloth and it’s a two-dimensional structure made of 0s and 1s. And the thing the old master kept referring to as the “flower program” (花本), the pre-designed sequence of warp-lifting, is essentially a program: follow it, and you weave exactly the pattern you intended.

Figure 2: A woven bamboo horse I finished in Anji, mounted for framing

That thought flashed by and I didn’t make much of it. But then in March I went to the Netherlands and spent a week in Amsterdam. The two experiences sat together, and some feelings that had been vague began to sharpen: how could a single strip of bamboo, a single motion, connect to the looms of two centuries ago, to today’s computers, even to a country’s industrialization?

I looked it up later and found I wasn’t the first to make the connection. Many historians of technology argue that weaving is one of the earliest forms of large-scale information encoding. Now, in June, I’m sitting down to try to lay out this thread. This isn’t a technical piece. It’s a personal, cultural reflection.

A Cultural Contrast

Let me start with the Netherlands.

What struck me most about that country wasn’t the windmills or the tulips, but a quality that the whole nation seems to exude: restraint, order, and an almost engineering-like way of being. The canals are dug, the land reclaimed from the sea is called a “polder,” and the windmills were never there to look pretty. They were built to pump water and drain the land. A country that sits below sea level literally engineered itself out of the sea. Though the windmills aren’t only about drainage. The wind in the Netherlands is genuinely strong. In March I got battered by it. It’s a wind that pushes at you constantly. So a below-sea-level country hauled itself out of the sea with engineering, and along the way put that fierce wind to work too.

Figure 3: A canal and church in central Amsterdam

Figure 4: Windmills at Zaanse Schans

This quality seeps into every corner of daily life. Shops close around six in the evening, the streets slowly quiet down, and you almost never hear a car horn. Cars stop well ahead of time to let you cross. One evening I wandered a little too close to the bike lane, and a passing car actually rolled down its window to remind me to use the sidewalk. It surprised me at first, and then I understood: the rules there are clear, and everyone assumes you’ll follow them too.

This temperament even shows up in how people get around. In the city center, not many people drive. Bicycles are the default. For one thing, the Netherlands is flat, so cycling takes no effort. For another, dedicated bike lanes are everywhere, and the streets are narrow, which makes them ill-suited to cars. Add that commutes are generally short, and cycling is just right. What’s interesting is that even in the southern suburbs, where the roads are wide and every household has a car, you still see a lot of people on bikes. For them, a bicycle isn’t exercise. It’s the default way to commute.

Figure 5: A street in the southern suburbs of Amsterdam: wide roads, cars in every driveway, and still plenty of cyclists

I set that against the noise and speed of China, the loud, garish signboards back home. These two temperaments are really two paths of modernization. The Netherlands is an early, slow-and-steady kind of country, from the Age of Discovery to the world’s first stock exchange, from water engineering to today’s ASML and Philips. It has always moved forward in a “long-term, restrained, systematic” way. China is a late developer chasing from behind, relying on speed, scale, and a generation’s sheer exertion to cover in a few decades what took others centuries.

Neither path is higher or lower than the other, but they settle into completely different urban textures and national characters. The ease of “closing at six” that the Dutch have is something China won’t reach for many years; and the Chinese speed of “just do it,” of laying a high-speed rail line overnight, is something the Netherlands probably couldn’t match either.

But here’s the interesting part. If you trace both civilizations back through history, these two seemingly different cultures share a common starting point. Textiles.

Why Textiles Were the Starting Point of Industrialization

Our generation is so familiar with the phrase “Industrial Revolution” that it’s almost gone numb. But have you ever asked: where did the first Industrial Revolution actually break out? Not in steel, not in coal, not in the railways. In textiles.

The British Industrial Revolution is practically a history of textile machinery. The flying shuttle in 1733 made weaving faster. The spinning jenny in 1764 let one person spin many threads at once. The water frame in 1769 began replacing human power with water power. The power loom in 1785 turned weaving into an automated process. Every one of these inventions happened inside the textile industry.

Why textiles of all things? At first it feels counterintuitive, since textiles seem too “light,” lacking the heft of making steel or guns. But think about it for a moment and it becomes obvious.

Textiles are a basic need. Everyone has to wear clothes, clothes wear out, and they have to be replaced constantly. That’s an enormous market that already existed back in the agrarian age. Once the demand is there, any gain in efficiency turns immediately into profit, into capital that can be reinvested.

More importantly, the processes of textile production are especially easy to mechanize. Spinning and weaving are deeply repetitive motions with a fixed rhythm and a clear logic, so machines can directly replace the human hand. Steelmaking, by contrast, has a far higher technical barrier, the science of chemistry wasn’t yet mature, and machine manufacturing itself needed an existing industrial base before it could even start.

And textile production has a moderate investment threshold. Building a textile mill was far cheaper than building a steel plant, so merchant capital and money from colonial trade flowed in easily. A large share of early British industrial capital came from the cotton trade and colonial trade, and that money went first into textile mills.

The most crucial point is that textiles don’t exist in isolation. They pull an entire supply chain along with them. To spin and weave, you first have to grow and transport cotton. To run machines, you have to build textile machinery. To power the machines, you need steam engines. To fuel steam engines, you have to mine coal. To move cotton and cloth, you have to build railways. So the steam engine was first deployed at scale in textile mills, the earliest railways carried cotton and cloth, and the earliest machine manufacturing served textile equipment. A supposedly “light” industry ended up dragging the entire mechanical and energy industries into being.

So textiles became the progenitor of industry not because they were the most complex, but because they were the first to satisfy four conditions at once: large demand, mechanizable, low threshold, and able to pull others along. Textiles were humanity’s first trial plot on the way from handicraft into an industrial system.

Late Developers All Come Up This Same Road

Here’s a pattern I’d never noticed before: the industrialization of late developers, almost without exception, begins with textiles.

Take Japan. Today when you think of Japanese industry, you think of Toyota cars. But how did Toyota begin? Its founder, Sakichi Toyoda, didn’t start by making cars. He invented automatic looms. In 1924 he invented the Type G automatic loom, and in 1926 he founded Toyoda Automatic Loom Works, the origin of the entire Toyota Group. It was his son Kiichiro who later moved the business into automobiles.

So Toyota cars quite literally “grew out of a loom.” Its whole system of lean production, kanban management, total quality control, carries in its bones the obsession with precision, process, and zero defects that came from making looms.

Then look at South Korea. Samsung today is a global electronics and semiconductor giant, but when it was founded it exported dried fish, vegetables, and fruit, only later moving into textiles and sugar. As South Korea industrialized after the war, textiles and garments were among its earliest foreign-exchange earners, and Samsung built its first fortune on trade and textiles in that period before pivoting to electronics in the 1970s. The Hyundai Group took a similar path, starting in engineering and heavy industry before expanding into automobiles, shipbuilding, and heavy industry.

Britain was first; Japan and Korea are the late-developer cases. You’ll notice the sequence is strikingly consistent across all three: textiles first, then light industry, then machinery, then heavy industry, and finally high-tech. Nobody designed this path. It was forced by cost and by the market. Because textiles are the lowest-threshold, largest-market form of manufacturing, every nation that wants to turn from an agrarian country into an industrial one has to start from the easiest, most necessary step.

Behind this lies a very plain truth: modernization doesn’t happen in one leap. It needs a starting point that can earn money, accumulate experience, and train the first generation of industrial workers. And textiles happen to be exactly that kind of starting point.

Where China Sits on This Road

Writing this far, I can’t help but turn and look back at China itself.

China’s modern industrialization also began with textiles. The most famous Chinese-owned enterprises of the late Qing and early Republic were almost all cotton mills. Zhang Jian founded the Dasheng Cotton Mill in Nantong. The Rong brothers, Zongjing and Desheng, founded the Shenxin Cotton Mills in Wuxi. They were China’s first generation of national industrial capitalists, rising out of textiles and holding up half of modern Chinese industry. The thinking of Zhang Jian’s generation was plain and direct: foreigners were using machines to weave cloth and taking our money, so we had to set up our own mills, weave our own cloth, and keep that money at home.

That was China’s first stretch. Back then, China was walking the very road that Britain, Japan, and Korea had all walked.

Then the road broke. War, turmoil, the planned economy. Chinese industrialization took many detours. It wasn’t until reform and opening up that it reconnected. In the 1980s the coast was blanketed with processing factories for garments, shoes, and toys. In essence it was still that same old road of “starting with textiles and light industry,” only this time China relaunched it through exports, cheap labor, and the role of factory to the world.

Further on, through the 1990s and 2000s, China pushed into heavy industry: steel, cement, shipbuilding, chemicals, catching up to the world’s front ranks one after another. Then electronics, home appliances, mobile phones. Then the internet, high-speed rail, new-energy vehicles, solar, semiconductors, and the AI that everyone talks about today.

String this line together and you realize China is actually completing the same road that every late developer has walked, only faster, fiercer, and at a far greater scale. From Zhang Jian’s cotton mills to today’s new-energy vehicles and large AI models is barely over a hundred years. In just over a century we’ve run the entire course of industrialization that took Britain more than two hundred and Japan more than a hundred, and we’re still going.

This often leaves me with a complicated feeling. On one hand, admiration: generation after generation, from Zhang Jian to today’s engineers, really did turn an agrarian country into the factory of the world and then into a country that now leads in quite a few high-tech fields. On the other hand, a faint unease: this road was run so fast that a lot of things got left behind.

A Thread That Never Broke

Inside this long thread of industrialization, there’s one detail I never forgot: that moment in the bamboo-weaving workshop in Anji back in February.

The “flower program” the old master described actually has a very old tradition in China. The drawloom of the Han dynasty used a pre-designed system of cords to control which warp threads rose and fell, so a weaver following it could reproduce complex patterns. Then in the early nineteenth century, the Frenchman Joseph Marie Jacquard pushed the idea a big step forward, using punched cards to control the weaving pattern: where the card had a hole, a hook passed through and lifted the warp; where there was no hole, the hook was blocked and the warp stayed down. Hole or no hole, that’s 1 and 0.

Jacquard’s punched cards were later borrowed by the computing pioneer Charles Babbage for his Analytical Engine, and Ada Lovelace used them to write the first computer program in history. Further on, IBM rose on punched-card machines, and if you trace today’s entire computing industry back to its source, it leads, improbably, to a loom.

And that’s not the end of it. The Transformer, matrix operations, the weights of neural networks in today’s AI, at bottom are all about computing relationships across vast fields of “warp and weft.” A loom decides which warp threads to lift and when; a neural network decides which parameters relate to which others. The logic is the same.

So the word “weaving” is more than the starting point of an industry. It’s a metaphor: for thousands of years, human beings have been learning how to encode complex things into structure, from a bolt of cloth, to a program, to a neural network. Every nation does this. They’re just at different stages.

I plan to write the technical bloodline of this story separately, in another piece. Here I’ll only point to it, to make one thing clear: the thread that began with that strip of bamboo has never broken.

Summary

From a bamboo-weaving session in Anji in February, to the canals and windmills of Amsterdam in March, and on to the industrialization of Britain, Japan, Korea, and China, what I want to say really comes down to one thing.

Every nation’s modernization is a road from a single thread woven into a net. Textiles are the common starting point of that road, not because textiles are so lofty, but because they are the most necessary, the easiest, the quickest to earn that first money and train that first generation of workers. From that starting point, some walk with restraint, like the Netherlands; some move with ferocity, like China. Some take two hundred years, some a hundred, some only a few decades. But the starting point is the same, and so is the direction: from that thread in the hand, step by step, woven into the whole of industrial civilization as we know it today.

The Dutch ease of closing at six, of no honking, of swans gliding through the canals, is a place China hasn’t reached yet. The Chinese speed of just doing it, of rolling something out overnight, is something the Netherlands couldn’t pull off either. Between these two temperaments of modernization there is no higher or lower, only trade-offs. And behind the trade-offs lies the question of where a country sits in its industrialization, and what rhythm it’s willing to pay for that net.

That day in Anji, holding a strip of bamboo, I clumsily followed the old master, lifting and pressing, weaving a small, lopsided patch of pattern. I wasn’t thinking about any of this then. But looking back, from that single lift and press, you can trace upward to the Han dynasty drawloom, outward to a country’s industrialization, and forward, faintly, to today’s computers and AI. A single strip of bamboo, connected to so much.

Perhaps the evolution of civilization, in the end, is just humanity learning, over and over, how to weave one thing into another.

Starting from a single thread.

Why GPUs Became the Foundation of AI: A GPU Primer for K8s Veterans

Jimmy Song — Wed, 17 Jun 2026 14:04:42 +0000

Note

This post builds on Ameen Alam’s three-part GPU Architecture series and draws on TrendForce, NVIDIA (Slinky/Slurm, the GPUDirect data path) and Mirantis material on GPU infrastructure and agentic AI. It’s written for Kubernetes veterans who have never touched a GPU and never trained or served a model.

Figure 1: GPU: the foundation of AI

Why a cloud-native veteran is writing about GPUs

For the past decade my home turf has been containers and Kubernetes. From service mesh to the whole cloud-native ecosystem, I’ve spent almost every day on scheduling, networking, storage and observability, but always on the CPU side of cloud native. Last year I formally moved into AI infrastructure (AI Infra).

Once I dove in, I found the concept density absurd. Token, Transformer, Tensor Core, HBM, KV Cache come at you one after another, and almost every doc and article assumes you already know them, which is deeply unfriendly to engineers who have never trained a model or run inference.

I quickly hit on a trick: don’t learn from scratch, use what you already know by heart as a translator.

The mental model a Kubernetes veteran already carries, scheduling, Jobs, Deployments, controllers, caches, utilization, multi-tenancy, and then microservices, service mesh, distributed systems, event-driven, high-availability, maps onto AI Infra almost one to one. Once you build that mapping, the intimidating concepts become “oh, it’s just the GPU version of X”. This “translate AI through cloud-native eyes” approach got me up to speed fast, and I’m writing it down to help friends with the same background skip the detour.

This is the first post in that translation series, tackling the most fundamental question: why does AI absolutely need GPUs? Later posts will cover GPU resource management, scheduling and observability, topics a K8s veteran knows well. If you’re also crossing over from cloud native, I hope this saves you a few days of digging.

You type a sentence and the AI replies word by word. For someone who has used K8s, the most natural mental model is: AI is a “workload” that runs on GPUs, and a GPU is a special kind of “node” built for it.

Translating the jargon into K8s

The AI world throws around terms that read like scripture to anyone who hasn’t done training or inference. Here’s a cheat sheet; I’ll re-explain each term in plain words below.

AI concept	In plain words	K8s veteran’s analogy
Token	A small unit text is chopped into; AI generates them one at a time	A log line, a text chunk
Model	A “scoring program” holding billions of numbers (weights)	A giant image whose “weights” aren’t code
Training	Feed it huge data, tune params repeatedly, build the model	Run a Job to build an image; offline, batch, care about throughput
Inference	Model is done, serve user requests and emit answers	Run a Deployment taking traffic; online, care about latency
Transformer	The architecture shared by nearly all LLMs (GPT, Claude, LLaMA)	A controller design pattern
Tensor Core	A GPU unit that only does “matrix multiply” but insanely fast	A sidecar worker that only does batched multiply
HBM	The GPU’s on-board high-bandwidth memory; model and cache live here	Node-local RAM, but with bandwidth that crushes normal RAM
KV Cache	Attention state stored at inference time, grows with the conversation	A pod-local session notebook that gets thicker the longer you talk

Table 1: AI jargon → K8s veteran cheat sheet

Memorize this table; the rest follows.

The question most people never ask

GPUs are everywhere in AI now, so accepted that most people skip past it. We care about which card to rent, which framework to use, which model to deploy, but rarely stop to ask the deeper question:

Why GPUs?

Why not the more general CPU? Why did a chip originally built to render game graphics become the foundation of the most important technology shift in a generation?

The answer isn’t just “it’s fast”, it’s about how computation itself is organized, and why AI workloads demand an architecture fundamentally different from fifty years of software.

Two philosophies of computing: the craftsman and the factory

You know CPUs well. K8s’s control plane, etcd, the scheduler all run on CPU; it excels at executing complex instructions one after another, with few cores (8 to 128) but each highly capable. A CPU optimizes for latency: how fast can I finish one complex task? Like a master craftsman doing one intricate job at a time.

A GPU takes the opposite path: execute simple instructions across massive amounts of data at once. A modern data-center GPU has thousands of small cores, and it optimizes for throughput: how many simple tasks can I finish at the same moment? Like a factory floor of thousands of workers, each doing one identical step simultaneously.

Figure 2: The CPU craftsman vs the GPU factory: latency-first vs throughput-first computing

For any single complex job, the craftsman (CPU) is faster; but as long as the work is uniform and parallelizable, the factory (GPU) crushes the craftsman in total output per second. For decades CPU dominated because most software (web servers, databases, operating systems) is inherently serial. Then deep learning arrived.

Why AI broke the CPU

The core of a neural network is, essentially, a giant pile of matrix multiplications (an operation that batch-multiplies-and-adds two sets of numbers). When a model processes a token, it runs thousands or tens of thousands of these multiplies, and they don’t depend on each other.

The key insight in one line: these operations are naturally parallel.

A 128-core CPU looks at this workload and sweeps through it sequentially; a many-thousand-core GPU chops the matrix into blocks and hands them to its small cores to run at once. Same work, orders of magnitude faster on GPU, not because a single GPU core is faster (it’s slower), but because the problem is parallel and the GPU is built for exactly that shape of computation.

That’s the real reason AI runs on GPUs: not marketing, not legacy, architectural fit.

The three things inside a GPU, in K8s terms

1. Tensor Core: the sidecar that only does batched multiply

A GPU has two kinds of cores. Regular CUDA cores are general-purpose workers that can compute anything; a Tensor Core is a dedicated unit that does exactly one thing: multiply two small matrices in a single shot. But that one thing it does insanely fast, finishing in one cycle what would take a regular core thousands.

AI’s core operation is matrix multiplication, so Tensor Cores are tailor-made for it. In K8s terms: regular CUDA cores are like general pods in a Deployment that do everything; a Tensor Core is like a highly specialized sidecar that only does “batched multiply”, single-function but with crushing throughput.

2. HBM: the node’s high-speed local memory

Computing fast isn’t enough; you have to get the data. A GPU’s memory is layered just like a CPU node’s, and if you understand a K8s node’s L1/L2 cache, RAM and local disk, you understand the GPU’s:

Figure 3: GPU memory hierarchy: faster and smaller up top, slower and bigger below; HBM bandwidth is the lifeblood of AI

The bottom layer, HBM (High Bandwidth Memory), is the GPU’s “main memory”; model weights, intermediate results and KV cache all live here. It’s characterized by large capacity (hundreds of GB per card) and extremely high bandwidth.

Why build HBM at all? Because traditional VRAM (GDDR) lies flat on the circuit board, you run out of routing space and hit a bandwidth wall. HBM’s answer is to stack memory chips vertically (connected by Through-Silicon Vias, TSVs), packed right against the GPU die with an interface thousands of lines wide running in parallel. By analogy: normal memory is like a warehouse spread across a parking lot where the movers can’t keep up; HBM is like building that warehouse into a dozens-story tower right next to the workshop, with TSV elevators running up and down at full speed.

Here’s a counter-intuitive fact: modern AI is memory-bound more than compute-bound. Tensor Cores do matrix multiplication faster than HBM can feed them data, so the GPU is often waiting. That’s why each new GPU generation (H200 → Blackwell → Vera Rubin) sees its most important upgrade in HBM bandwidth, not compute (4.8 → 8 → 22 TB/s, source).

Since data movement is the bottleneck, NVIDIA’s other move is to let data bypass the CPU and reach the GPU directly, the three “expressways” collectively called GPUDirect:

GPUDirect Storage: data goes straight from NVMe to GPU memory, without detouring through host memory and the CPU.
GPUDirect RDMA: GPU memory talks directly to the NIC (InfiniBand), so cross-node gradient exchange no longer hops through the CPU.
NVLink: GPUs inside one machine connect directly and share memory.

In one line: the CPU is no longer the traffic cop for every data movement. In K8s terms, it’s like sending data over a direct data plane instead of routing every packet through the apiserver. NVIDIA’s edge isn’t just fast compute; it’s that the entire data highway from storage to GPU, GPU to GPU, and GPU to network has been straightened out.

3. Transformer + Token: the program that emits answers word by word

A Transformer isn’t hardware, it’s a model architecture (a program structure). GPT, Claude, LLaMA and other large models are all built on it. What it does is actually plain:

Read a piece of text, predict the next token.

A token is a small unit text is chopped into (a character, half a word). A Transformer feeds the input token sequence in, emits “the most likely next token”, appends it, feeds the new sequence back in, predicts the next-next, and so on, generating the whole answer one piece at a time.

Figure 4: Token and Transformer: AI plays relay, reading the tokens so far at each step and predicting the next

In K8s terms: a Transformer is a controller whose reconcile logic is “look at the current state (tokens so far) → emit the next action (the next token)”, looping continuously. That’s where the “word by word” effect of AI assistants comes from.

Training vs Inference: writing the recipe vs serving the dish

This is the easiest pair to confuse when starting out, yet the most critical. Training and inference are two different things with completely different hardware needs.

Training: feed huge data, repeatedly tune those billions of parameters, “teach” the model into existence. It runs for weeks to months as an offline batch job, cares about throughput, not per-request latency. In K8s terms, it’s a long-running Job whose goal is to build a model image.
Inference: once the model is trained and online, it takes each user’s input, computes an answer and emits it back. The user is waiting, so it cares about latency; it must serve thousands of users at once, so it cares about concurrency. It’s like a Deployment taking traffic.

Figure 5: Training vs Inference: training is a Job writing the recipe (offline, throughput), inference is a Deployment serving the dish (online, latency)

This directly affects which GPU metrics you should care about. The headline numbers (Tensor Core FLOPS, NVLink bandwidth, cluster size) are mostly training metrics; what actually determines whether your AI assistant feels snappy (HBM capacity, memory bandwidth, KV cache management) are inference metrics, and they get little airtime. Yet inference is where most production AI actually runs.

Multi-GPU collaboration: an AI cluster is just a distributed system

With single-card covered, back to reality: the biggest models don’t fit on one card, and training routinely needs hundreds or thousands of cards working together. At that point the GPU cluster is essentially a distributed system, and almost everything you know from cloud native applies.

Many GPUs = a microservice cluster. A card is like a service instance; the model is sharded across cards, each computes a slice and the results are stitched back. That’s exactly the microservices playbook of splitting by responsibility and sharing load.

Inter-card communication = the service-mesh data plane. Cards constantly exchange data (gradients especially during training). Within a rack it’s NVLink (several TB/s), across racks it’s InfiniBand. It’s just the east-west traffic between microservices: in-node pod-to-pod direct is fastest (NVLink), cross-node cross-cluster goes over the network (IB). NVIDIA packaging GPUs, switches, DPUs and RDMA into a whole rack (Vera Rubin NVL72) is exactly a service mesh unifying the data plane, control plane and observability.

Distributed training’s all-reduce = distributed consensus. Each training step synchronizes and averages the gradients across all cards, a step called all-reduce. Anyone who has done etcd or Raft gets it instantly: it’s distributed-system consensus and consistency, except here you’re syncing gradients, not a state-machine log.

Continuous batching = event-driven. During inference, new requests are dropped into the running batch as they arrive, no waiting for the whole batch to finish. Requests are events, the batcher is a consumer, batch-then-process: that’s the event-driven / message-queue mindset, all to keep the GPU busy.

Disaggregated inference = microservices. Split prefill (process input, compute-heavy) and decode (emit tokens, memory-bandwidth-heavy) into separate GPU pools that scale independently, with KV cache passed between them over NVLink/RDMA. That’s entirely the microservices pattern of splitting by responsibility and scaling each part; KV cache is the session context passed between services.

See the pattern? An AI cluster isn’t a new species, it’s your familiar distributed-systems, microservices and service-mesh toolkit replayed on GPU hardware. The instincts you built tuning traffic on Istio and Envoy, or consensus on etcd, are worth exactly as much here.

Running GPUs on K8s: Slurm for training, K8s for inference

By now, as a K8s veteran, you’re bound to ask: so what actually schedules a GPU cluster? The interesting answer: training and inference often run on two different schedulers.

On the training side, the HPC world’s heavyweight is Slurm. It manages 65% of the world’s TOP500 supercomputers (NVIDIA), and large AI training teams have years invested in Slurm scripts, fair-share policies and accounting. Training is a long-running batch job that wants topology awareness (place chatty GPUs close together), long exclusive holds and high throughput, areas Slurm has refined for over a decade.

On the inference side, the home field is Kubernetes. Inference is an online service that wants fast autoscaling, per-request scheduling and integration with service mesh and observability, exactly K8s’s strengths.

So what if a team needs both, maintain two environments? NVIDIA open-sourced the Slinky project to solve exactly this, in a very K8s-native way:

slurm-operator: turns each Slurm daemon (slurmctld for scheduling, slurmd for compute, slurmdbd for accounting) into a K8s CRD and Pod, with the control plane made highly available through Pod regeneration instead of Slurm’s native HA. Config changes sync automatically via ConfigMap/Secret, workers autoscale with HPA, scale-in drains running jobs first, and upgrades use PodDisruptionBudget to protect in-flight work.
GPU Operator + DCGM Exporter: auto-installs drivers and the device plugin, and can label metrics by Slurm job ID, giving you per-job GPU metrics (scraped by Prometheus).
ComputeDomains + DRA: for cross-node-NVLink machines like GB200 NVL72, K8s uses DRA (Dynamic Resource Allocation) to dynamically manage the cross-node GPU interconnect domain, so distributed training hits full NVLink bandwidth across nodes.

NVIDIA itself runs Slinky in production up to 8,000+ GPUs, with NCCL all-reduce/all-gather performance matching bare Slurm, the K8s layer adding almost no overhead (NVIDIA production data). K8s is becoming the substrate for GPU computing, with Slurm as the scheduling layer on top.

Training and inference have fundamentally different GPU-scheduling needs, so they can’t be managed the same way. That’s also why GPU resource management (covered later) has to be scenario-specific, and why solutions like HAMi aim to be a unified GPU resource-management layer across Slurm + K8s hybrid environments.

The real trap of inference: KV Cache

If there’s one concept that separates “GPU theory” from “AI inference reality”, it’s KV Cache.

When a Transformer generates each token, it has to reference the relationships among all previous tokens (this is “attention”). To avoid recomputing from scratch every time, it stores each token’s attention state, and that store is the KV Cache.

The problem is it grows without bound:

A user sends a 10,000-token conversation; the model stores a state entry for each.
To generate the next token, it reads the entire KV Cache.
A 70B model at 32K context can produce a KV Cache of 8 to 16 GB per request, larger than the model itself.
50 concurrent users land, and KV Cache eats hundreds of GB of memory.

Figure 6: KV Cache is a session notebook that thickens the longer you talk: every new token forces a full read of the whole notebook

In K8s terms: KV Cache is like a session notebook local to a Pod. The longer the conversation, the thicker the notebook, and every new word forces a full read from cover to cover. So inference memory is often eaten not by the model but by this “notebook”.

That’s also why we got paged attention (managing the notebook like virtual-memory pages to cut fragmentation, popularized by vLLM) and NVIDIA Dynamo’s multi-tier cache (hot pages in HBM, warm offloaded to CPU memory, cold spilled to NVMe). Half an inference engineer’s job is managing this notebook.

GPU utilization lies

K8s folks know “utilization lies” best: a pod reporting Running isn’t necessarily doing work, it might be in CPU steal or waiting on IO. GPUs are worse.

That “GPU utilization %” in monitoring usually only measures “is the GPU executing any kernel”, not how efficiently. A card can report 90% utilization while its Tensor Cores are actually busy only 30% of the time, with the rest spent on memory ops, kernel-launch overhead, or just waiting for data.

The more honest metric is SM Efficiency (SM activity rate): it looks at how many SMs are doing useful work each clock cycle. A card showing 100% utilization in nvidia-smi may have an SM Efficiency of only 20-30%. Many companies think their “GPUs are maxed out” when in fact huge amounts of compute are spinning idle. So to judge whether a GPU is truly working, don’t look at utilization, look at SM Efficiency.

The metrics that actually matter (mapping to the QPS, P99 latency and resource levels you watch in K8s):

Token throughput: tokens generated per second per card, how many users you can serve (like QPS).
TTFT (Time To First Token): time from receiving a request to emitting the first token, sets the “responsiveness feel” (like cold-start time).
TPOT (Time Per Output Token, also called ITL / inter-token latency): average time to produce each token, sets streaming smoothness (like P99). Serving frameworks like vLLM and TGI generally use TPOT; NVIDIA more often calls it ITL, same thing.
Memory breakdown: how much is weights vs KV cache vs transient activations, tells you whether you’re memory-bound (like breaking down a pod’s memory usage).

Why NVIDIA is so hard to dislodge

You can’t talk GPUs without NVIDIA’s dominance. The hardware is excellent, but hardware alone can’t explain why AMD, Intel and a crowd of startups have failed to gain ground. The answer is ecosystem depth.

CUDA isn’t just a parallel-computing platform, it’s a programming model, compiler, runtime and a stack of libraries, refined for 18 years. Nearly every AI framework (PyTorch, TensorFlow, JAX) grew up on CUDA first and was ported elsewhere as an afterthought. In K8s terms: CUDA is to GPUs roughly what the Linux kernel + containerd + the whole CNCF toolchain are to the container ecosystem, not something you replace by swapping a kernel.

Switching to another card means re-validating every layer of your inference stack (kernels, libraries, frameworks, serving, monitoring, ops). The switching cost isn’t buying different hardware, it’s rebuilding an entire software ecosystem. That’s the “CUDA moat”.

The future: AI factories

NVIDIA no longer describes its business in terms of “GPUs” or even “data centers”, but as “AI factories”: facilities that continuously convert electricity, silicon and data into intelligence. Beneath the language is a real architectural shift:

Power is now the primary constraint: a single Vera Rubin rack draws over 100 kW (source), and a mid-size training cluster needs 10 to 50 MW, comparable to a small town. The GPU is no longer the hard part; securing reliable power is.
Cooling moves off air: liquid cooling is now standard for high-density GPUs.
Networking decides GPU placement: AI cluster design now starts with network topology and works backward to where GPUs go, the reverse of traditional data centers.
Memory bandwidth remains the scaling frontier: each generation adds compute, but what actually moves the experience is HBM bandwidth.
It’s fundamentally HA + datacenter ops: power redundancy, cooling failover, latency determined by network topology, single-point failure and DR are all old problems for anyone who has done distributed-systems HA. The AI factory isn’t new magic; it’s your HA-architecture skillset moved into a data center an order of magnitude denser in power and compute.

Don’t retire the CPU just yet: agentic AI is rebalancing the CPU:GPU ratio

After all this GPU praise, you might think the CPU is sidelined in the AI era. The opposite is true: agentic AI is putting the CPU back at center stage.

In traditional LLM inference the CPU mostly just compresses and routes data for the GPU, so AI data centers ran CPU:GPU ratios as low as 1:4 or even 1:8. Agents are different: they plan tasks autonomously, call tools, route data between sub-agents and decide whether a task is complete, and all of that orchestration logic lands squarely on the CPU. Add that agents are often trained with reinforcement learning, where every action has to be evaluated by the CPU, and the CPU load gets heavier still.

Research notes that in agentic scenarios the CPU-side tool processing (Python interpretation, web crawling, retrieval, database queries) can account for up to 90% of total latency. According to TrendForce, Arm estimates that traditional AI data centers need about 30 million CPU cores per GW, a figure that will surge to 120 million in the agent era, a 4x increase; the CPU:GPU ratio will shift from 1:4~~1:8 toward **1:1~~1:2**. That’s also why in 2026 NVIDIA started selling the Vera CPU standalone and Arm got into the CPU business directly.

The signal for K8s veterans is clear: the future AI node is a mixed-workload node where CPU and GPU are billed together, and scheduling and resource management must handle both, not just stare at the GPU.

What this means for me (a K8s veteran)

After all this hardware, it lands on what I work on: only by first understanding why the GPU is the foundation of AI can you understand why “GPU resource management” is a real problem.

When a card costs hundreds of thousands of yuan, a whole rack draws over a hundred kilowatts, and production GPUs still run below capacity because memory bandwidth, KV cache or batching aren’t tuned, just “handing out cards” is nowhere near enough. How to run multiple tenants safely on one card (like K8s scheduling many pods onto one node), how to partition resources between the very different workloads of training and inference, how to push utilization from “looks full” to “actually full”, that’s exactly what GPU virtualization and sharing (the work I do) solves.

Here’s a sobering number: per the ClearML AI infrastructure survey, only about 7% of enterprises hit over 85% GPU utilization at peak, more than half sit at 51-70%, and 15% are below 50%. In other words, a big chunk of the GPUs companies pay dearly for are spinning idle. That’s usually not a hardware shortage, it’s scheduling and management not keeping up.

My own take is that GPU resource management is moving through three stages: Allocation → Utilization → Efficiency. Stage one only answers “who gets this card” (device plugin, exclusive or time-slicing); stage two asks “is this card full” (dynamic batching, elastic autoscaling, where most enterprises are stuck); stage three asks “is this card’s compute producing maximum value” (SM Efficiency, tokens per watt, bandwidth utilization). The real challenge is stage three, and the future GPU scheduler shouldn’t be a resource allocator but a fine-grained orchestrator of the GPU’s internals, reading how many SMs are active, the Tensor Core utilization, how much memory KV cache holds, and only then deciding whether one more request fits.

Plainly put, the AI era is replaying the cloud-native story: from “single-machine exclusive” toward “multi-tenant sharing + scheduling + observability.” And the GPU is the main stage of that play.

This is only the first half: who else is at the table besides NVIDIA

By now you’ve probably noticed this post barely mentions anyone but NVIDIA. Unavoidable, it’s the absolute protagonist today. But if you think the AI accelerator world begins and ends with NVIDIA, you’re very wrong.

In fact, an “anti-NVIDIA alliance” is gathering from all sides:

Cloud-vendor silicon: Google’s TPU is on its sixth generation and backs almost all of its own AI; AWS’s Trainium (training) and Inferentia (inference) keep spreading; Meta and Microsoft aren’t sitting still.
Traditional chip giants: AMD presses hard with the Instinct line and ROCm, Intel holds ground with Gaudi and oneAPI.
China’s heterogeneous accelerator ecosystem: Huawei Ascend, Hygon DCU, Cambricon, Moore Threads, Enflame, Kunlunxin, Metax, Biren…, sprinting through the domestic-substitution window, with their software stacks climbing from “works” toward “works well”.
The open ecosystem fights back: the UXL Alliance, OpenAI Triton, and PyTorch’s native AMD/TPU backends are all gnawing at the walls of the CUDA moat.

What does that mean for a K8s veteran? It means the future AI cluster is almost certainly heterogeneous: a rack might hold NVIDIA, AMD, TPU and domestic cards side by side, and one schedule has to manage several completely different kinds of hardware.

So the really interesting questions follow:

How do you mix multiple accelerator types in one cluster and still schedule and observe them uniformly?
Where do K8s device plugins and DRA (Dynamic Resource Allocation) evolve to, so they can elegantly describe this menagerie of hardware?
Will the CUDA moat be breached by the open ecosystem, or will NVIDIA rule long-term like x86 did?
What pieces are still missing before domestic heterogeneous accelerators are truly production-ready?

I’ll dig into those in later posts. This one only nails “why GPUs”; the next will tackle the big chess game of how AI infrastructure should schedule things once “GPUs aren’t just one kind”.

References

Ameen Alam, Why GPUs Became the Foundation of Modern AI (Part 1 of the trilogy)
Ameen Alam, The Hidden Technology Behind Modern AI GPUs (Part 2)
Ameen Alam, Real AI Infrastructure and the Future of GPUs (Part 3)
TrendForce, The Great Rebalance: How Agentic AI Is Reshaping the CPU/GPU Ratio
Anton Polyakov (NVIDIA), Running Large-Scale GPU Workloads on Kubernetes with Slurm (Slinky / Slurm on K8s)
Kawonise, Why NVIDIA GPU Architecture Is Perfect for AI: GPU Data Path for a Single Node (GPUDirect data path)
Mirantis, GPU Infrastructure Automation and Strategy (GPU infra automation and utilization)

GPU Utilization Is Breaking: AI Infrastructure Needs a New Definition of Efficiency

Jimmy Song — Wed, 17 Jun 2026 06:22:24 +0000

Every GPU should not just be used. It should create value.

We have all been chasing GPU utilization

For the past few years, whether it is Kubernetes GPU scheduling, vGPU, MIG, or HAMi, everyone has really been doing the same thing: pushing one number up.

GPU Utilization

It makes sense. GPUs are expensive. An H100 can run anywhere from a few dollars to over ten dollars per GPU-hour, and nobody can afford to let a GPU sit idle. So the entire AI Infra community’s narrative for the past few years has boiled down to one line:

Maximize GPU utilization.

I have been involved in the HAMi community for a long time, and a few posts I have written, like Kubernetes as the GPU Control Plane for AI and From GPU to Token: An Eight-Layer Observability Stack for AI Infrastructure, are really about the same thing: how to slice GPUs finer, share them more thoroughly, and schedule them more sensibly.

But recently I read Arjun Kaarat’s piece in Towards Data Science, When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI, and it made me rethink a question:

Is GPU utilization really the metric we should be optimizing for?

There is one line in the article that made me pause for a few seconds the first time I read it:

“GPUs can be busy without being productive.”

Arjun Kaarat

It punctures a layer of illusion. The number we have spent so much effort pushing up may have been answering the wrong question from the start.

GPU Busy is not GPU Productive

Kaarat tells a representative story in the article.

At 2 AM, an infrastructure team gets paged: inference latency just spiked 60%. They open the monitoring dashboard, and GPU utilization looks perfectly normal:

GPU: 79%
GPU: 82%
GPU: 84%

Looks healthy. So the usual playbook kicks in: trigger autoscaling, add nodes, add GPUs. The cloud bill climbs, but latency barely improves.

An hour later, they find the root cause: three nodes had quietly entered a RAID rebuild state, storage throughput was severely dragged down, and the inference tasks around them were starving. The scheduler kept treating these nodes as “still healthy enough” because the GPU and memory metrics looked fine, but the underlying disk performance had collapsed.

What strikes me most about this story is that it is not a rare edge case. It is a failure mode that is becoming common.

Many teams look at their monitoring and see:

GPU: 82%
GPU: 84%
GPU: 79%

and think “our cluster is busy and healthy.” But at the same time:

Latency ↑
Queue ↑
Throughput ↓
Cost ↑

The real problem is that the GPU is waiting.

Waiting for what?

The retrieval pipeline to feed embeddings over;
The SSD to read context out of storage;
The CPU to prepare the data pipeline;
The KV cache to have room for new requests;
Storage I/O to not be squeezed out by background tasks.

The GPU end is full, but the data path feeding it is empty, blocked, or collapsing. From the dashboard the GPU reads 84%, but the actual output may be less than half. Kaarat describes this state precisely in the original piece:

A GPU that appears active may still spend meaningful time waiting for the system around it.

That is the overlooked gap between “busy” and “productive”.

The “illusion” of GPU utilization

The RAID story above is still just a “point failure”. The more compelling part of Kaarat’s article describes a systemic phenomenon: Fragmentation.

Consider a cluster with three nodes, after running a mixed wave of GenAI workloads:

Node	GPU compute	HBM	Storage bandwidth	I/O CPU
A	available	nearly full	available	available
B	available	available	saturated	available
C	limited	available	available	saturated

Table 1: Residual resources on three nodes after a GenAI wave

Now a new inference job arrives, with an ordinary footprint: a little GPU, a little VRAM, decent storage bandwidth, decent I/O capacity.

In total, the cluster still has plenty of resources. A has GPU and bandwidth, B has VRAM, C has bandwidth and CPU. But no single node can take this job on its own.

That is fragmentation. I drew it out, roughly like this:

Figure 1: The cluster is not short on resources. It is short on resources of the right shape.

The cluster is not empty. It has just been carved into “leftovers” that can no longer be used productively. Kaarat sums up the phenomenon in one line, which I think is the most memorable sentence in the whole piece:

The cluster is not empty. It is fragmented into leftovers that are difficult to use productively.

Put another way:

The cluster does not lack resources. It lacks resources of the right shape.

This judgment matters a lot for a project like HAMi, which builds a GPU resource control plane. We used to think that “slicing the card finely and letting more people share it” was the answer to fragmentation. But Kaarat points to a deeper problem: fragmentation is not just a GPU-layer concern. It spans GPU, HBM, storage bandwidth, and I/O CPU. You can slice the GPU as finely as you like, but if the storage dimension is choked, that node is still unavailable for the next genuinely useful task.

What HAMi solves

Let me directly answer a question: what role does HAMi play in this chain?

HAMi solves a very specific, and very foundational, problem:

Can the GPU be used by more people.

What it does can be summarized in one line: reduce fragmentation at the GPU layer. The concrete forms include:

GPU Sharing: letting multiple Pods share one card instead of one card per Pod;
vGPU / HAMi-core: doing memory isolation and compute throttling in userspace, slicing one card into MB-level virtual devices;
MIG integration: managing NVIDIA MIG hardware partitions in software;
Heterogeneous GPU abstraction: abstracting more than a dozen device families, including NVIDIA, Ascend, Cambricon, Hygon, and Vastai, into semantics the scheduler can consume uniformly;
DRA compatibility: keeping pace with the evolution of the Kubernetes resource model.

This is the first layer of efficiency. It answers the question “can the GPU be put to use”.

At this layer, HAMi’s value is already clear: in real environments with mixed domestic and foreign GPUs, mixed training and inference, and multi-tenant sharing, HAMi lets one card serve more workloads, so the cluster is no longer wasted by a coarse-grained “one card per Pod” model.

But note: this is only the first chapter of the efficiency story.

What comes after HAMi

If I step back and look at it from a higher vantage point, “improving GPU utilization” is actually solved across three distinct layers.

Layer one: can the GPU be sliced, shared, and allocated?

This is what a project like HAMi solves. It corresponds to the GPU resource control plane.

Layer two: who gets to use the GPU? Who runs first, who queues, who has priority?

This is what schedulers like Volcano, Kueue, and KAI Scheduler solve. It corresponds to job queuing, fair share, priority, and gang scheduling.

Layer three: can the GPU actually run? Are the data path, storage I/O, and KV cache keeping up?

This is exactly where Kaarat’s article sounds the alarm. When a node’s RAID is rebuilding, its SSD queue is exploding, and its I/O CPU is eaten by background tasks, no matter how much GPU you allocate to it and no matter how elegantly the scheduler queues its tasks, it still cannot produce effective compute.

Draw these three layers together, and you get what next-generation AI infrastructure should actually look like:

Figure 2: From GPU utilization to productive GPU-hours

I think this diagram is the single most important one for understanding the whole picture.

For the past few years, almost all of the industry’s attention has been on the bottom two layers: Kubernetes and HAMi. Those two layers have essentially solved “can the GPU be put to use”. Volcano, Kueue, and KAI are also mature at layer two, solving queuing and priority.

But layer three, storage / I/O-aware scheduling, is currently almost a blank. And that is precisely the layer Kaarat’s article keeps emphasizing, the one that is becoming increasingly valuable in modern GenAI systems. Because for workloads like RAG, long context, and multimodal, the bottleneck has long since shifted from “is the GPU enough” to “is the data path feeding the GPU clear”.

To put it bluntly: HAMi slices the GPU finely, and Volcano queues the jobs well, but if a node assigned a task cannot feed its GPU, then all that upstream effort is just pumping blood into an idle endpoint.

What should next-gen AI infrastructure optimize for

Based on the layering above, I want to make one clear point: we should upgrade the optimization target from “GPU utilization” to “Productive GPU-Hours”.

This is not wordplay. It is a shift that translates directly into money.

Kaarat does the math in the article. A 1000-H100 cluster, at a blended cost of about $3 per GPU-hour, runs around $26 million a year. If fragmentation and I/O stall quietly waste 10% of the effective GPU time, that is roughly $2.6 million a year of wasted spend. Not because the GPUs are missing, but because the system failed to use them efficiently.

That math can be translated into a simple contrast.

The past target:

Maximize GPU Utilization

Today’s target:

Maximize Productive GPU-Hours

The future target:

Maximize Productive Compute
Across Heterogeneous AI Clusters

This evolution maps exactly onto the three-layer structure in the diagram above. That final line, “Across Heterogeneous AI Clusters”, is the heterogeneous narrative HAMi has been pushing all along: in the future you will not optimize just one kind of GPU. You will maximize effective compute uniformly across completely different cards from NVIDIA, Ascend, Cambricon, and Hygon.

In other words, HAMi’s long-term value should not be boxed into the old “improve GPU utilization” narrative. Its real direction is: a resource control plane that lets Productive GPU-Hours be maximized across heterogeneous AI clusters.

From utilization to productive GPU-hours

If I had to summarize this whole line of thinking in one sentence, I would put it like this:

HAMi solves “how to let more GPUs be used”, and next-generation AI infrastructure has to solve “how to let every GPU actually create value”.

That is also the deepest line Kaarat’s article left me with:

The real question is no longer “Are the GPUs busy?”

It is: “Are they productively busy?”

It also makes me rethink what the HAMi community’s tagline for the next phase should be. We used to say “let GPUs be shared by more people”. Next, we should probably move toward:

Turn GPU utilization into productive GPU-hours.

Let every GPU not just be used, but genuinely create value.

GPU utilization was never the endpoint. It was only the first chapter of the efficiency story. Only when we start talking about Productive GPU-Hours, and start caring about storage, I/O, the data path, and the aggregate output of heterogeneous clusters, does AI infrastructure truly enter its second chapter.

Acknowledgments and references

Parts of this post were inspired by Arjun Kaarat’s piece in Towards Data Science, When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI, and quote its published paper and article. My thanks to the author. Both diagrams in this post (resource fragmentation, the efficiency layering) were redrawn by the author based on the ideas in the original, not copied from it.

Arjun Kaarat. When GPU Utilization Lies: The Hidden Systems Problem Slowing Modern AI. Towards Data Science, 2026.
Kaarat, A., Batthula, V. J. R., & Segall, R. Fitting the Void: Residual-Aware Geometric Packing for GenAI Workloads. IEEE, 2025.

When an Agent Becomes a Distributed State Machine: Agentic AI Infrastructure Reliability

Jimmy Song — Tue, 16 Jun 2026 12:45:16 +0000

Statement

This is my reading and critique of the paper “AI Infrastructure Reliability Features and Architecture for Agentic AI” (June 2026), shared by Hesham ElBakoury in the Open Compute Project (OCP) community. It blends my personal engineering perspective from the Kubernetes / AI Infra space, and is not a translation of the original paper.

An Agent is not a single inference request but a long-running distributed state machine; therefore, Agent reliability is fundamentally a distributed-systems problem.

Why I wanted to write about this paper

After years in cloud native, I have a stubborn instinct: whether a new technology is mature is not judged by how powerful its model is or how flashy the demo looks, but by whether it has a systematic language for reliability. Kubernetes won not on the container runtime, but on liveness/readiness probes, the reconciler loop, PDs/PDBs, and the whole SRE vocabulary that made “long-running distributed state” legible.

So when I came across the paper Hesham ElBakoury shared in the Open Compute Project (OCP) community, it caught my eye. It proposes no new algorithm or framework; the entire paper does one thing: systematically translate the traditional SRE reliability vocabulary onto Agentic AI. This is exactly what I have been circling around for the past six months in Agentic Runtime Realism and Ark Agentic Runtime, Analyzed, without a canonical reference to align against. So I decided to write a dedicated post: unpack its framework, then give it a critical evaluation from the AI Infra practitioner’s point of view.

One-line positioning: it reads more like an SRE white paper for Agentic AI than a systems paper. Its value is not in novelty but in “building consensus”.

One-line summary

Traditional AI cares about model accuracy, while Agentic AI must care about “reliability during long-running operation”, so fault tolerance, recovery, monitoring, security, and state management must be elevated to first-class citizens of architectural design.

The fundamental split between Traditional AI and Agentic AI

The paper argues the fundamental difference between Traditional AI and Agentic AI lies in the execution model. Traditional AI is one-shot request-response, focused on accuracy, latency, and throughput; Agentic AI is a continuous loop, where a single error is no longer just a wrong answer but a wrong decision that may change every subsequent action of the Agent.

Figure 1: Traditional AI’s request-response vs Agentic AI’s perceive→think→act→observe loop

The comparison table below states this most clearly. I suggest you focus on the last row, “failure impact”, because that is the thesis of the whole paper:

Aspect	Traditional AI	Agentic AI
Execution model	Request-response	Continuous loop
State management	Stateless	Stateful
Trigger	Human-triggered	Self-initiated
Time span	Single interaction	Long-running session
Failure impact	Single wrong output	Cascading behavior changes
Resource usage	Bursty	Continuous

Table 1: Execution model: Traditional AI vs Agentic AI

The point of this table is not the technical details but the shift in mental model: when the impact of failure escalates from “one wrong answer” to “behavior-level cascading errors”, the weight of reliability must be reassigned from the very bottom of the architecture. Engineering fault tolerance for a stateless API and for a self-directing, continuously running state machine are simply not the same problem.

The core contribution: a five-dimension Agent reliability framework

This is the most memorable part of the paper. The author decomposes Agent reliability into five dimensions, forming a complete evaluation coordinate system. I drew it out: “Agent Reliability” sits in the center, with the five dimensions fanning out like petals:

Figure 2: The five-dimension Agent reliability framework: the first four describe the Agent’s own behavior, the fifth describes the platform that runs it

One by one:

Functional Reliability: can the Agent complete the task? Concerns correctness, accuracy, consistency, completeness. Plainly: can this Agent get the job done?
Temporal Reliability: can the Agent stay stable over time? Concerns timeliness, responsiveness, stability, durability. Plainly: can it keep getting the job done?
Environmental Reliability: is it still effective when the environment changes? Concerns adaptability, robustness, portability. Plainly: can it still work in a different environment?
Social Reliability: this is an Agent-unique dimension. Concerns safety, trustworthiness, collaborativity, explainability. Plainly: can it collaborate safely with humans and other Agents?
Systemic Reliability: the infrastructure layer. Concerns availability, fault tolerance, scalability, security. Plainly: is the platform underneath the Agent solid?

Why this framework matters

The real value of these five dimensions is that they turn “Agent reliability” from a vague concept into a decomposable, measurable, separately-owned engineering problem. The first four describe behavioral attributes of the Agent itself; the fifth describes the platform that carries it, and that fifth dimension is exactly the landing point those of us doing AI Infra / Kubernetes should pick up.

Treat the Agent as a “stateful service”

The paper’s second core judgment is one I strongly agree with: the biggest infrastructure challenge for an Agent is not inference, it is state.

An Agent simultaneously holds Memory, Goal, Plan, Context, and tool state. Treating it as HTTP API + Model to operate is the root cause of why most demos today cannot survive production. Its true shape is closer to a composite of database + workflow engine + LLM + distributed system:

Figure 3: Wrong ops mindset (left) vs right ops mindset (right): an Agent is fundamentally a stateful distributed system

This is entirely consistent with where the industry is heading: LangGraph, Temporal, and the OpenAI Responses API are all promoting “stateful, long-running logic” to a first-class citizen. When an Agent runs for hours or even days, its state is more critical than the model parameters themselves, and far easier to lose for good on a restart. A model can be reloaded, but a three-hour planning context, once lost, is usually lost.

Fault tolerance: the “golden trio” of Agent systems

The paper provides a catalog of Agent fault-tolerance patterns:

Mechanism	Effect	Complexity
Redundancy	Standby Agent takes over at any time	High
Checkpointing	Save state for recovery	Medium
Heartbeat	Liveness detection	Low
Circuit Breaker	Isolate abnormal Agents, prevent cascading	Low
Rollback	Revert a wrong decision	Medium
Quorum	Multiple Agents vote to reach consensus	High
Self-Healing	Automatically detect and correct errors	High

Table 2: Agent fault-tolerance pattern catalog: mechanism, effect, and implementation complexity

The author values the Checkpointing + Redundancy + Heartbeat trio most, calling it the “golden trio” of Agent systems. I drew a closed loop to show how they relate; none of the three can be missing, and drop any one link and the recovery chain is broken:

Figure 4: The Agent fault-tolerance golden trio: heartbeat detects the issue → checkpointing saves the scene → redundancy switches the instance

The closed-loop logic of the golden trio

Heartbeat detects problems fast, checkpointing saves recoverable state, and redundancy takes over seamlessly on failure. Together they form the closed loop of “detect the problem → save the scene → switch the instance”. This is also why these mechanisms all sound like old friends from distributed systems: because Agent reliability is, by nature, a distributed-systems problem.

Recovery: porting traditional DR thinking onto the Agent

This section essentially ports the RTO/RPO thinking of traditional DR (disaster recovery) onto the Agent scenario. The paper compares several recovery approaches:

Recovery approach	RTO	RPO	Complexity
Cold Start	minutes to hours	High (total loss)	Low
Warm Start	seconds to minutes	Medium	Medium
Hot Start	seconds	Low (minimal loss)	High
Checkpoint Recovery	seconds to minutes	Medium (since last checkpoint)	Medium
State Reconstruction	minutes to hours	Low (fully recoverable)	High

Table 3: Agent recovery approaches compared: RTO, RPO, and implementation complexity

The author’s conclusion: the Agent fits Checkpoint Recovery best. The reason was foreshadowed above: an Agent’s state is far more important than its model parameters. Checkpoint Recovery strikes the most balanced trade-off among RTO, RPO, and implementation complexity. That is also why checkpointing sits at the center of the “golden trio” earlier.

Agent observability = infrastructure metrics + behavior analysis

This section resonates with me the most. The paper argues that traditional monitoring is far from enough: traditional monitoring watches CPU, memory, latency, and QPS, while an Agent must additionally monitor the behavior itself.

Agent Health: heartbeat, resource utilization, error rate, decision latency.
Behavioral: action frequency, decision patterns, state transitions, goal progress.
System: Agent count, communication volume, infrastructure health.

The “behavior” layer is unique to Agents.

The easiest trap to fall into

An Agent’s CPU may be low and its latency normal, but if its “action frequency” suddenly spikes or its “decision pattern” deviates from baseline, that is often the truly dangerous signal. In other words: Agent Observability = Infra Metrics + Behavior Analytics. Watching only infrastructure metrics means you will perfectly miss the Agent’s “behavioral runaway”.

This lines up completely with the thinking I discussed in GPU to Token Observability: in the Agent era, behavioral signals must become first-class citizens of observability, rather than staying stuck at the hardware / inference-metric layer.

The recommended hybrid architecture

The paper compares four architectures:

Architecture	Complexity	Scalability	Fault isolation	Suitability for Agents
Monolithic	Low	Limited	Poor	Poor
Layered	Medium	Medium	Medium	Good
Microservices	High	High	Excellent	Excellent
Event-driven	High	High	Good	Excellent

Table 4: Four architectures compared for Agent systems

The final recommendation: production-grade Agent systems adopt a hybrid architecture, the combination of “layered + microservices + event-driven”:

Figure 5: The hybrid architecture for production-grade Agent systems: layered + microservices + event-driven

Layering provides clear separation of concerns, microservices provide independent scaling and fault isolation, and event-driven provides loosely-coupled async collaboration. No single architecture can satisfy an Agent system’s demands for clarity, elasticity, and collaboration at once, so hybrid is the inevitable conclusion. For K8s veterans this layered stack should look very familiar: it is the cloud-native layering of governance, ported verbatim onto the Agent.

The biggest value of this paper, from the AI Infra perspective

From the Kubernetes / AI Infra standpoint, what this paper really conveys is a paradigm shift: Agent infrastructure will move from “Serving” to “Runtime”.

Figure 6: Paradigm shift: from model Serving to Agent Runtime, the focus shifts fully rightward

The past cared about GPU utilization, throughput, and latency; the future cares about state recovery, long-task continuity, Agent isolation, Agent scheduling, Agent observability, and Agent security governance.

My read on the trend

The next phase of AI Infra is not better model Serving, but a more reliable Agent Runtime. This is exactly the direction I have kept emphasizing in Ark Agentic Runtime, Analyzed and Agentic Runtime Realism: the Agent is moving from “a class you import” to “a workload you must govern”.

My evaluation

Strengths:

Proposes a fairly complete Agent reliability framework (the five-dimension model), turning a vague concept into a measurable engineering problem.
Systematically ports traditional SRE thinking onto the Agent scenario, with clear mappings for fault tolerance, recovery, and monitoring.
Offers strong engineering guidance on architecture selection (layered / microservices / event-driven / hybrid).

Weaknesses:

Lacks real production cases: the three cases in the paper (autonomous vehicle fleets, supply-chain optimization, customer-service bots) read more as illustrative scenarios than verifiable engineering practice.
The math models (series/parallel reliability, Markov chains, MTBF/MTTR) are essentially standard reliability-engineering textbook material, not innovation.
It never touches real Agent Runtime implementations like Kubernetes, Ray, Temporal, or LangGraph, so it feels light on engineering grounding.
Its discussion of GPU and inference systems, the core AI Infra resource layer, is shallow, barely staying at the “compute/storage/network” level of abstraction.

My score

By academic novelty: 6.5 / 10. By “giving AI Infra practitioners a mental framework for Agent reliability”: 8 / 10.

Its biggest insight is not some technical detail but that opening line: an Agent is not a single inference request but a long-running distributed state machine.

Summary

The real contribution of this paper is that it re-categorizes the reliability problem of Agentic AI. It is no longer a model problem, nor a prompt-engineering problem, but a distributed-systems problem, and specifically a distributed-systems problem that is stateful, long-running, and self-directing.

For AI Infra practitioners, this means two things. First, the traditional SRE toolbox (checkpointing, redundancy, heartbeat, circuit breaker, RTO/RPO) can be ported over directly, but it must be extended to the “behavior layer”. Second, the focus of infrastructure will shift from “how to serve models faster” to “how to run Agents more reliably”, which is the Agent Runtime.

The AI platforms of the future must not only run fast, they must run stable.

References

From GPU to Token: The 8-Layer Observability Stack for AI Infrastructure

Jimmy Song — Tue, 09 Jun 2026 03:31:16 +0000

GPU utilization is not the destination. Token cost is the true North Star metric for AI infrastructure.

For the past few years, one of the hottest topics in AI infrastructure has been GPU scheduling. Whether it’s Kubernetes, Volcano, Kueue, or HAMi, they all fundamentally solve the same problem — how to make expensive and scarce GPUs more efficiently utilized.

But as more enterprises begin running production-grade Large Language Model (LLM) services, a new phenomenon has emerged: GPU utilization is high, but users still complain about slow response times; GPU clusters are near full capacity, but business throughput hasn’t grown proportionally; VRAM and compute still have headroom, yet TTFT (Time To First Token) continues to degrade. This reveals a fundamental truth — GPU utilization alone is no longer sufficient to describe the real operational state of modern AI systems.

For traditional cloud-native applications, we focus on CPU, memory, network, and disk. For AI systems, however, we also need to care about: whether GPUs are truly performing useful computation, whether NCCL (NVIDIA Collective Communications Library) communication has become a bottleneck, whether Kubernetes has correctly allocated resources, whether KV Cache is exhausted, whether Token latency meets user experience requirements, and whether the cost per Token is reasonable. In other words, the observability target for AI infrastructure has expanded from the GPU to the entire inference chain.

Recently, while participating in the development of an industry standard — “Technical Capability Requirements for Computing Power Efficiency Enhancement: Heterogeneous Computing Services” organized by the China Academy of Information and Communications Technology (CAICT) — a core question came up repeatedly in discussions with experts: How do GPU and Token relate? How do we measure GPU output through Tokens? The standard introduces the concept of “Token as a Service,” shifting the unit of measurement for computing services from traditional GPU hours to Tokens, yet a mature practical framework for building a complete observability chain from hardware to Token was still missing.

Around the same time, I came across a technical article on GPU and LLM observability (GPU & LLM Inference Observability — Layer-by-Layer Coverage) that proposed a constructive approach: decomposing the AI system into eight observability layers from GPU hardware to business cost, which neatly fills the observability gap between GPU and Token. This article reorganizes and interprets the original content, removes product-specific implementation details and vendor promotion, and extends the model by incorporating Kubernetes, HAMi, and modern LLM inference architectures.

This article aims to answer one question:

After we’ve solved GPU scheduling, what should we observe next?

8-Layer Observability Architecture Overview

The diagram below shows the eight observability layers from GPU hardware to business cost, each corresponding to different observability targets and areas of responsibility:

Figure 1: 8-Layer AI Infrastructure Observability Architecture

Original image source GitHub.

Each layer collects data through the OTel Collector and routes it to metrics, tracing, and logging backends, forming a complete observability loop. Let’s examine each layer in detail.

L1 GPU Hardware Layer

This layer focuses on the GPU itself. The core question is: Is the GPU running healthy?

The table below lists the key observability metrics for the GPU hardware layer:

Category	Key Metrics
Compute	GPU Utilization, SM Occupancy, Tensor Core Activity
Memory	VRAM Usage, HBM Bandwidth
Interconnect	NVLink Throughput, PCIe Throughput
Reliability	ECC Error, XID Error
Thermal	Temperature, Power Draw, Throttle Reason

Table 1: L1 GPU Hardware Layer Key Metrics

Many teams only track GPU Utilization, but there’s a critical distinction — GPU Utilization ≠ GPU Efficiency. Two GPUs might both show 90% Utilization: one executing Tensor Core operations, the other waiting for memory access. The actual performance can be vastly different. Therefore, SM (Streaming Multiprocessor) Occupancy and Tensor Core Activity are often more valuable than Utilization alone.

L2 CUDA Runtime and Communication Layer

For distributed training, GPU computation is usually not the bottleneck — communication is. The core question at this layer is: Is the GPU computing, or waiting for other GPUs?

The table below lists the key metrics for the CUDA runtime and communication layer:

Category	Metrics
NCCL	AllReduce, AllGather, ReduceScatter
Communication	Duration, Bandwidth
Kernel	Execution Time, P99 Duration
Straggler	Rank Skew

Table 2: L2 CUDA Runtime and Communication Layer Key Metrics

In large-scale training scenarios, a single slow node can drag down the entire training job. By monitoring NCCL communication Duration and Bandwidth, as well as Skew between Ranks, you can quickly pinpoint communication bottlenecks.

L3 Host / OS Layer

Many GPU problems are ultimately not GPU problems. When GPU utilization is abnormal, the root cause may lie in the host’s CPU, memory, disk, or network.

The table below lists the key metrics at the host level:

Category	Metrics
CPU	Utilization, IO Wait
Memory	Usage, Swap
Disk	Throughput, Latency
Network	Retransmit, Bandwidth
Process	RSS, Thread Count

Table 3: L3 Host / OS Layer Key Metrics

A common misconception: when GPU utilization is low, the first reaction is that the GPU isn’t fast enough. In reality, the GPU is often waiting for data — a slow DataLoader, insufficient CPU, network congestion, or inadequate storage performance can all leave the GPU idle.

L4 Kubernetes and Scheduling Layer

This layer is the core of cloud-native AI infrastructure. The question to answer is: Who owns the GPU? You need to know which Pod is using the GPU, which Namespace consumes the most resources, which team has the highest cost, and which model occupies the most VRAM.

The table below lists the key observability dimensions at the scheduling layer:

Category	Key Attributes
Ownership	Pod, Container
Workload	Deployment, Job
Organization	Namespace, Team
Scheduling	GPU Sharing, GPU Partition
Topology	Node, AZ, Region

Table 4: L4 Kubernetes and Scheduling Layer Key Dimensions

Projects like HAMi, Volcano, and Kueue primarily operate at this layer. HAMi solves core problems like GPU Sharing, GPU Partitioning, and heterogeneous GPU scheduling, while observability answers the question: Has scheduling actually improved resource utilization? The observability data at this layer is the foundation for resource auditing and cost allocation.

L5 Training Runtime Layer

The training phase requires monitoring the model’s runtime state. The table below lists the key metrics for the training runtime:

Category	Metrics
Efficiency	MFU, TFLOPS, Step Time
Gradient	Norm, NaN, Inf
Loss	Training Loss
Data	DataLoader Wait
Checkpoint	Save Time, Restore Time

Table 5: L5 Training Runtime Key Metrics

The most important metric here is MFU (Model FLOPs Utilization). Many training jobs show high GPU utilization but only 30% MFU, meaning a significant amount of GPU time is not being converted into actual training progress. MFU is the golden metric for measuring training efficiency — it directly reflects the ratio of hardware compute converted into effective training computation.

L6 Inference Engine Layer

This is the most critical layer in production environments and the one seeing the fastest growth in industry attention. The table below lists the core metrics for inference engines:

Metric	Description
TTFT	Time To First Token
ITL	Inter Token Latency
Queue Wait	Request queuing time
Throughput	Token throughput
Batch Size	Batch processing efficiency
KV Cache Usage	KV Cache utilization

Table 6: L6 Inference Engine Key Metrics

Many teams still focus solely on GPU Utilization, but the true capacity metric for inference systems is often KV Cache (Key-Value Cache) Utilization.

Why KV Cache Matters More Than GPU Utilization

In modern LLM inference systems, GPU compute usually has headroom, but KV Cache often runs out first. A typical scenario: GPU utilization is only 60%, KV Cache is at 95%, new requests start queuing, and TTFT spikes rapidly.

For inference systems, KV Cache is more akin to a database’s Buffer Pool — it often determines the capacity ceiling of the entire system. When KV Cache approaches saturation, the system is forced to perform Eviction, leading to context loss, request retries, and ultimately latency spikes and throughput drops.

L7 GenAI API Layer

With the development of OpenTelemetry GenAI Semantic Conventions, the industry is converging on unified AI observability standards. The table below lists the key metrics at the API layer:

Metric	Description
Input Tokens	Input token count
Output Tokens	Output token count
Request Duration	Request latency
TTFT	First Token latency
Token Throughput	Token throughput

Table 7: L7 GenAI API Layer Key Metrics

This layer marks the shift from infrastructure-centric to application-centric observability. API-layer metrics directly face end users and business systems, serving as the core data source for SLA/SLO quality assessment.

L8 Business and Cost Layer

Ultimately, enterprises don’t care about GPU utilization — they care about: Is the GPU creating value? The table below lists the core metrics for the business and cost layer:

Metric	Description
Cost per GPU Hour	GPU hourly cost
Cost per Token	Token cost
Cost per Request	Request cost
Idle GPU Cost	Idle resource cost
Tokens per Watt	Inference efficiency

Table 8: L8 Business and Cost Layer Key Metrics

The core competitive metric for future AI infrastructure may not be GPU Utilization, but rather Cost per Useful Token — the comprehensive cost of producing one useful Token. This metric unifies hardware cost, energy consumption, inference efficiency, and business value into a single measurement framework.

Cross-Layer Troubleshooting

Real production issues often span multiple layers. The table below lists common cross-layer failure symptoms and their potential causes:

Symptom	Possible Causes
TTFT Increase	CPU IO Wait, Queue Depth, KV Cache Pressure
Throughput Drop	NCCL, Batch Size, Network
Latency Spike	KV Cache Eviction
OOM	Insufficient VRAM, Oversized Batch
Training Stall	NCCL Straggler

Table 9: Cross-Layer Troubleshooting Reference

Modern AI operations are evolving from single-point monitoring to cross-layer observability. The root cause of a single-layer metric anomaly often lies in another layer. Only by building cross-layer correlation capabilities can teams achieve rapid root-cause identification and precise troubleshooting.

From GPU Control Plane to AI Observability Plane

Over the past few years, the industry has focused primarily on projects like Kubernetes Scheduler, Volcano, Kueue, and HAMi, solving the problem of how to allocate GPUs. In the coming years, the industry will start asking whether GPUs are truly generating value, leading to the formation of a new AI infrastructure technology stack:

GPU Hardware
↓
Kubernetes Control Plane
↓
Inference Runtime
↓
Observability Plane
↓
Optimization Plane

GPU scheduling determines how resources are allocated. Observability determines whether resources are being used correctly. Together, they form the next-generation AI Infrastructure Stack. The evolution from GPU Control Plane to AI Observability Plane marks a new era where AI infrastructure transitions from “resource management” to “value management.”

Summary

AI infrastructure observability is undergoing a fundamental paradigm shift. We used to look only at GPU utilization; now we need to build a complete observability system across eight layers — from GPU hardware, CUDA runtime, host OS, Kubernetes scheduling, training runtime, inference engine, GenAI API, to business cost.

Key takeaways:

L1-L3 focus on hardware and systems: GPU health, CUDA communication efficiency, and host resource adequacy
L4 focuses on resource allocation: Kubernetes scheduling and GPU sharing, with tools like HAMi solving allocation problems
L5-L6 focus on model runtime: Training efficiency (MFU) and inference capacity (KV Cache) are the core metrics
L7-L8 focus on business value: From Token throughput to cost per Token, ultimately measuring whether GPUs are creating value

GPU scheduling is the starting point for AI infrastructure, but not the end goal. The 8-layer observability stack from GPU to Token is the critical closed loop that ensures AI infrastructure truly delivers business value.

References

GPU & LLM Inference Observability — Layer-by-Layer Coverage - GitHub

My Personal AI Stack: Building a Continuously Running Personal AI Infrastructure for ~$100/Month

Jimmy Song — Sun, 07 Jun 2026 16:58:18 +0800

A personal AI infrastructure is not a single tool — it’s a system of long-term synergy.

Figure 17: My Personal AI Stack

Many people talk about AI Agents, Second Brain, Personal Knowledge Management (PKM), and digital avatars.

But over the past year, I’ve come to realize that what I’m actually building is not some AI assistant — it’s a continuously running Personal AI Infrastructure.

It’s not a single product, nor a single model. It’s a set of systems that work together over the long term.

This system helps me think, research, write, code, manage knowledge, process emails, maintain my website, and accumulate long-term memory — every single day.

If I had to summarize it in one sentence:

ChatGPT handles thinking, OpenClaw handles execution, Obsidian handles memory, GitHub handles publishing.

From Tools to Infrastructure

Most AI workflow articles follow a similar structure: start with the model, then the plugins, then the editor.

But I increasingly feel that tools are not the point.

What matters is how these tools work together.

My work spans:

AI infrastructure research
Open source community operations
Technical writing
Developer Relations
Product and ecosystem building

Every day produces a massive amount of information:

ChatGPT conversations
Technical research
GitHub activity
Community discussions
Email newsletters
Hacker News
Discord
WeChat and Lark messages

The problem is never a lack of information — it’s how to organize it.

So I gradually built a Personal AI Stack around my own work.

Workflow Perspective

This is not a single tool, not a single model. It emphasizes continuous synergy across four layers: thinking, memory, execution, and publishing.

Overall Architecture

The architecture diagram below shows how my Personal AI Stack works in layers.

Figure 18: Personal AI Stack Architecture Layers

The core components for each layer are listed below.

Layer	Components
Interface Layer	ChatGPT, Telegram, Discord, Lark, WeChat
Reasoning Layer	ChatGPT, GLM-5.1, Claude Code, Codex
Memory Layer	Obsidian, Markdown, iCloud
Execution Layer	OpenClaw, Gmail, Calendar, Lark CLI
Publishing Layer	GitHub, Hugo, Cloudflare Pages

Table 5: Personal AI Stack Layers and Components

ChatGPT: My Thinking System

Although many workflows revolve around Agents, the tool I use most frequently is actually ChatGPT.

I primarily use it for:

Deep Research
Technical analysis
Architecture discussions
Content planning
Writing assistance
Career decisions

Rather than calling it an assistant, it’s more like:

Research Partner
Technical Advisor
Thinking Companion

Years of accumulated conversations have helped it gradually understand my background, projects, and long-term goals.

Many articles, talks, and technical judgments actually originate from these ongoing conversations.

OpenClaw: My Execution System

OpenClaw is the OpenClaw Agent I deployed on my Mac mini M4 at home.

I mainly interact with OpenClaw through Telegram.

To avoid context mixing, I use separate Telegram groups for different topics:

HAMi
AI Handbook
Personal
Work
Research
Blog

This naturally creates Context Isolation.

OpenClaw handles:

Gmail email management
Apple Calendar scheduling
Scheduled tasks
Obsidian operations
GitHub operations
Website maintenance
Lark knowledge base operations
Automated workflows

It’s more like a Chief of Staff than a chatbot.

OpenClaw’s Positioning

OpenClaw’s value lies in unifying scattered execution entry points into structured workflows, rather than being a simple chat interface.

OpenClaw’s Workflow

OpenClaw’s main path unfolds through the Telegram entry point for multi-platform execution.

Figure 19: OpenClaw Execution Path

Lark: Company Workflow Entry Point

I had barely used Lark before.

After joining my current company, which heavily promotes Lark adoption, all daily workflows, knowledge bases, and collaborative communication happen in Lark.

At first, I just treated it as an enterprise IM tool.

But looking at it now, Lark is more of a company-level workflow entry point:

Group chats
Documents
Knowledge bases
Approvals
Tasks
Automation

For me, what truly changed the experience was Lark CLI.

Through Lark CLI, I can more easily manage the company’s knowledge bases, documents, and some process-driven information.

This turns Lark from merely a chat tool into something that OpenClaw can incorporate into its automation system.

From the perspective of a Personal AI Stack, Lark serves as:

The organizational workflow layer.

It connects information, knowledge, and tasks in a company context.

Obsidian: Working Memory

Obsidian is my most frequently used knowledge tool.

But I don’t consider it my final knowledge base.

For me:

Obsidian
=
Working Memory

This is where I store:

Daily Notes
Weekly Reports
Research Notes
Inbox
Drafts
Fleeting thoughts

Starting three years ago, I developed a habit of consistently writing weekly reports.

These reports record:

Work progress
Learning content
Community activities
Project evolution
Personal reflections

In a sense, weekly reports constitute my daily memory.

And Obsidian is the carrier of these memories.

GitHub and jimmysong.io: Long-term Memory

Many people think of a blog as a content publishing platform.

But for me, jimmysong.io is closer to a long-term memory system.

This website has been continuously maintained for nearly ten years.

What’s recorded here is not just technical articles, but more importantly:

My viewpoints
My judgments
My experiences
My growth trajectory

Unlike Obsidian, content that makes it to the website typically goes through:

Research
↓
Thinking
↓
Validation
↓
Writing
↓
Revision
↓
Publishing

Therefore:

Obsidian
=
Working Memory

jimmysong.io
=
Long-term Memory

The following flow shows the basic path from Obsidian to website publishing.

Figure 20: Long-term Memory Publishing Flow

Information Flow

There are clear boundaries between content sources, curation methods, and output channels. This diagram reveals my information flow logic.

Figure 21: Personal AI Stack Information Flow

GitHub: Execution and Publishing Layer

For me, GitHub is no longer just a code repository.

It also handles:

Blog content
Website source code
Documentation system
AI Handbook
AI Native Landscape

All content eventually makes its way into GitHub.

GitHub Actions automatically handles:

Building
Testing
Publishing

Finally, everything is served through Cloudflare Pages.

The diagram below shows the publishing path from Markdown to static site.

Figure 22: GitHub Publishing Pipeline

My Development Toolchain

I currently use three main AI development tools.

Claude Code

My daily development workhorse.

Despite the name Claude Code, I primarily use Zhipu’s GLM-5.1 model.

It handles:

Coding
Refactoring
Debugging
Documentation maintenance

Codex

Mainly used for:

Project initialization
Large-scale code generation
Automated execution of complex tasks

OpenClaw

Handles automation work beyond development.

The three form a clear division of labor.

The diagram below shows how my toolchain collaborates.

Figure 23: Development Toolchain Collaboration

Why Not Claude?

This is one of the questions I get asked most often.

Objectively speaking, Claude is indeed very strong in code generation and code understanding.

I seriously considered using Claude as my primary model.

But ultimately, I chose not to.

The reason is not about model capability — it’s about overall return on investment.

First, there’s the account issue.

I’ve registered Claude accounts multiple times in the past, and each time I ran into restrictions and bans.

Second, there’s the cost issue.

Because I pay for all my AI tools out of pocket.

So I care more about:

Capability / Cost

Rather than:

Absolute Capability

For enterprise users, Claude Max might be a very reasonable choice.

But for individual paying users, the conclusion may be different.

This article discusses a Personal AI Stack that is entirely self-funded.

If the company reimburses expenses, or if you have an enterprise budget, many choices would change.

Why Not Self-host Large Models?

This is another frequently asked question.

Many people believe:

Buy a GPU + Open-source Model = Free AI

In reality, this is often not the case.

If the goal is simply to get a stable, powerful AI assistant, I lean toward:

Subscription > Self-hosting

The reasons include:

Time Cost

Maintaining a model is work in itself.

Including:

CUDA
Drivers
Inference frameworks
Model upgrades
Networking issues

All of these require time.

And I’d rather spend my time on:

Writing
Community
Product
Technical research

Cost Issue

Currently:

ChatGPT Plus
GLM Coding Plan Max

Total monthly cost is under 600 CNY.

While a high-end GPU:

RTX 5090D
RTX PRO
Enterprise GPU

Typically costs tens of thousands of CNY.

Plus:

Electricity
Depreciation
Maintenance

For my use case, it’s simply not worth it.

Model Upgrade Speed

Cloud models upgrade every month.

Local models require manual follow-up.

For knowledge workers:

Using the latest model is more important than owning a model.

Subscription vs. Self-hosting

For individual users, subscription services typically offer advantages in time cost, upgrade speed, and stability. They let you focus on results rather than infrastructure maintenance.

Daily Tools

Beyond the core systems described above, I also rely on some daily tools to complete the workflow.

Atlas Browser: My primary browser for web reading, research, and information gathering. Noteworthy content is saved to my knowledge base via Obsidian Clipper.
Warp: My most frequently used terminal tool. Its modern interactive experience and AI capabilities make command-line work more efficient.
Typora: A Markdown editor I’ve used for a long time, ideal for immersive writing and long-form editing. Many blog posts and documents are completed here.
CodexBar: Used to monitor usage of ChatGPT, Codex, Claude Code, and other tools. For heavy AI users, token consumption has become a resource metric worth tracking.
Sogou Input Method: My primary voice input tool. Compared to keyboard input, voice better matches my thinking habits, especially when working remotely, writing, and communicating with AI.

These tools are not the core of the system themselves, but they form the foundational experience layer of the entire Personal AI Stack, making information acquisition, content creation, and daily development smoother.

My AI Usage Scale

Current approximate consumption:

GLM-5.1

Weekly:

600 million Tokens

Monthly:

2.4 billion Tokens

Codex

Weekly:

200 million Tokens

Monthly:

800 million Tokens

Total:

Approximately 3.2 billion Tokens per month

That’s roughly 100 million Tokens consumed per day. Through ChatGPT Plus and GLM Coding Plan subscriptions, this is much more cost-effective than paying per token — otherwise, these tokens would cost at least $500 per month.

How Much Does This System Cost Per Month?

The table below compares the fixed monthly costs.

Service	Monthly Cost
ChatGPT Plus	$19.99
GLM Coding Plan Max	¥422.10
iCloud+	$0.99

Table 6: Monthly Cost Breakdown

Approximately:

¥573 / month

OpenClaw runs on my Mac mini M4.

Hardware includes:

Mac mini M4
Samsung 990 Pro 1TB
HAGIBIS Dock

Total investment:

¥4,469

Amortized over four years:

Approximately ¥100 / month

Total Cost

The entire Personal AI Stack’s fixed cost is approximately:

¥700 CNY / month (~$100)

This cost pie chart shows the monthly spending structure of the Personal AI Stack.

Figure 24: Personal AI Stack Monthly Cost Breakdown

FAQ

Is This the Most Powerful Setup?

No.

This is not a “most powerful AI tool configuration guide.”

This is a completely self-funded Personal AI Stack designed for long-term individual use.

If your company reimburses expenses, or if you have a higher budget, you could choose Claude Max, Cursor, more API services, or even a local GPU workstation.

But my goal is not to pursue the absolute best — it’s to achieve stable, sustainable, and cumulative productivity within a personal budget.

Why Not Sync All ChatGPT Conversations to Obsidian?

Because I don’t want to turn Obsidian into a chat log repository.

What I care about more is:

Which content is worth preserving long-term?

Manual saving is itself a curation process.

That’s more important than automatically syncing everything.

Why Not Use Notion?

It’s not because Notion isn’t good.

It’s because I prefer Markdown First.

The benefits of Markdown include:

Local-first
Version controllable
Portable
AI-friendly
Suitable for long-term preservation

Why Use Telegram Only for OpenClaw?

Because Telegram’s Bot ecosystem and API are better suited as an Agent entry point.

WeChat, Lark, and Discord are more for person-to-person communication and community collaboration for me.

Why Emphasize Lark?

Because after joining my current company, I truly started using Lark.

The company’s entire workflow revolves around Lark.

For me, Lark isn’t a personal knowledge base — it’s a company knowledge and collaboration system.

Through Lark CLI, it can further become a workflow entry point that OpenClaw can operate.

Why Not Use a Local AI Workstation?

Because my main work is not training models or running inference services.

My main work is:

Thinking
Research
Writing
Open source community
Software development

Subscription models + a Mac mini is more than enough.

A local AI workstation requires significant investment, complex maintenance, and the model upgrade speed may not keep up with cloud services.

Why Not Buy a High-end NVIDIA GPU?

If the primary purpose is learning CUDA, GPU scheduling, or AI infrastructure, buying a GPU has value.

But if the primary purpose is daily productivity, a high-end GPU isn’t necessarily cost-effective.

For me:

Subscribing to models is more important than owning a GPU.

Workflow Matters More Than Models

My biggest takeaway from the past year is:

Many people believe the core of AI is the model.

But my experience suggests the opposite.

What truly affects productivity is often not the model leaderboard, but workflow design.

A 95-score model in an excellent workflow is usually more valuable than a 100-score model in a chaotic workflow.

In a sense, this is very similar to the evolution of the cloud-native world.

GPUs are important, but scheduling systems are equally important.

Models are important, but workflows are equally important.

Agents are important, but long-term memory and execution systems are equally important.

For me, the ultimate goal of the Personal AI Stack is not to replace people.

It’s to connect thinking, memory, and execution — freeing up more time for what truly matters.

Conclusion

This article defines my Personal AI Stack as a long-term runnable infrastructure, rather than a single tool or a single model.

The core conclusion is: in a self-funded scenario, stable workflows, continuous memory systems, and executable automation are more productive than chasing the most powerful model.

If your goal is to accumulate long-term value, investing time in “how things work together” often yields better returns than investing time in “model rankings.”

Token Is More Than a Billing Unit, It's Becoming the Resource Unit of the AI Era

Jimmy Song — Thu, 04 Jun 2026 06:42:40 +0000

The Linux Foundation recently announced plans to establish the Tokenomics Foundation.

Figure 1: Tokenomics Foundation

When I first saw the news, my immediate reaction wasn’t:

Yet another foundation.

It was:

AI infrastructure is shifting from “managing GPUs” to “managing Tokens.”

That shift may matter more than the foundation itself.

The official rationale from the Linux Foundation is straightforward:

As enterprises begin deploying generative AI and agents at scale, the Token has become the new unit of technology spend. The foundation will partner with the FinOps Foundation to establish Token cost management, benchmarks, open standards, and best practices.

If you stop there, most people would reduce it to:

FinOps for AI

Or:

AI cost management

But I think there’s more to it than that.

The Cloud Era Managed Resources

For the past two decades, the infrastructure industry has been managing resources.

We discussed:

CPU
Memory
Storage
Network
GPU

Kubernetes was no different.

Whether it was the scheduler, autoscaling, or resource quotas, everything fundamentally answered one question:

How do we allocate resources?

That was the central question of the cloud era.

The AI Era Starts Managing Outcomes

With the rise of AI, an interesting shift began.

Enterprises increasingly don’t care about:

How many GPUs were used

They care far more about:

How many Tokens were produced

For most enterprises:

GPU is cost.
Token is output.
GPU, CPU, and memory are merely means of production.
Token is the final deliverable.

J.R. Storment (former Executive Director of the FinOps Foundation) said something on LinkedIn that stuck with me:

Tokens are what all the hardware is being built to produce.

All hardware is ultimately producing Tokens.

From this perspective:

GPU
 ↓
Inference
 ↓
Token

A new value chain is forming.

Token Could Become the New Resource Model

Over the past few years, I’ve been tracking how Kubernetes is evolving in the AI era.

More and more signals suggest:

Kubernetes is evolving from a Compute Control Plane into an AI Control Plane.

Once AI becomes the dominant workload, the resources we manage may no longer be just:

cpu: 8
memory: 32Gi
gpu: 1

They may gradually become:

token
context
latency
throughput

Metrics much closer to business value.

The most interesting thing about the Tokenomics Foundation isn’t whether it will define new standards.

It’s that it implicitly acknowledges one thing:

Token has started to become a resource.

Just like CPU in the cloud era.

Implications for AI Infrastructure

For teams building AI infrastructure, this shift deserves serious thought.

Today, many projects (including HAMi) focus on:

GPU utilization
GPU sharing
GPU scheduling
GPU virtualization

These are all important.

But from an enterprise perspective, they ultimately don’t buy GPU utilization.

They buy:

Cost Per Token

Or even further:

Value Per Token

Given the same pool of GPUs:

Whoever can produce more Tokens,
Whoever can reduce the cost per million Tokens,
Whoever can increase the business value of each Token,

Will capture greater commercial value.

That’s why I believe the significance of the Tokenomics Foundation lies beyond the standards themselves.

It’s driving the entire industry from:

Resource management

Toward:

Value management

My Take

I don’t think the Tokenomics Foundation will become the next Kubernetes.

It’s more like the FinOps Foundation, OpenCost, or OpenTelemetry.

What it’s trying to define isn’t software.

It’s the metering system for the AI era.

The past decade:

CPU was the language of infrastructure

The next decade:

Token may become the language of AI infrastructure

If this trend holds, then the GPUs, inference frameworks, scheduling systems, and Agent Runtimes we discuss today are ultimately just parts of a larger Token economy.

And that may be the real signal the Linux Foundation is sending by launching the Tokenomics Foundation.

AI Native Landscape Launches as a Standalone Site

Jimmy Song — Thu, 04 Jun 2026 06:12:10 +0000

AI Native Landscape now has its own home. 600+ curated open-source projects, a brand-new standalone site, and direct search from your AI coding tools.

How It Started

Last July, I started collecting and organizing AI open-source projects on my personal website. The AI space was moving incredibly fast, with new projects popping up every day. GitHub is a treasure trove, full of tools and frameworks worth studying and using. I wanted to organize them systematically so others (and myself) could find the truly useful ones.

At first, everything lived on jimmysong.io. But as the catalog grew, maintaining it became a pain. The main site already had plenty of content, and AI-related pages mixed in meant that updating a single project required rebuilding the entire site. So this year I decided to spin it out as a standalone open-source project: ai-native-landscape.

All pages previously at jimmysong.io/en/ai/* now redirect to the new site at landscape.jimmysong.io. Existing links and bookmarks still work.

Not Just a List, But a Scored One

There are plenty of AI project directories out there, but most just list a name and a link and call it a day. That’s not enough.

GitHub has thousands upon thousands of projects. You see a name, click through, and find out the last commit was six months ago and nobody’s responding to Issues. You just spent time researching something that’s essentially abandoned.

That’s why AI Native Landscape does something different: every project gets a score.

The scoring is based on four dimensions:

Activity: commit frequency, release cadence, Issue response time
Community: contributor count, PR merge speed, discussion engagement
Quality: Stars, Forks, dependency relationships
Sustainability: maintenance history, team stability

These four dimensions combine into an overall health score. At a glance, you can tell whether a project is worth your time.

And the scores update daily, automatically. Today’s score might differ from tomorrow’s.

The project list itself is also human-curated. Projects that have gone inactive don’t make the cut. I want every project on this list to be “alive,” so the time you spend studying them won’t be wasted.

What’s Covered

The catalog spans eight major areas of AI:

Category	Description
Agent Systems	Frameworks, orchestration, and workflows
Knowledge & Context	RAG, vector databases, document processing, knowledge graphs
Models & Modalities	Foundation models, toolkits, speech and vision generation
Inference & Runtime	Model serving, inference engines, sandboxes, GPU acceleration
Training, Evaluation & Optimization	Frameworks, fine-tuning, benchmarks, observability
Developer Tooling	MCP protocols, coding agents, IDE tools, SDKs
Applications & Experience	Chat interfaces, workflow automation, low-code platforms
Platform & Infrastructure	Cloud-native AI, data platforms, security and operations

Currently 600+ open-source projects, each with bilingual descriptions in English and Chinese.

AI Skill Search

The landscape supports direct search from AI coding tools. No browser needed. Install with one command:

npx landscape-search

Works with Claude Code, Cursor, GitHub Copilot, OpenAI Codex, Cline, Aider, and other popular AI coding tools. After installation, just search in natural language:

“Find me an MCP-compatible agent framework”

No API key required. No local data files. Your AI tool fetches the live project catalog and returns ranked results.

Tech Stack

Built with Astro, deployed on Cloudflare Pages, with TypeScript for type safety. Project data lives in bilingual Markdown files. The build pipeline validates, indexes, and generates OG images in one pass.

Get Involved

GitHub is everyone’s treasure trove, and I hope this project can grow with the community’s help.

Star the repo: github.com/rootsongjc/ai-native-landscape

Submit a project: If you know an AI open-source project worth listing:

Open an Issue on GitHub
Or submit a PR directly, add bilingual Markdown files under data/projects/

See the contributing guide for details.

Links:

Live site: landscape.jimmysong.io
GitHub repo: rootsongjc/ai-native-landscape
Skill install: npx landscape-search

AI Infra Industry Trends: From Compute Bottlenecks to Ecosystem Evolution

Jimmy Song — Sun, 31 May 2026 01:18:27 +0000

This slide deck offers a practitioner’s perspective on the key trends in AI Infrastructure: the core bottleneck has shifted from “compute scarcity” to “efficiency governance”, GPU virtualization and intelligent scheduling deliver the highest ROI, the cloud-native ecosystem is upgrading from “application scheduling” to “compute scheduling”, and Agent workloads will drive a new scheduling paradigm.

Use the controls or keyboard shortcuts below to navigate the embedded interactive slides.

Slide: AI Infra Industry Trends

Key Topics

Evolving AI Infra Bottlenecks — From compute scarcity to efficiency governance
Real Roles of CPU / GPU / Scheduling — How each layer functions in production
Cloud Native × Open-Source Scheduling — The shift from Cloud Native to AI Native
Training / Inference / Agent Compute Demand — Resource characteristics and trend analysis by scenario

About the Speaker

Jimmy Song, CNCF Ambassador and founder of the Cloud Native Community (China). Long-time practitioner in cloud-native infrastructure and AI Infrastructure, focusing on GPU compute scheduling, virtualization, and open-source ecosystem development.

Kubernetes as the GPU Control Plane: HAMi v2.9 and Next-Gen AI Infra

Jimmy Song — Thu, 14 May 2026 06:34:19 +0000

Recently, I’ve been following the progress of domestic GPU scheduling and Kubernetes AI resource models. With the release of HAMi v2.9, I want to share several observations on how the AI Infra control plane is evolving.

Figure 1: Kubernetes as the GPU Control Plane for AI

Why Discuss This Topic Now

When DeepSeek R1 was released in early 2025, most people focused on the fact that it trained a model competitive with OpenAI o1 for just $5.6 million. What struck me, however, was that as inference costs plummeted, GPU utilization issues would quickly come to the forefront.

As models became more useful and inference demand exploded, “one GPU per model” rapidly became a luxury. Meanwhile, the NVIDIA H200 export saga accelerated the adoption of domestic compute. First, a sales ban; then, at the end of 2025, a 25% tariff under Trump; and by January 2026, Chinese customs had cleared zero units. Policy now mandates that over 40% of data center chips must be domestically produced by 2026.

The reality is harsh: not only are GPUs scarce, but you must also learn to use NVIDIA, Ascend, Cambricon, Hygon, and other very different platforms simultaneously.

That’s why I believe HAMi v2.9 is more significant than it appears on the surface.

GPUs Are No Longer Just About the Card

Kubernetes has always managed GPUs in a rather crude way:

resources:
 limits:
 nvidia.com/gpu: 1

This was sufficient in 2019, when the main question was simply whether a Pod needed a GPU or not.

But that’s no longer enough. An inference service might only need 4GB of VRAM, multiple small models can share a single card, training jobs care about GPU topology and interconnect bandwidth, and multi-tenancy requires fault domain isolation. Treating GPUs as integer resources is like using an abacus for statistical analysis—not impossible, but the mental model is out of sync with reality.

The most notable feature in HAMi v2.9 is the HAMi-core mode for the Ascend 910C. Previously, sharing Ascend cards relied on SR-IOV hardware virtualization, which was coarse-grained and inflexible. HAMi-core takes a different approach: it uses LD_PRELOAD to intercept ACL calls in user space, enabling memory isolation at the MB level and compute throttling by percentage.

In short: it’s managed by software, not hardware slicing.

This is reminiscent of how SDN abstracted the network control plane from hardware devices—GPU partitioning is shifting from a hardware capability to a cluster control plane capability. Considering that Huawei shipped 810,000 Ascend 910C cards last year—nearly half of all domestic chips—this capability has significant real-world impact.

DRA: Kubernetes Finally Has a Robust Device Resource Model

Kubernetes v1.34 (September 2025) officially promoted DRA (Dynamic Resource Allocation) to GA, and Red Hat OpenShift 4.21 followed suit. This is a big deal.

The Device Plugin solved “how to connect GPUs to K8s,” but not “how to express complex AI resource requirements.” Device Plugins only know how many cards are on a node, not how much VRAM you need, what topology, or what isolation level.

DRA standardizes device resource declaration, allocation, and management via ResourceClaim and DeviceClass. HAMi-DRA takes a pragmatic approach: it doesn’t require users to change how they declare resources. Instead, it uses a Mutating Webhook to automatically convert existing Device Plugin-style declarations into the DRA model. Legacy systems don’t need to change, but can still leverage new capabilities.

I liken this to what CSI did for storage: it didn’t eliminate vendor differences, but allowed Kubernetes to consume different storage capabilities in a unified way. DRA does the same for AI accelerators—NVIDIA, Ascend, AMD, Vastai cards will never be identical, but the scheduling layer should speak a common language.

A Complete Control Plane Path

If we look beyond individual features and consider HAMi-core, DRA, CDI, and the scheduler together, they actually correspond to different layers of GPU resource management:

HAMi-core: How to partition and isolate devices internally
DRA: How to declare, allocate, and bind resources
CDI: How to standardize device injection into container runtimes
Scheduler/Webhook: How to schedule, admit, and observe

Connecting these layers, from top to bottom, forms the complete Kubernetes GPU Control Plane:

Figure 2: Kubernetes as the GPU Control Plane for AI Workloads

This is a complete control plane path. In v2.9, Volcano vGPU was upgraded to v0.19 with enhanced CDI support. While this may seem like a minor improvement in device injection, it actually completes a critical link in this chain.

Heterogeneity Is the Main Battlefield for Domestic AI Infra

The reality for domestic AI clusters: you can’t build infrastructure around just one type of GPU.

Enterprise environments often have NVIDIA, Ascend, Biren, Cambricon, Hygon, Muxi, Kunlunxin, Vastai, and other devices coexisting. Each card has different drivers, runtimes, virtualization capabilities, and monitoring methods. HAMi v2.9 adds support for Vastai, covering more than ten types of heterogeneous compute devices. Mixed training and inference, online and offline workloads, domestic and overseas GPUs, multi-team and multi-tenant resource pools—in these scenarios, unified scheduling is far more important than single-card performance.

Key Judgments

GPU sharing will shift from a cost-saving measure to a default requirement. After the explosion of inference workloads, not every workload deserves exclusive access to an entire card. Exclusive allocation will increasingly become a luxury.

DRA is the way forward, but migration will be gradual. The Device Plugin ecosystem is too large to disappear overnight. HAMi-DRA’s compatibility layer shows the project team understands this.

Heterogeneous scheduling will become the core challenge for AI Infra. Whoever can abstract different vendor devices into a unified scheduling language will control the key position in the control plane.

Kubernetes will be reshaped by AI workloads. From scheduling semantics to resource models, AI requires much greater expressiveness than traditional web services. DRA, CDI, topology-aware scheduling—these are not isolated evolutions, but all point to one thing: Kubernetes is evolving from a container orchestrator to the control plane for AI computing.

Summary

The significance of HAMi v2.9 is not just in supporting a particular device or partitioning method, but in making one thing clear: the next generation of AI infrastructure competition is not just about model frameworks or GPU counts, but about the control plane.

GPUs are shifting from external devices on nodes to native resources within the Kubernetes control plane. Whoever defines the resource model for the AI era will define the long-term boundaries of AI Infra.

Kubernetes's Anxiety and Rebirth in the AI Wave

Jimmy Song — Fri, 03 Apr 2026 05:20:28 +0000

Kubernetes hasn’t been replaced by AI, but it’s being redefined by it. Anxiety is the prelude to rebirth.

After attending KubeCon EU 2026 in Amsterdam, I’ve been pondering a key question: Kubernetes isn’t obsolete, but it’s no longer “enough”; it hasn’t been replaced by AI, but it’s being redefined by AI.

Figure 1: KubeCon EU 2026 slogan: Keep Cloud Native Moving. This event had over 13,000 registrations, making it the largest KubeCon to date.

This was my third time attending KubeCon in Europe. Over the past few years, you can actually see the community’s mindset shift through the event slogans:

2024 Paris: La vie en Cloud Native

→ Cloud Native has become a “way of life,” the default state
2025 London: No slogan, just the 10th anniversary

→ Kubernetes reached a milestone, focusing on retrospection rather than moving forward
2026 Amsterdam: Keep Cloud Native Moving

→ But the question is: where is it moving?

The absence of a slogan in 2025 was a signal in itself:

When an ecosystem starts commemorating the past instead of defining the future, it’s already at an inflection point.

This article doesn’t recap the talks, but instead distills my observations at KubeCon into insights about Kubernetes’ anxiety and rebirth in the AI wave.

The Root of Anxiety: Is Kubernetes Facing a “Crisis”?

The biggest change at KubeCon was that AI has completely replaced traditional cloud native topics. The focus shifted from service optimization and microservices management to how to deploy and manage AI workloads on Kubernetes, especially inference tasks and GPU scheduling.

Figure 2: Before KubeCon officially started, the Maintainer Summit was all about AI.

Kubernetes, as the foundational infrastructure, was once the core of the cloud native world. With the explosive growth of AI models, the question now is whether Kubernetes can still serve as a “universal” platform for everything, which has become a new source of anxiety.

The AI boom brings real challenges: Can Kubernetes’ “universality” adapt to the complexity of AI workloads?

The Focus Brought by the AI Boom

AI’s popularity has shifted the cloud native spotlight entirely to artificial intelligence. AI coding, OpenClaw, large language models, and generative models have all drawn widespread attention. AI has become the core computing demand in the real world.

This surge in demand raises the question: Can Kubernetes continue to serve as the infrastructure platform for complex tasks? Especially with issues like GPU sharing, inference model scheduling, VRAM allocation, and device attribute selection, is the traditional Kubernetes resource model sufficient?

In the past, Kubernetes handled compute, storage, and networking as foundational infrastructure. But with the rapid development of AI, its “universality” is being challenged. Particularly for inference tasks, Kubernetes’ model appears thin.

Comparing with OpenStack: Will Kubernetes Repeat History?

OpenStack once aimed to be a complete open-source cloud platform, but ultimately failed to sustain growth due to complexity and a lack of flexibility in adapting to new technologies.

Will Kubernetes follow the same path? I believe Kubernetes has different strengths: as a container and microservices orchestration platform, it’s widely adopted and has strong community and vendor support. It doesn’t try to replace all cloud provider capabilities but serves as an infrastructure control plane to help users manage resources.

Figure 3: Cloud native contributors remain active. The crowd at the KubeCon EU 2026 Maintainer Summit shows the community’s vitality.

However, as AI workloads become mainstream, Kubernetes must find a new position to avoid being replaced by “AI-optimized platforms.”

Kubernetes’ Challenge: The GPU Resource Management Gap

At KubeCon, NVIDIA announced the donation of the GPU DRA (Dynamic Resource Allocation) driver to the CNCF, marking the upstreaming of GPU resource management. GPU sharing and scheduling have become urgent issues for Kubernetes.

Traditionally, Kubernetes relied on the Device Plugin model to schedule GPUs, only supporting allocation by device count (e.g., nvidia.com/gpu: 1). But for AI inference tasks, more information is needed for resource scheduling, such as VRAM size, GPU topology, and sharing strategies. NVIDIA DRA makes GPU resource management more flexible and intelligent, gradually easing the “GPU resource crunch” in AI workloads.

This shift means Kubernetes is no longer just a “container orchestration platform,” but is becoming the infrastructure layer for AI-specific resource scheduling.

Against this backdrop, both the community and industry are exploring finer-grained GPU resource abstraction and scheduling mechanisms. For example, the open-source project HAMi is building a GPU resource management layer for AI workloads on top of Kubernetes, supporting GPU sharing, VRAM-level allocation, and heterogeneous device scheduling.

Figure 4: HAMi demo at KubeCon EU 2026 Keynote

These efforts are not about replacing Kubernetes, but about filling the resource model gaps for the AI era. In the long run, this layer may evolve into a “GPU Abstraction Layer” similar to CNI/CSI, becoming a key part of AI-native infrastructure.

The Production “Gap”: Many AI PoCs, Few in Production

A common post-event summary was: Many PoCs, but “everyday production deployments” are still rare. Pulumi summarized it as:

lots of working demos, very few production setups people trust

This shows that while many AI workload solutions succeed in technical demos, the transition from experimentation to production remains difficult. Whether it’s GPU resource sharing or inference request scheduling, whether Kubernetes as the foundation can support this transformation is still an open question.

The Rise of Inference Systems: Kubernetes’ Scheduling Boundaries Are Challenged

Another major event at this KubeCon was llm-d being contributed to the CNCF as a Sandbox project.

If GPU DRA represents the upstreaming of device resource models, then llm-d represents another critical evolution: Distributed LLM inference capabilities are moving from proprietary engineering implementations to standardized, community-driven collaboration in cloud native.

This is significant not just because it’s another open-source project, but because it shows that Kubernetes’ challenges in the AI era are no longer just about “how to schedule GPUs,” but also “how to host inference systems themselves.” As prefill/decode separation, request routing, KV cache management, and throughput optimization move into the infrastructure layer, Kubernetes’ boundaries are being redefined.

Traditionally, the Kubernetes scheduler focused on Pod scheduling. But in AI inference scenarios, scheduling is not just about picking a node—it’s about selecting the most suitable inference instance based on request characteristics. Factors like model state, request queue depth, and cache hit rate all need to be considered. This process is increasingly managed by inference runtimes, forming new “request-level scheduling” systems.

This leads to an overlap between the Kubernetes scheduler and inference systems, forcing Kubernetes to rethink its role: should it keep expanding, or collaborate with inference systems?

AI-Native Infrastructure: The Key Challenge for Production

At the AI Native Summit, the real needs for AI-native infrastructure were especially clear. The focus was no longer “can it run on Kubernetes,” but how to make AI workloads routine, stable, and production-ready on Kubernetes.

Figure 5: At the AI Native Summit after KubeCon, Linux Foundation Chairman Jonathan said cloud native is entering the AI-native era.

The core challenge is delivery. Unlike traditional apps, AI model weights are often huge—tens of GB or even TB—making model delivery and data management extremely complex. Traditional container delivery systems (like image layers) struggle with such massive data and complex versioning.

A key direction for Kubernetes is to standardize model weight and data delivery, using ImageVolume and OCI artifacts to solve AI model delivery and version management on Kubernetes. This not only reduces “cold start” times but also provides infrastructure support for multi-tenancy and compliance.

Summary

Kubernetes won’t be replaced by AI, but it’s being reshaped as the core of infrastructure. This anxiety is the force driving its evolution—it’s moving from a “general-purpose infrastructure platform” to an “AI-powered multifunctional base”. Some even call it the AI operating system.

In the future, Kubernetes’ core competitiveness will no longer be just container management, but how effectively it can schedule and manage AI workloads, and how it can make AI a routine part of operations. This was my biggest takeaway from the AI Native Summit and KubeCon, and it’s what I look forward to in the Kubernetes ecosystem over the next few years.

References

Day One in Amsterdam: Kubernetes Is Rethinking AI

Jimmy Song — Sun, 22 Mar 2026 20:41:19 +0000

Today marks my first day at KubeCon Europe 2026. The most striking feeling is: the world is vast, but this community is truly small.

Figure 11: Jimmy on the first day of KubeCon EU 2026

One strong impression stands out:

The world is big, but this circle is really small.

Old Friends, New Cycle

At the Maintainer Summit, I met many familiar faces—

Colleagues from Ant Group, friends from Tetrate, and some people I’ve known for nearly a decade. Together, we’ve journeyed from the early days of Kubernetes, Service Mesh, and cloud native infrastructure to today.

In a sense, this generation has fully experienced:

The rise of Kubernetes
The standardization of Cloud Native
The microservices and service mesh boom
And now, the era of AI Infrastructure

This isn’t about “new people entering the field,” but rather—

The same group stepping into a new technology cycle.

What Is the Maintainer Summit Discussing?

If you ask:

What is the Kubernetes community most concerned about right now?

Today’s answer is very clear:

👉 How to run AI workloads better on Kubernetes

Figure 12: The Maintainer Summit’s main topic is AI Infra

Many topics at the Maintainer Summit revolved around:

Scheduling models for LLM / AI workloads
GPU / accelerator resource management
Integrating inference systems with Kubernetes
Redefining the roles of data plane vs. control plane
How observability tools like OTel monitor AI workloads

In other words:

Kubernetes hasn’t been replaced by AI; it’s actively “absorbing” AI.

Key Signal: GPUs Are Becoming the “Infrastructure Layer”

Today, I had an in-depth discussion with CNCF TOC, Red Hat, and the vLLM community.

The core question was:

How should GPUs be “platformized”?

Some consensus is already clear:

GPUs are no longer just devices
They are now a schedulable, partitionable, and shareable resource layer

Figure 13: TOC meeting discussing GPU resource management and LLM Serving integration

At the Maintainer Summit in Amsterdam, we had deep discussions with CNCF TOC, Red Hat, and the vLLM community about GPU resource management and LLM Serving integration in Kubernetes scenarios, and explored potential collaboration between vLLM and HAMi.

Behind this is a major paradigm shift:

Past	Now
GPU = Node resource	GPU = Infrastructure layer
Exclusive use	Multi-tenant sharing
Static binding	Dynamic scheduling
Managed within frameworks	Unified management at the platform layer

This is exactly what we’ve been working on in HAMi.

HAMi: From “Project” to “Reference Pattern”

Another interesting change today:

HAMi is no longer just a “community project”—it’s becoming:

A reference implementation (reference pattern) for AI Infra

Figure 14: Li Mengxuan, CTO of Dynamia, sharing HAMi’s design and practice at KubeCon EU 2026 Maintainer Summit

This is reflected in several ways:

Invited to present at the Maintainer Summit
Participating in CNCF TOC discussions
Involved in incubating review demos
Exploring joint content with the vLLM community (even discussing a joint blog 👀)

Especially in conversations with Red Hat and vLLM, a clear trend emerged:

GPU resource management and LLM serving are becoming coupled

That is:

Upper layer: vLLM / inference frameworks
Lower layer: GPU scheduling / sharing

A new “interface layer” is gradually forming.

This is a direction worth betting on.

Figure 15: At the TAG Workshop, HAMi was discussed as an Incubating demo

A Caution: The AI Infra Startup Boom Hasn’t Really Begun

At the same time, I have a somewhat “counterintuitive” observation:

We haven’t yet seen a large wave of AI Infra (K8s-focused) startups.

Most companies I saw today:

Many are pivoting from CI/CD, Service Mesh, or Gateway
Many are traditional cloud vendors extending into AI
Many are working on models, agents, or even lower-level tech

But those truly focused on:

“Making AI workloads run better on Kubernetes”

There are actually not many startups at this layer.

This could mean two things:

1) This Layer Isn’t Fully Formed Yet

Currently, most activity is at:

The model layer (LLM / foundation models)
The application layer (Agent / Copilot)

But not at:

The scheduling layer
The resource layer
The runtime layer

2) Or, the Barrier to Entry Is Very High

Because at its core, this is:

The intersection of Cloud Native × GPU × AI workload

It’s not just “wrapping AI,” but a fundamental re-architecture at the infrastructure level.

My Take

If we break down the AI technology stack:

Agent / Application
 ↓
LLM Serving (vLLM, etc.)
 ↓
AI Runtime / Scheduling
 ↓
GPU Resource Layer
 ↓
Hardware

Most innovation today is concentrated in:

The top two layers (Agent / LLM)

But the real long-term moat lies in:

The middle two layers (Runtime + Resource Layer)

And Kubernetes is very likely to remain:

The default platform for this middle layer

Summary

Today’s takeaway:

Kubernetes is not obsolete; it’s being redefined.

And our generation is shifting from:

“Cloud Native Builders”

to:

“AI Infrastructure Builders”

More to come tomorrow.

HAMi Website Refactor: Why HAMi Docs and Website Underwent a Complete Redesign

Jimmy Song — Tue, 17 Mar 2026 08:55:52 +0800

This redesign is more than a style update—it’s a step toward clearer technical communication and better user experience. Try the new HAMi website at https://project-hami.io and submit issues here.

Over the past two months, I conducted a thorough refactor of the documentation website (see GitHub). Externally, it looks like a “visual redesign”, but from the perspective of community maintainers and content builders, it’s a comprehensive upgrade of information architecture, content system, and frontend experience.

This article aims to systematically explain three things: why we did this refactor, what exactly changed, and what these changes mean for the HAMi community.

Why Refactor the Website and Documentation

HAMi is a CNCF-hosted open source project initiated and contributed by Dynamia, with growing influence in GPU virtualization, heterogeneous compute scheduling, and AI infrastructure. The community content is expanding, and user types are becoming more diverse: from first-time visitors to engineers and enterprise users seeking deployment docs, architecture diagrams, case studies, and ecosystem information.

The original site was functional, but as content grew, several issues became apparent:

The homepage lacked information density, making it hard to quickly grasp the project’s overall value.
Connections between docs, blogs, and community info were not smooth; content entry points were scattered.
Search experience was unstable; external solutions were not ideal in practice.
Mobile experience had many details needing improvement, especially navigation, card layouts, and footer areas.
Visual style was inconsistent, making it hard to convey community influence and engineering maturity.

For a fast-evolving open source community, the website is not just a “place for docs”, but the public interface of the community. It needs to serve as project introduction, knowledge gateway, adoption proof, community connector, and brand expression.

So the goal of this refactor was clear: not just superficial beautification, but to truly upgrade the website into HAMi’s systematic community entry point.

What Was Done in This Refactor

This update was not a single-point change, but a series of systematic improvements.

Homepage Redesign and Complete Information Architecture Overhaul

The most obvious change is the homepage.

We redesigned the homepage structure, moving away from simply stacking content blocks, and instead organizing the page around the main narrative: “Project Positioning → Core Capabilities → Ecosystem Entry → Content Accumulation → Community Trust”.

Specifically, the homepage received several key upgrades:

Rebuilt the Hero section to strengthen first-screen information delivery and action entry.
Optimized CTA design so users can quickly access docs, blogs, and resources.
Added and enhanced multiple homepage sections to showcase project value and community reach in a more structured way.
Adjusted visual hierarchy, background atmosphere, and scroll rhythm, transforming the homepage from a “content list” into a “narrative page”.

These changes include Hero animations and atmosphere layers, research/story sections, new resource entry sections, refreshed CTAs, unified background design, and ongoing reduction of visual noise. Together, they solve a core problem: enabling visitors to understand what HAMi is and why it’s worth exploring further within seconds.

Architecture Diagrams

Key diagrams were redrawn for clearer technical communication. This helps users grasp HAMi’s role in AI infrastructure.

Figure 1: HAMi website homepage architecture diagram

For HAMi, this change is critical. The community faces not just a single feature, but a set of system-level challenges involving Kubernetes, schedulers, GPU Operators, heterogeneous devices, and enterprise platforms. Improved diagrams make the website a better technical entry point.

Added Case Studies, Community, and Ecosystem Sections to Make Impact Visible

Another important direction was strengthening the “community proof” layer.

Many open source project sites fall into the trap of having complete docs, but users can’t tell if the project is truly adopted, if the community is active, or if the ecosystem is expanding. The HAMi website redesign consciously addresses this.

Figure 2: HAMi ecosystem and device support

Figure 3: HAMi adopters

Figure 4: HAMi contributor organizations

Blog & Reading Experience

Blog cards, lists, and metadata were unified for easier reading and sharing. Blogs are now a core communication layer.

Figure 5: HAMi website blog list page

Mobile Optimization

Navigation, card layouts, footer, and search were improved for smoother mobile browsing.

Figure 6: HAMi website mobile view

Footer layout was enhanced for better navigation and credibility. Built-in search replaced unreliable external solutions, improving content accessibility.

Figure 7: HAMi website footer

Figure 8: HAMi website built-in search

What This Redesign Means for the HAMi Community

From screenshots, it looks like “the website looks better”. But from a community-building perspective, its significance is deeper.

First, HAMi’s external expression is more systematic.

The website is no longer just a collection of scattered pages, but is forming a complete narrative chain: users can understand project value from the homepage, capability details from docs, practical paths from blogs, and community impact from ecosystem modules.

Second, community content assets are reorganized.

Previously, valuable articles, diagrams, and explanations existed but were hard to find. Now, through homepage sections, navigation, and search refactor, these contents are more effectively connected.

Third, HAMi’s community image is more mature.

A mature open source project needs not just an active code repository, but clear, stable, and sustainable website expression. Structure, style, and usability are part of the community’s engineering capability.

Fourth, this lays the foundation for expanding case studies, adopters, contributors, and ecosystem content.

With the framework sorted, adding more case studies, collaboration entry points, or showcasing more adopters and partners will be more natural and easier for users to understand.

As a Community Contributor, My Top Three Takeaways from This Redesign

In summary, I believe this refactor got three things right:

Upgraded the website from a “content dump” to a “community gateway”.
Combined visual optimization with information architecture adjustment, not just a skin change.
Improved basic experiences like search, mobile, navigation, and footer.

These may not be as flashy as launching a new feature, but they directly impact content dissemination, user comprehension, and the project’s long-term image.

For infrastructure projects like HAMi, technical capability is fundamental, but clearly communicating, organizing, and continuously presenting that capability is also a form of infrastructure.

Summary

This HAMi documentation and website refactor is essentially an upgrade to the community’s “expression layer” infrastructure.

It improves visual and reading experience, reorganizes content, homepage narrative, search paths, mobile access, and community signal display. Homepage redesign, architecture diagram redraw, unified blog style, mobile optimization, enhanced footer, and switching from external to built-in search together constitute a true “refactor”.

Externally, it helps more people quickly understand HAMi; internally, it provides a stable platform for the community to accumulate case studies, expand the ecosystem, and serve adopters and contributors.

The website is not an accessory to the open source community, but part of its long-term influence. HAMi’s redesign is about taking this seriously.

If you’re interested in Kubernetes GPU virtualization, add me on WeChat jimmysong or scan the QR code below.

Check out the HAMi project on GitHub

GTC 2026 Eve: AI is Becoming the New Infrastructure

Jimmy Song — Sun, 15 Mar 2026 11:34:06 +0800

AI is quietly reshaping the infrastructure landscape, and GTC 2026 may become a key node in this transformation.

Next week, one of the most important technology conferences in the AI industry, NVIDIA GTC 2026, will be held in San Jose, USA.

For many people, GTC is just a GPU technology conference. But if you follow the development of the AI industry over the past few years, you’ll find an interesting phenomenon:

Many important narratives about AI infrastructure are gradually taking shape at GTC.

From CUDA, DGX, to AI Factory, and most recently Jensen Huang’s proposed AI Five-Layer Cake, NVIDIA is constantly attempting to redefine the computing infrastructure of the AI era.

This is why many people call GTC:

AI’s “Woodstock.”

Figure 1: NVIDIA GTC Conference

This year’s GTC (March 16-19) is expected to cover various levels of the AI stack, including:

AI Chips
AI Data Centers
AI Agents
Robotics
Inference Computing

According to NVIDIA’s official blog, this year’s keynote will focus on the complete AI stack from chips to applications.

If we put these signals together, we can actually see a larger trend:

AI is transforming from an “applied technology” into “infrastructure.”

The Perspective of Industrial Revolutions

From a longer time scale, the technological revolutions in human history are essentially infrastructure revolutions.

We usually divide industrial revolutions into four times.

In the table below, you can see the infrastructure corresponding to each industrial revolution:

Industrial Revolution	Infrastructure
Steam Revolution	Steam Engine
Electrical Revolution	Power Grid
Digital Revolution	Computer
Internet Era	Network

Table 1: Industrial Revolutions and Corresponding Infrastructure

First Industrial Revolution: Steam

The steam engine allowed humans to utilize mechanical power on a large scale for the first time. Production no longer relied on human or animal power, but on machines.

Second Industrial Revolution: Electricity

Electricity changed not only the source of power, but also the organization of production. Assembly lines, large-scale manufacturing, and modern industrial systems are all built on the foundation of the power grid.

Third Industrial Revolution: Computers

Computers allowed information to be processed digitally. Software became a production tool.

Fourth Industrial Revolution: Internet and Intelligence

The internet connects all computers together. Cloud computing transforms computing resources into infrastructure. And AI gives machines a certain degree of “cognitive ability.”

The True Significance of AI

If we observe these industrial revolutions, we discover a pattern:

Each industrial revolution produces a new General Purpose Infrastructure.

And AI is likely to become the next-generation infrastructure.

NVIDIA even directly stated in a recent article:

AI is essential infrastructure, like electricity and the internet.

In other words:

AI is no longer just an applied technology, but a new factor of production.

NVIDIA’s Five-Layer Cake

Recently, Jensen Huang proposed a very interesting concept: AI Five-Layer Cake.

Figure 2: AI Five Layer Cake (Image source: NVIDIA)

AI is broken down into five layers:

Energy
Chips
AI Infrastructure
Models
Applications

This model actually illustrates one thing:

AI is a complete industrial system.

Jensen Huang even described AI at Davos as:

“One of the largest-scale infrastructure constructions in human history.”

Signals GTC 2026 May Release

This year’s GTC is expected to release several important directions.

Inference Computing

The focus of AI in the past was training. But the main load of AI in the future is likely to be Inference.

Analysts expect that by 2030, 75% of computing demand in the AI data center market will come from inference.

Agentic AI

The past AI model was:

User → Model → Answer

The Agent model is more complex:

User → Agent → Tools → Model → Action

The flowchart below shows the main interaction paths in the Agent model:

Figure 3: Agentic AI Interaction Flow

AI is no longer just answering questions, but executing tasks.

Agent Platform

Recent media reports suggest that NVIDIA may launch a new Agent platform: NemoClaw, aimed at helping enterprises deploy AI Agents.

If this project is truly released, it means NVIDIA’s stack will become the following structure:

Figure 4: NVIDIA Agent Platform Architecture

This is actually a complete AI stack.

Agents Change Computing Workloads

The emergence of Agents brings new computing workload issues.

Past AI workloads were mainly:

Training
Inference

But Agents bring a third type of workload:

Agent Workloads

The figure below shows the diverse workload types related to Agents:

Figure 5: Agent Workloads Structure

The characteristic of this workload is highly fragmented. GPUs are no longer occupied for long periods, but rather face many small requests. This poses new challenges for infrastructure.

AI-Native Infrastructure

For the past few years, I’ve been thinking about a question:

What is AI-native infrastructure?

It is clearly not just “Kubernetes with GPUs.” I’m more inclined to believe it needs to possess several characteristics.

GPU as a First-Class Resource

In the cloud computing era, CPU is the core resource. In the AI era, GPU is the core resource.

Heterogeneous Computing

Real-world AI chips are not limited to NVIDIA:

NVIDIA
Ascend
Cambricon
Metax
Moore Threads

Future AI infrastructure must be able to manage heterogeneous computing.

GPU is a very expensive resource. If it cannot be shared, utilization will be very low. This is why GPU virtualization and slicing are becoming increasingly important.

AI Scheduling

AI scheduling includes not only traditional CPU and Memory, but also:

GPU
VRAM
Topology
Bandwidth

A Possible AI Tech Stack

Combining the above trends, the future AI stack may present the following structure:

Figure 6: AI Tech Stack Evolution

This structure is very close to NVIDIA’s Five-Layer Cake.

My Judgment

Combining signals from GTC, AI Factory, Agents, and AI Five-Layer Cake, we can see a very obvious trend:

AI is rewriting computing infrastructure.

Future competition may not just be “who has the best model,” but:

Who has the best AI Infrastructure.

Just like the past few decades:

Electricity determines industrial capability
Internet determines information capability
Cloud computing determines software capability

The future may be:

AI Infrastructure determines intelligence capability.

Summary

If we stretch the time scale a bit longer, we may be in a new historical stage.

AI is no longer just a technological tool. It is becoming new infrastructure.

Just like:

Electricity
Internet
Cloud computing

And AI-native infrastructure is likely to become one of the most important technology directions for the next decade.

When GPUs Move Toward Open Scheduling: Structural Shifts in AI Native Infrastructure

Jimmy Song — Fri, 13 Feb 2026 14:32:46 +0000

The future of GPU scheduling isn’t about whose implementation is more “black-box”—it’s about who can standardize device resource contracts into something governable.

Figure 1: GPU Open Scheduling

Introduction

Have you ever wondered: why are GPUs so expensive, yet overall utilization often hovers around 10–20%?

Figure 2: GPU Utilization Problem: Expensive GPUs with only 10-20% utilization

This isn’t a problem you solve with “better scheduling algorithms.” It’s a structural problem - GPU scheduling is undergoing a shift from “proprietary implementation” to “open scheduling,” similar to how networking converged on CNI and storage converged on CSI.

In the HAMi 2025 Annual Review, we noted: “HAMi 2025 is no longer just about GPU sharing tools—it’s a more structural signal: GPUs are moving toward open scheduling.”

By 2025, the signals of this shift became visible: Kubernetes Dynamic Resource Allocation (DRA) graduated to GA and became enabled by default, NVIDIA GPU Operator started defaulting to CDI (Container Device Interface), and HAMi’s production-grade case studies under CNCF are moving “GPU sharing” from experimental capability to operational excellence.

This post analyzes this structural shift from an AI Native Infrastructure perspective, and what it means for Dynamia and the industry.

Why “Open Scheduling” Matters

In multi-cloud and hybrid cloud environments, GPU model diversity significantly amplifies operational costs. One large internet company’s platform spans H200/H100/A100/V100/4090 GPUs across five clusters. If you can only allocate “whole GPUs,” resource misalignment becomes inevitable.

“Open scheduling” isn’t a slogan—it’s a set of engineering contracts being solidified into the mainstream stack.

Standardized Resource Expression

Before: GPUs were extended resources. The scheduler didn’t understand if they represented memory, compute, or device types.

Figure 3: Open Scheduling Standardization Evolution

Now: Kubernetes DRA provides objects like DeviceClass, ResourceClaim, and ResourceSlice. This lets drivers and cluster administrators define device categories and selection logic (including CEL-based selectors), while Kubernetes handles the full loop: match devices → bind claims → place Pods onto nodes with access to allocated devices.

Even more importantly, Kubernetes 1.34 stated that core APIs in the resource.k8s.io group graduated to GA, DRA became stable and enabled by default, and the community committed to avoiding breaking changes going forward. This means the ecosystem can invest with confidence in a stable, standard API.

Standardized Device Injection

Before: Device injection relied on vendor-specific hooks and runtime class patterns.

Now: The Container Device Interface (CDI) abstracts device injection into an open specification. NVIDIA’s Container Toolkit explicitly describes CDI as an open specification for container runtimes, and NVIDIA GPU Operator 25.10.0 defaults to enabling CDI on install/upgrade—directly leveraging runtime-native CDI support (containerd, CRI-O, etc.) for GPU injection.

This means “devices into containers” is also moving toward replaceable, standardized interfaces.

On this standardization path, HAMi’s role needs redefinition: it’s not about replacing Kubernetes—it’s about turning GPU virtualization and slicing into a declarative, schedulable, governable data plane.

Data Plane Perspective

HAMi’s core contribution expands the allocatable unit from “whole GPU integers” to finer-grained shares (memory and compute), forming a complete allocation chain:

Device discovery: Identify available GPU devices and models
Scheduling placement: Use Scheduler Extender to make native schedulers “understand” vGPU resource models (Filter/Score/Bind phases)
In-container enforcement: Inject share constraints into container runtime
Metric export: Provide observable metrics for utilization, isolation, and more

This transforms “sharing” from ad-hoc “it runs” experimentation into engineering capability that can be declared in YAML, scheduled by policy, and validated by metrics.

Scheduling Mechanism: Enhancement, Not Replacement

HAMi’s scheduling doesn’t replace Kubernetes—it uses a Scheduler Extender pattern to let the native scheduler understand vGPU resource models:

Filter: Filter nodes based on memory, compute, device type, topology, and other constraints
Score: Apply configurable policies like binpack, spread, topology-aware scoring
Bind: Complete final device-to-Pod binding

This architecture positions HAMi naturally as an execution layer under higher-level “AI control planes” (queuing, quotas, priorities)—working alongside Volcano, Kueue, Koordinator, and others.

Figure 4: HAMi Scheduling Architecture (Filter → Score → Bind)

CNCF public case studies provide concrete answers: in a hybrid, multi-cloud platform built on Kubernetes and HAMi, 10,000+ Pods run concurrently, and GPU utilization improves from 13% to 37% (nearly 3×).

Figure 5: CNCF Production Case Studies: Ke Holdings 13%→37%, DaoCloud 80%+ utilization, SF Technology 57% savings

Here are highlights from several cases:

Case Study 1: Ke Holdings (February 5, 2026)

Environment: 5 clusters spanning public and private clouds
GPU models: H200/H100/A100/V100/4090 and more
Architecture: Separate “GPU clusters” for large training tasks (dedicated allocation) vs “vGPU clusters” with HAMi fine-grained memory slicing for high-density inference
Concurrent scale: 10,000+ Pods
Outcome: Overall GPU utilization improved from 13% to 37% (nearly 3×)

Case Study 2: DaoCloud (December 2, 2025)

Hard constraints: Must remain cloud-native, vendor-agnostic, and compatible with CNCF toolchain
Adoption outcomes:
- Average GPU utilization: 80%+
- GPU-related operating cost reduction: 20–30%
- Coverage: 10+ data centers, 10,000+ GPUs
Explicit benefit: Unified abstraction layer across NVIDIA and domestic GPUs, reducing vendor dependency

Case Study 3: Prep EDU (August 20, 2025)

Negative experience: Isolation failures in other GPU-sharing approaches caused memory conflicts and instability
Positive outcome: HAMi’s vGPU scheduling, GPU type/UUID targeting, and compatibility with NVIDIA GPU Operator and RKE2 became decisive factors for production adoption
Environment: Heterogeneous RTX 4070/4090 cluster

Case Study 4: SF Technology (September 18, 2025)

Project: EffectiveGPU (built on HAMi)
Use cases: Large model inference, test services, speech recognition, domestic AI hardware (Huawei Ascend, Baidu Kunlun, etc.)
Outcomes:
- GPU savings: Large model inference runs 65 services on 28 GPUs (37 saved); test cluster runs 19 services on 6 GPUs (13 saved)
- Overall savings: Up to 57% GPU savings for production and test clusters
- Utilization improvement: Up to 100% GPU utilization improvement with GPU virtualization
Highlights: Cross-node collaborative scheduling, priority-based preemption, memory over-subscription

These cases demonstrate a consistent pattern: GPU virtualization becomes economically meaningful only when it participates in a governable contract—where utilization, isolation, and policy can be expressed, measured, and improved over time.

Strategic Implications for Dynamia

From Dynamia’s perspective (and as VP of Open Source Ecosystem), the strategic value of HAMi becomes clear:

Two-Layer Architecture: Open Source vs Commercial

HAMi (CNCF open source project): Responsible for “adoption and trust,” focused on GPU virtualization and compute efficiency
Dynamia enterprise products and services: Responsible for “production and scale,” providing commercial distributions and enterprise services built on HAMi

Figure 6: Dynamia Dual Mechanism: Open Source vs Commercial

This boundary is the foundation for long-term trust—project and company offerings remain separate, with commercial distributions and services built on the open source project.

Global Narrative Strategy

The internal alignment memo recommends a bilingual approach:

First layer: Lead globally with “GPU virtualization / sharing / utilization” (Chinese can directly use “GPU virtualization and heterogeneous scheduling,” but English first layer should avoid “heterogeneous” as a headline)

Second layer: When users discuss mixed GPUs or workload diversity, introduce “heterogeneous” to confirm capability boundaries—never as the opening hook

Core anchor: Maintain “HAMi (project and community) ≠ company products” as the non-negotiable baseline for long-term positioning

The Right Commercialization Landing

DaoCloud’s case study already set vendor-agnostic and CNCF toolchain compatibility as hard constraints, framing vendor dependency reduction as a business and operational benefit—not just a technical detail. Project-HAMi’s official documentation lists “avoid vendor lock” as a core value proposition.

In this context, the right commercialization landing isn’t “closed-source scheduling”—it’s productizing capabilities around real enterprise complexity:

Systematic compatibility matrix
SLO and tail-latency governance
Metering for billing
RBAC, quotas, multi-cluster governance
Upgrade and rollback safety
Faster path-to-production for DRA/CDI and other standardization efforts

Forward View: The Next 2–3 Years

My strong judgment: over the next 2–3 years, GPU scheduling competition will shift from “whose implementation is more black-box” to “whose contract is more open.”

The reasons are practical:

Hardware Form Factors and Supply Chains Are Diversifying

OpenAI’s February 12, 2026 “GPT‑5.3‑Codex‑Spark” release emphasizes ultra-low latency serving, including persistent WebSockets and a dedicated serving tier on Cerebras hardware
Large-scale GPU-backed financing announcements (for pan-European deployments) illustrate the infrastructure scale and financial engineering surrounding accelerator fleets

These signals suggest that heterogeneity will grow: mixed accelerators, mixed clouds, mixed workload types.

Low-Latency Inference Tiers Will Force Systematic Scheduling

Low-latency inference tiers (beyond just GPUs) will force resource scheduling toward “multi-accelerator, multi-layer cache, multi-class node” architectural design—scheduling must inherently be heterogeneous.

Open Scheduling Is Risk Management, Not Idealism

In this world, “open scheduling” isn’t idealism—it’s risk management. Building schedulable governable “control plane + data plane” combinations around DRA/CDI and other solidifying open interfaces, ones that are pluggable, multi-tenant governable, and co-evolvable with the ecosystem—this looks like the truly sustainable path for AI Native Infrastructure.

The next battleground isn’t “whose scheduling is smarter”—it’s “who can standardize device resource contracts into something governable.”

Conclusion

When you place HAMi 2025 back in the broader AI Native Infrastructure context, it’s no longer just the year of “GPU sharing tools”—it’s a more structural signal: GPUs are moving toward open scheduling.

Figure 7: Open Scheduling Future Vision

The driving forces come from both ends:

Upstream: Standards like DRA/CDI continue to solidify
Downstream: Scale and diversity (multi-cloud, multi-model, even accelerators beyond GPUs)

For Dynamia, HAMi’s significance has transcended “GPU sharing tool”: it turns GPU virtualization and slicing into declarative, schedulable, measurable data planes—letting queues, quotas, priorities, and multi-tenancy actually close the governance loop.

AI Learning Resources: 44 Curated Collections from Our Cleanup

Jimmy Song — Sun, 08 Feb 2026 12:20:05 +0000

“The best way to learn AI is to start building. These resources will guide your journey.”

Figure 1: AI Learning Resources Collection

In my ongoing effort to keep the AI Resources list focused on production-ready tools and frameworks, I’ve removed 44 collection-type projects—courses, tutorials, awesome lists, and cookbooks.

These resources aren’t gone—they’ve been moved here. This post is a curated collection of those educational materials, organized by type and topic. Whether you’re a complete beginner or an experienced practitioner, you’ll find something valuable here.

Why Remove Collections from AI Resources?

My AI Resources list now focuses on concrete tools and frameworks—projects you can directly use in production. Collections, while valuable, serve a different purpose: education and discovery.

By separating them, I:

Keep the resources list actionable and focused
Create a dedicated space for learning materials
Make it easier to find what you need

📚 Awesome Lists (14 Collections)

Awesome lists are community-curated collections of the best resources. They’re perfect for discovering new tools and staying updated.

Must-Explore Awesome Lists

Awesome Generative AI

Models, tools, tutorials, and research papers
Great for: Comprehensive overview of generative AI landscape

Awesome LLM

LLM resources: papers, tools, datasets, applications
Great for: Deep dive into large language models

Awesome AI Apps

Practical LLM applications, RAG examples, agent implementations
Great for: Real-world implementation examples

Awesome Claude Code

Claude Code commands, files, and workflows
Great for: Maximizing Claude Code productivity

Awesome MCP Servers

MCP servers for modular AI backend systems
Great for: Building with Model Context Protocol

Specialized Awesome Lists

Awesome ChatGPT Prompts - Prompt examples for various scenarios
Awesome LLM Apps - LLM applications with code examples
Awesome Multimodal LLM - Multimodal model resources
Awesome MCP Clients - MCP client tools and SDKs
Awesome Claude Skills - Claude Skills and workflows
Awesome GitHub Copilot - Copilot customizations
Awesome Nano Banana Pro - Image model prompts and examples
Awesome SaaS - AI platform templates
Awesome Claude Code Subagents - Claude Code subagents

🎓 Courses & Tutorials (9 Curricula)

Structured learning paths from universities and tech companies.

Microsoft’s AI Curriculum

AI for Beginners

12 weeks, 24 lessons covering neural networks, deep learning, CV, NLP
Great for: Complete AI foundation
Format: Lessons, quizzes, projects

Machine Learning for Beginners

12-week, 26-lesson curriculum on classic ML
Great for: ML fundamentals without deep math
Format: Project-based exercises

Generative AI for Beginners

18 lessons on building GenAI applications
Great for: Practical GenAI development
Format: Hands-on projects

AI Agents for Beginners

11 lessons on agent systems
Great for: Understanding autonomous agents
Format: Project-driven learning

EdgeAI for Beginners

Optimization, deployment, and real-world Edge AI
Great for: On-device AI applications
Format: Practical tutorials

MCP for Beginners

Model Context Protocol curriculum
Great for: Building with MCP
Format: Cross-language examples and labs

Official Platform Courses

Hugging Face Learn Center

Free courses on LLMs, deep RL, CV, audio
Great for: Hands-on Hugging Face ecosystem
Format: Interactive notebooks

OpenAI Cookbook

Runnable examples using OpenAI API
Great for: OpenAI API best practices
Format: Code examples and guides

PyTorch Tutorials

Basics to advanced deep learning
Great for: PyTorch mastery
Format: Comprehensive tutorials

🍳 Cookbooks & Example Collections (5 Collections)

Practical code examples and recipes.

Claude Cookbooks

Notebooks and examples for building with Claude
Great for: Anthropic Claude integration
Format: Jupyter notebooks

Hugging Face Cookbook

Practical AI cookbook with Jupyter notebooks
Great for: Open models and tools
Format: Hands-on examples

Tinker Cookbook

Training and fine-tuning examples
Great for: Fine-tuning workflows
Format: Platform-specific recipes

E2B Cookbook

Examples for building LLM apps
Great for: LLM application development
Format: Recipes and tutorials

arXiv Paper Curator

6-week course on RAG systems
Great for: Production-ready RAG
Format: Project-based learning

📖 Guides & Handbooks (5 Resources)

In-depth guides on specific topics.

Prompt Engineering Guide

Comprehensive prompt engineering resources
Great for: Mastering prompt design
Format: Guides, papers, lectures, notebooks

Evaluation Guidebook

LLM evaluation best practices from Hugging Face
Great for: Assessing LLM performance
Format: Practical guide

Context Engineering

Design and optimize context beyond prompt engineering
Great for: Advanced context management
Format: Practical handbook

Context Engineering Intro

Template and guide for context engineering
Great for: Providing project context to AI assistants
Format: Template + guide

Vibe-Coding Workflow

5-step prompt template for building MVPs with LLMs
Great for: Rapid prototyping with AI
Format: Workflow template

🗂️ Template & Workflow Collections

Reusable templates and workflows.

Claude Code Templates

Code templates for various programming scenarios
Great for: Claude AI development
Format: Template collection

n8n Workflows

2,000+ professionally organized n8n workflows
Great for: Workflow automation
Format: Searchable catalog

N8N Workflows Catalog

Community-driven reusable workflow templates
Great for: Workflow import and versioning
Format: Template catalog

📊 Research & Evaluation

Academic and evaluation resources.

LLMSys PaperList

Curated list of LLM systems papers
Great for: Research on training, inference, serving
Format: Paper collection

Free LLM API Resources

LLM providers with free/trial API access
Great for: Experimentation without cost
Format: Provider list

🎨 Other Notable Resources

System Prompts and Models of AI Tools

Community-curated collection of system prompts and AI tool examples
Great for: Prompt and agent engineering
Format: Resource collection

ML Course CS-433

EPFL Machine Learning Course
Great for: Academic ML foundation
Format: Lectures, labs, projects

Machine Learning Engineering

ML engineering open-book: compute, storage, networking
Great for: Production ML systems
Format: Comprehensive guide

Realtime Phone Agents Course

Build low-latency voice agents
Great for: Voice AI applications
Format: Hands-on course

LLMs from Scratch

Build a working LLM from first principles
Great for: Understanding LLM internals
Format: Repository + book materials

💡 How to Use This Collection

For Complete Beginners

Start with: Microsoft’s AI for Beginners
Practice with: PyTorch Tutorials
Explore: Awesome AI Apps for inspiration

For Developers

Build skills: OpenAI Cookbook + Claude Cookbooks
Find tools: Awesome Generative AI + Awesome LLM
Learn workflows: n8n Workflows Catalog

For Researchers

Stay updated: Awesome Generative AI + LLMSys PaperList
Deep dive: Awesome LLM
Implement: Hugging Face Cookbook

For Product Builders

Find examples: Awesome AI Apps
Learn workflows: n8n Workflows Catalog
Study patterns: Awesome LLM Apps

🔄 What Was NOT Removed

Agent frameworks and production tools remain in the AI Resources list, including:

AutoGen - Microsoft’s multi-agent framework
CrewAI - High-performance multi-agent orchestration
LangGraph - Stateful multi-agent applications
Flowise - Visual agent platform
Langflow - Visual workflow builder
And 80+ more agent frameworks

These are functional tools you can use to build applications, not educational collections. They belong in the AI Resources list.

📝 Summary

I removed 44 collection-type projects from the AI Resources list to keep it focused on production tools:

14 Awesome Lists - Discover new tools and stay updated
9 Courses & Tutorials - Structured learning paths
5 Cookbooks - Practical code examples
5 Guides & Handbooks - In-depth resources
4 Template Collections - Reusable workflows
7 Other Resources - Research and evaluation

These resources remain incredibly valuable for learning and discovery. They just serve a different purpose than the production-focused tools in my AI Resources list.

Next Steps:

Bookmark this post for future reference
Explore the AI Resources list for production tools (agent frameworks, databases, etc.)
Check out my blog for more AI engineering insights

Acknowledgments: This collection was compiled during my AI Resources cleanup initiative. Special thanks to all the maintainers of these awesome lists, courses, and collections for their invaluable contributions to the AI community.

Standing on Giants' Shoulders: The Traditional Infrastructure Powering Modern AI

Jimmy Song — Sun, 08 Feb 2026 08:00:00 +0000

“If I have seen further, it is by standing on the shoulders of giants.” — Isaac Newton

Figure 1: Standing on Giants’ Shoulders: The Traditional Infrastructure Powering Modern AI

In the excitement surrounding LLMs, vector databases, and AI agents, it’s easy to forget that modern AI didn’t emerge from a vacuum. Today’s AI revolution stands upon decades of infrastructure work—distributed systems, data pipelines, search engines, and orchestration platforms that were built long before “AI Native” became a buzzword.

This post is a tribute to those traditional open source projects that became the invisible foundation of AI infrastructure. They’re not “AI projects” per se, but without them, the AI revolution as we know it wouldn’t exist.

The Evolution: From Big Data to AI

Era	Focus	Core Technologies	AI Connection
2000s	Web Search & Indexing	Lucene, Elasticsearch	Semantic search foundations
2010s	Big Data & Distributed Computing	Hadoop, Spark, Kafka	Data processing at scale
2010s	Cloud Native	Docker, Kubernetes	Model deployment platforms
2010s	Stream Processing	Flink, Storm, Pulsar	Real-time ML inference
2020s	AI Native	Transformers, Vector DBs	Built on everything above

Table 1: Evolution of Data Infrastructure

Big Data Frameworks: The Data Engines

Before we could train models on petabytes of data, we needed ways to store, process, and move that data.

Apache Hadoop (2006)

GitHub: https://github.com/apache/hadoop

Hadoop democratized big data by making distributed computing accessible. Its HDFS filesystem and MapReduce paradigm proved that commodity hardware could process web-scale datasets.

Why it matters for AI:

Modern ML training datasets live in HDFS-compatible storage
Data lakes built on Hadoop became training data reservoirs
Proved that distributed computing could scale horizontally

Apache Kafka (2011)

GitHub: https://github.com/apache/kafka

Kafka redefined data streaming with its log-based architecture. It became the nervous system for real-time data flows in enterprises worldwide.

Why it matters for AI:

Real-time feature pipelines for ML models
Event-driven architectures for AI agent systems
Streaming inference pipelines
Model telemetry and monitoring backbones

Apache Spark (2014)

GitHub: https://github.com/apache/spark

Spark brought in-memory computing to big data, making iterative algorithms (like ML training) practical at scale.

Why it matters for AI:

MLlib made ML accessible to data engineers
Distributed data processing for model training
Spark ML became the de facto standard for big data ML
Proved that in-memory computing could accelerate ML workloads

Search Engines: The Retrieval Foundation

Before RAG (Retrieval-Augmented Generation) became a buzzword, search engines were solving retrieval at scale.

Elasticsearch (2010)

GitHub: https://github.com/elastic/elasticsearch

Elasticsearch made full-text search accessible and scalable. Its distributed architecture and RESTful API became the standard for search.

Why it matters for AI:

pioneered distributed inverted index structures
Proved that horizontal scaling was possible for search workloads
Many “AI search” systems actually use Elasticsearch under the hood
Query DSL influenced modern vector database query languages

OpenSearch (2021)

GitHub: https://github.com/opensearch-project/opensearch

When AWS forked Elasticsearch, it ensured search infrastructure remained truly open. OpenSearch continues the mission of accessible, scalable search.

Why it matters for AI:

Maintains open source innovation in search
Vector search capabilities added in 2023
Demonstrates community fork resilience

Databases: From SQL to Vectors

The evolution from relational databases to vector databases represents a paradigm shift—but both have AI relevance.

Traditional Databases That Paved the Way

Dgraph (2015) - Graph database proving that specialized data structures enable new use cases
TDengine (2019) - Time-series database for IoT ML workloads
OceanBase (2021) - Distributed database showing ACID transactions could scale

Why they matter for AI:

Proved that specialized database engines could outperform general-purpose ones
Database internals (indexing, sharding, replication) are now applied to vector databases
Multi-model databases (graph + vector + relational) are becoming the norm for AI apps

Cloud Native: The Runtime Foundation

When Docker and Kubernetes emerged, they weren’t built for AI—but AI couldn’t scale without them.

Docker (2013) & Kubernetes (2014)

GitHub: https://github.com/kubernetes/kubernetes

Kubernetes became the operating system for cloud-native applications. Its declarative API and controller pattern made it perfect for AI workloads.

Why it matters for AI:

Model deployment platforms (KServe, Seldon Core) run on K8s
GPU orchestration (NVIDIA GPU Operator, Volcano, HAMi) extends K8s
Kubeflow made K8s the standard for ML pipelines
Microservice patterns enable modular AI agent architectures

Service Mesh & Serverless

Istio (2016), Knative (2018) - Service mesh and serverless platforms that proved:

Network-level observability applies to AI model calls
Scale-to-zero is essential for cost-effective inference
Traffic splitting enables A/B testing of ML models

Why they matter for AI:

AI Gateway patterns evolved from API gateways + service mesh
Serverless inference platforms use Knative-style autoscaling
Observability patterns (tracing, metrics) are now standard for ML systems

API Gateways: From REST to LLM

API gateways weren’t designed for AI, but they became the foundation of AI Gateway patterns.

Kong, APISIX, KGateway

These API gateways solved rate limiting, auth, and routing at scale. When LLMs emerged, the same patterns applied:

AI Gateway Evolution:

Traditional API Gateway (2010s)
 ↓
Rate Limiting → Token Bucket Rate Limiting
Auth → API Key + Organization Management
Routing → Model Routing (GPT-4 → Claude → Local Models)
Observability → LLM-specific Telemetry (token usage, cost)
 ↓
AI Gateway (2024)

Why they matter for AI:

Proved that centralized API management scales
Plugin architectures enable LLM-specific features
Traffic management patterns apply to prompt routing
Security patterns (mTLS, JWT) now protect AI endpoints

Workflow Orchestration: The Pipeline Backbone

Data engineering needs pipelines. ML engineering needs pipelines. AI agents need workflows.

Apache Airflow (2015)

GitHub: https://github.com/apache/airflow

Airflow made pipeline orchestration accessible with its DAG-based approach. It became the standard for ETL and data engineering.

Why it matters for AI:

ML pipeline orchestration (feature engineering, training, evaluation)
Proved that DAG-based workflow definition works at scale
Prompt engineering pipelines use Airflow-style orchestration
Scheduler patterns are now applied to AI agent workflows

n8n, Prefect, Flyte

Modern workflow platforms that evolved from Airflow’s foundations:

n8n (2019) - Visual workflow automation with AI capabilities
Prefect (2018) - Python-native workflow orchestration for ML
Flyte (2019) - Kubernetes-native workflow orchestration for ML/data

Why they matter for AI:

Multi-modal agents need workflow orchestration
RAG pipelines are essentially ETL pipelines for embeddings
Prompt chaining is DAG-based orchestration

Data Formats: The Lakehouse Foundation

Before we could train on massive datasets, we needed formats that supported ACID transactions and schema evolution.

Delta Lake, Apache Iceberg, Apache Hudi

These table formats brought reliability to data lakes:

Why they matter for AI:

Training datasets need versioning and reproducibility
Feature stores use Delta/Iceberg as storage formats
Proved that “big data” could have transactional semantics
Schema evolution handles ML feature drift

The Invisible Thread: Why These Projects Matter

What do all these projects have in common?

They solved scaling first - AI training/inference needs horizontal scaling
They proved distributed systems work - Modern AI is fundamentally distributed
They created ecosystem patterns - Plugin systems, extension points, APIs
They established best practices - Observability, security, CI/CD
They built developer habits - YAML configs, declarative APIs, CLI tools

The AI Native Continuum

Modern “AI Native” infrastructure didn’t replace these projects—it builds on them:

Traditional Project	AI Native Evolution	Example
Hadoop HDFS	Distributed model storage	HDFS for datasets, S3 for checkpoints
Kafka	Real-time feature pipelines	Kafka → Feature Store → Model Serving
Spark ML	Distributed ML training	MLlib → PyTorch Distributed
Elasticsearch	Vector search	ES → Weaviate/Qdrant/Milvus
Kubernetes	ML orchestration	K8s → Kubeflow/KServe
Istio	AI Gateway service mesh	Istio → LLM Gateway with mTLS
Airflow	ML pipeline orchestration	Airflow → Prefect/Flyte for ML

Table 2: From Traditional to AI Native

Why We’re Removing Them from AI Resources List

This post honors these projects, but we’re also removing them from our AI Resources list. Here’s why:

They’re not “AI Projects”—they’re foundational infrastructure.

Hadoop, Kafka, Spark are data engineering tools, not ML frameworks
Elasticsearch is search, not semantic search
Kubernetes is general-purpose orchestration
API gateways serve REST/GraphQL, not just LLMs

But their absence doesn’t diminish their importance.

By removing them, we acknowledge that:

AI has its own ecosystem - Transformers, vector DBs, LLM ops
Traditional infra has its own domain - Data engineering, cloud native
The intersection is where innovation happens - AI-native data platforms, LLM ops on K8s

The Giants We Stand On

The next time you:

Deploy a model on Kubernetes
Stream features through Kafka
Search embeddings with a vector database
Orchestrate a RAG pipeline with Prefect

Remember: You’re standing on the shoulders of Hadoop, Kafka, Elasticsearch, Kubernetes, and countless others. They built the roads we now drive on.

The Future: Building New Giants

Just as Hadoop and Kafka enabled modern AI, today’s AI infrastructure will become tomorrow’s foundation:

Vector databases may become the new standard for all search
LLM observability may evolve into general distributed tracing
AI agent orchestration may reinvent workflow automation
GPU scheduling may influence general-purpose resource management

The cycle continues. The giants of today will be the foundations of tomorrow.

Conclusion: Gratitude and Continuity

As we clean up our AI Resources list to focus on AI-native projects, we don’t forget where we came from. Traditional big data and cloud native infrastructure made the AI revolution possible.

To the Hadoop committers, Kafka maintainers, Kubernetes contributors, and all who built the foundation: Thank you.

Your work enabled ChatGPT, enabled Transformers, enabled everything we now call “AI.”

Standing on your shoulders, we see further.

Acknowledgments: This post was inspired by the need to refactor our AI Resources list. The 27 projects mentioned here are being removed—not because they’re unimportant, but because they deserve their own category: The Foundation.

My First Month at Dynamia: Why AI Native Infra Is Worth the Investment

Jimmy Song — Fri, 06 Feb 2026 12:56:35 +0000

Time flies—it’s already been a month since I joined Dynamia. In this article, I want to share my observations from this past month: why AI Native Infra is a direction worth investing in, and some considerations for those thinking about their own career or technical direction.

Introduction

After nearly five years of remote work, I officially joined Dynamia last month as VP of Open Source Ecosystem. This decision was not sudden, but a natural extension of my journey from cloud native to AI Native Infra.

But this article is not just about my personal choice. I want to answer a more universal question: In the wave of AI infrastructure startups, why is compute governance a direction worth investing in?

For the past decade, I have worked continuously in the infrastructure space: from Kubernetes to Service Mesh, and now to AI Infra. I am increasingly convinced that the core challenge in the AI era is not “can the model run,” but “can compute resources be run efficiently, reliably, and in a controlled manner.” This conviction has only grown stronger through my observations and reflections during this first month at Dynamia.

This article answers three questions: What is AI Native Infra? Why is GPU virtualization a necessity? Why did I choose Dynamia and HAMi?

What Is AI Native Infra

The core of AI Native Infrastructure is not about adding another platform layer, but about redefining the governance target: expanding from “services and containers” to “model behaviors and compute assets.”

I summarize it as three key shifts:

Models as execution entities: Governance now includes not just processes, but also model behaviors.
Compute as a scarce asset: GPU, memory, and bandwidth must be scheduled and metered precisely.
Uncertainty as the default: Systems must remain observable and recoverable amid fluctuations.

In essence, AI Native Infra is about upgrading compute governance from “resource allocation” to “sustainable business capability.”

Why GPU Virtualization Is Essential

Many teams focus on model inference optimization, but in production, enterprises first encounter the problem of “underutilized GPUs.” This is where GPU virtualization delivers value.

Structural idleness: Small tasks monopolize large GPUs, leaving them idle for long periods.
Pseudo-isolation risks: Native sharing lacks hard boundaries, so a single task OOM can cause cascading failures.
Scheduling failures: Some users queue for GPUs while others occupy but do not use them, leading to both shortages and idleness.
Fragmentation waste: There may be enough total GPU, but not enough full cards, making efficient packing impossible.
Vendor lock-in anxiety: Proprietary, tightly coupled solutions make migration costs uncontrollable.

In short: GPUs must not only be allocatable, but also splittable, isolatable, schedulable, and governable.

The Relationship Between HAMi and Dynamia

This is the most frequently asked question. Here is the shortest answer:

HAMi: A CNCF-hosted open source project and community focused on GPU virtualization and heterogeneous compute scheduling.
Dynamia: The founding and leading company behind HAMi, providing enterprise-grade products and services based on HAMi.

Open source projects are not the same as company products, but the two evolve together. HAMi drives industry adoption and technical trust, while Dynamia brings these capabilities into enterprise production environments at scale. This “dual engine” approach is what makes Dynamia unique.

What HAMi Provides

HAMi (Heterogeneous AI Computing Virtualization Middleware) delivers three key capabilities on Kubernetes:

Virtualization and partitioning: Split physical GPUs into logical resources on demand to improve utilization.
Scheduling and topology awareness: Place workloads optimally based on topology to reduce communication bottlenecks.
Isolation and observability: Support quotas, policies, and monitoring to reduce production risks.

Currently, HAMi has attracted over 360 contributors from 16 countries, with more than 200 enterprise end users, and its international influence continues to grow.

Market Trends: The AI Infrastructure Startup Wave

AI infrastructure is experiencing a new wave of startups. The vLLM team’s company raised $150 million, SGLang’s commercial spin-off RadixArk is valued at $4 billion, and Databricks acquired MosaicML for $1.3 billion—all pointing to a consensus: Whoever helps enterprises run large models more efficiently and cost-effectively will hold the keys to next-generation AI infrastructure.

Against this backdrop, the positioning of Dynamia and HAMi is even clearer. Many teams focus on “model performance acceleration” and “inference optimization” (like vLLM, SGLang), while we focus on “resource scheduling and virtualization”—enabling better orchestration of existing accelerated hardware resources.

The two are complementary: the former makes individual models run faster and cheaper, while the latter ensures that compute allocation at the cluster level is efficient, fair, and controllable. This is similar to extending Kubernetes’ CPU/memory scheduling philosophy to GPU and heterogeneous compute management in the AI era.

Why AI Native Infra Is Worth the Investment

My observations this month have convinced me that compute governance is the most undervalued yet most promising area in AI infrastructure. If you are considering a career or technical investment, here is my assessment:

First, this is a real and urgent pain point

Model training and inference optimization attract a lot of attention, but in production, enterprises first encounter the problem of “underutilized GPUs”—structural idleness, scheduling failures, fragmentation waste, and vendor lock-in anxiety. Without solving these problems, even the fastest models cannot scale in production. GPU virtualization and heterogeneous compute scheduling are the “infrastructure below infrastructure” for enterprise AI transformation.

Second, this is a clear long-term track

Frameworks like vLLM and SGLang emerge constantly, making individual models run faster. But who ensures that compute allocation at the cluster level is efficient, fair, and controllable? This is similar to extending Kubernetes’ success in CPU/memory scheduling to GPU and heterogeneous compute management in the AI era. This is not something that can be finished in a year or two, but a direction for continuous construction over the next five to ten years.

Third, this is an open and verifiable path

Dynamia chose to build on HAMi as an open source foundation, first solving general capabilities, then supporting enterprise adoption. This means the technical direction is transparent and verifiable in the community. You can form your own judgment by participating in open source, observing adoption, and evaluating the ecosystem—rather than relying on the black-box promises of proprietary solutions.

Fourth, this is a window of opportunity that is opening now

AI infrastructure is being redefined. Investing in its construction today will continue to yield value in the coming years. The vLLM team’s company raised $150 million, SGLang’s commercial spin-off RadixArk is valued at $4 billion, Databricks acquired MosaicML for $1.3 billion—all validating the same trend: Whoever helps enterprises run large models more efficiently will hold the keys to next-generation AI infrastructure.

I hope to bring my experience in cloud native and open source communities to the next stage of HAMi and Dynamia: turning GPU resources from a “cost center” into an “operational asset.” This is not just my career choice, but my judgment and investment in the direction of next-generation infrastructure.

Join the HAMi Community

Add me on WeChat (jimmysong) to join the HAMi community focused on GPU virtualization and heterogeneous compute scheduling.

If you are also interested in HAMi, GPU virtualization, AI Native Infra, or Dynamia, feel free to reach out.

Summary

From cloud native to AI Native Infra, my observations this month have only strengthened my conviction: The true upper limit of AI applications is determined by the infrastructure’s ability to govern compute resources.

HAMi addresses the fundamental issues of GPU virtualization and heterogeneous compute scheduling, while Dynamia is driving these capabilities into large-scale production. If you are also looking for a technical direction worth long-term investment, AI Native Infra—especially compute governance and scheduling—is a track with real pain points, a clear path, an open ecosystem, and an opening window of opportunity.

Joining Dynamia is not just a career choice, but a commitment to building the next generation of infrastructure. I hope the observations and reflections in this article can provide some reference for you as you evaluate technical directions and career opportunities.

If you are also interested in HAMi, GPU virtualization, AI Native Infra, or Dynamia, feel free to reach out.

The True Inflection Point of ADD: When Spec Becomes the Core Asset of AI-Era Software

Jimmy Song — Tue, 20 Jan 2026 07:51:36 +0000

The role of Spec is undergoing a fundamental transformation, becoming the governance anchor of engineering systems in the AI era.

The Essence of Software Engineering and the Cost Structure Shift Brought by AI

From first principles, software engineering has always been about one thing: stably, controllably, and reproducibly transforming human intent into executable systems.

Artificial Intelligence (AI) does not change this engineering essence, but it dramatically alters the cost structure:

Implementation costs plummet: Code, tests, and boilerplate logic are rapidly commoditized.
Consistency costs rise sharply: Intent drift, hidden conflicts, and cross-module inconsistencies become more frequent.
Governance costs are amplified: As agents can act directly, auditability, accountability, and explainability become hard constraints.

Therefore, in the era of Agent-Driven Development (ADD), the core issue is not “can agents do the work,” but how to maintain controllability and intent preservation in engineering systems under highly autonomous agents.

The ADD Era Inflection Point: Three Structural Preconditions

Many attribute the “explosion” of ADD to more mature multi-agent systems, stronger models, or more automated tools. In reality, the true structural inflection point arises only when these three conditions are met:

Agents have acquired multi-step execution capabilities

With frameworks like LangChain, LangGraph, and CrewAI, agents are no longer just prompt invocations, but long-lived entities capable of planning, decomposition, execution, and rollback.

Agents are entering real enterprise delivery pipelines

Once in enterprise R&D, the question shifts from “can it generate” to “who approved it, is it compliant, can it be rolled back.”

Traditional engineering tools lack a control plane for the agent era

Tools like Git, CI, and Issue Trackers were designed for “human developer collaboration,” not for “agent execution.”

When these three factors converge, ADD inevitably shifts from an “efficiency tool” to a “governance system.”

The Changing Role of Spec: From Documentation to System Constraint

In the context of ADD, Spec is undergoing a fundamental shift:

Spec is no longer “documentation for humans,” but “the source of constraints and facts for systems and agents to execute.”

Spec now serves at least three roles:

Verifiable expression of intent and boundaries

Requirements, acceptance criteria, and design principles are no longer just text, but objects that can be checked, aligned, and traced.

Stable contracts for organizational collaboration

When agents participate in delivery, verbal consensus and tacit knowledge quickly fail. Versioned, auditable artifacts become the foundation of collaboration.

Policy surface for agent execution

Agents can write code, modify configurations, and trigger pipelines. Spec must become the constraint on “what can and cannot be done.”

From this perspective, the status of Spec is approaching that of the Control Plane in AI-native infrastructure.

The Reality of Multi-Agent Workflows: Orchestration and Governance First

In recent systems (such as APOX and other enterprise products), an industry consensus is emerging:

Multi-agent collaboration no longer pursues “full automation,” but is staged and gated.
Frameworks like LangGraph are used to build persistent, debuggable agent workflows.
RAG (e.g., based on Milvus) is used to accumulate historical Specs, decisions, and context as long-term memory.
The IDE mainly focuses on execution efficiency, not engineering governance.

Figure 1: APOX user interface

APOX (AI Product Orchestration eXtended) is a multi-agent collaboration workflow platform for enterprise software delivery. Its core goals are:

To connect the entire process from product requirements to executable code with a governable Agentflow and explicit engineering artifact chain.
To assign dedicated AI agents to each delivery stage (such as PRD, PO, Architecture, Developer, Implementation, Coding, etc.).
To embed manual approval gates and full audit trails at every step, solving the “intent drift and consistency” governance problem that traditional AI coding tools cannot address.
The platform provides a VS Code plugin for real-time sync between local IDE and web artifacts, allowing Specs, code, tasks, and approval statuses to coexist in the repository.
Supports assigning different base models to different agents according to enterprise needs.

APOX is not about simply speeding up code generation, but about elevating “Spec” from auxiliary documentation to a verifiable, constrainable, and traceable core asset in engineering—building a control plane and workflow governance system suitable for Agent-Driven Development.

Such systems emphasize:

An explicit artifact chain from PRD → Spec → Task → Implementation.
Manual confirmation and audit points at every stage.
Bidirectional sync between Spec, code, repository, and IDE.

This is not about “smarter AI,” but about engineering systems adapting to the agent era.

The Long-Term Value of Spec: The Core Anchor of Engineering Assets

This is not to devalue code, but to acknowledge reality:

There will always be long-term differentiation in algorithms and model capabilities.
General engineering implementation is rapidly homogenizing.
What is hard to replicate is: how to define problems, constrain systems, and govern change.

In the ADD era, the value of Spec is reflected in:

Determining what agents can and cannot do.
Carrying the organization’s long-term understanding of the system.
Serving as the anchor for audit, compliance, and accountability.

Code will be rewritten again and again; Spec is the long-term asset.

Risks and Challenges of ADD: Living Spec and Governance Constraints

ADD also faces significant risks:

Can Spec become a Living Spec

That is, when key implementation changes occur, can the system detect “intent changes” and prompt Spec updates, rather than allowing silent drift?

Can governance achieve low friction but strong constraints

If gates are too strict, teams will bypass them; if too loose, the system loses control.

These two factors determine whether ADD is “the next engineering paradigm” or “just another tool bubble.”

The Trend Toward Control Planes in Engineering Systems

From a broader perspective, ADD is the inevitable result of engineering systems becoming “control planes”:

Engineering systems are evolving from “human collaboration tools” to “control systems for agent execution.”

In this structure:

Agent / IDE is the execution plane.
RAG / Memory is the state and memory plane.
Spec is the intent and policy plane.
Gates, audit, and traceability form the governance loop.

This closely aligns with the evolution path of AI-native infrastructure.

Summary

The winners of the ADD era will not be the systems with “the most agents or the fastest generation,” but those that first upgrade Spec from documentation to a governable, auditable, and executable asset. As automation advances, the true scarcity is the long-term control of intent.

AI Voice Dictation Input Methods Are Becoming the New Shortcut Key for the Programming Era

Jimmy Song — Sun, 18 Jan 2026 06:53:08 +0000

Voice input methods are not just about being “fast”—they are becoming a brand new gateway for developers to collaborate with AI.

Warning

On January 12, 2026, due to financial difficulties encountered during operations, the Miaoyan project announced the cessation of operations and the team was disbanded. The application will no longer be updated or maintained, but existing versions can continue to be used on the current device and system, and do not store any audio or transcription content.

Figure 1: Can voice input become the new shortcut for developers? My in-depth comparison experience.

AI Voice Input Methods Are Becoming the “New Shortcut Key” in the Programming Era

I am increasingly convinced of one thing: PC-based AI voice input methods are evolving from mere “input tools” into the foundational interaction layer for the era of programming and AI collaboration.

It’s not just about typing faster—it determines how you deliver your intent to the system, whether you’re writing documentation, code, or collaborating with AI in IDEs, terminals, or chat windows.

Because of this, the differences in voice input method experiences are far more significant than they appear on the surface.

My Six Evaluation Criteria for AI Voice Input Methods

After long-term, high-frequency use, I have developed a set of criteria to assess the real-world performance of AI voice input methods:

Response speed: Does text appear quickly enough after pressing the shortcut to keep up with your thoughts?
Continuous input stability: Does it remain reliable during extended use, or does it suddenly fail or miss recognition?
Mixed Chinese-English and technical terms: Can it reliably handle code, paths, abbreviations, and product names?
Developer friendliness: Is it truly designed for command line, IDE, and automation scenarios?
Interaction restraint: Does it avoid introducing distracting features that interfere with input itself?
Subscription and cost structure: Is it a standalone paid product, or can it be bundled with existing tool subscriptions?

Based on these criteria, I focused on comparing Miaoyan, Shandianshuo, and Zhipu AI Voice Input Method.

Miaoyan: Currently the Most “Developer-Oriented” Domestic Product

Miaoyan was the first domestic AI voice input method I used extensively, and it remains the one I am most willing to use continuously.

Figure 2: Miaoyan is currently my most-used Mac voice input method.

Command Mode: The Key Differentiator for Developer Productivity

It’s important to clarify that Miaoyan’s command mode is not about editing text via voice. Instead:

You describe your need in natural language, and the system directly generates an executable command-line command.

This is crucial for developers:

It’s not just about input
It’s about turning voice into an automation entry point
Essentially, it connects voice to the CLI or toolchain

This design is clearly focused on engineering efficiency, not office document polishing.

Usage Experience Summary

Fast response, nearly instant
Output is relatively clean, with minimal guessing
Interaction design is restrained, with no unnecessary concepts
Developer-friendly mindset

But there are some practical limitations:

It is a completely standalone product
Requires a separate subscription
Still in relatively small-scale use

From a product strategy perspective, it feels more like a “pure tool” than part of an ecosystem.

Note

Shandianshuo: Local-First Approach, Developer Experience Depends on Your Setup

Shandianshuo takes a different approach: it treats voice input as a “local-first foundational capability,” emphasizing low latency and privacy (at least in its product narrative). The natural advantages of this approach are speed and controllable marginal costs, making it suitable as a “system capability” that’s always available, rather than a cloud service.

Figure 3: Shandianshuo settings page

However, from a developer’s perspective, its upper limit often depends on “how you implement enhanced capabilities”:

If you only use it for basic transcription, the experience is more like a high-quality local input tool. But if you want better mixed Chinese-English input, technical term correction, symbol and formatting handling, the common approach is to add optional AI correction/enhancement capabilities, which usually requires extra configuration (such as providing your own API key or subscribing to enhanced features). The key trade-off here is not “can it be used,” but “how much configuration cost are you willing to pay for enhanced capabilities.”

If you want voice input to be a “lightweight, stable, non-intrusive” foundation, Shandianshuo is worth considering. But if your goal is to make voice input part of your developer workflow (such as command generation or executable actions), it needs to offer stronger productized design at the “command layer” and in terms of controllability.

Zhipu AI Voice Input Method: Stable but with Friction

I also thoroughly tested the Zhipu AI Voice Input Method.

Figure 4: Zhipu Voice Input Method settings interface

Its strengths include:

More stable for long-term continuous input
Rarely becomes completely unresponsive
Good tolerance for longer Chinese input

But with frequent use, some issues stand out:

Idle misrecognition: If you press the shortcut but don’t speak, it may output random characters, disrupting your input flow
Occasionally messy output: Sometimes adds irrelevant words, making it less controllable than Miaoyan
Basic recognition errors: For example, “Zhipu” being recognized as “Zhipu” (with a different character), which is a trust issue for professional users
Feature-heavy design: Various tone and style features increase cognitive load

Subscription Bundling: Zhipu’s Practical Advantage

Although I prefer Miaoyan in terms of experience, Zhipu has a very practical advantage:

If you already subscribe to Zhipu’s programming package, the voice input method is included for free.

This means:

No need to pay separately for the input method
Lower psychological and decision-making cost
More likely to become the “default tool” that stays

From a business perspective, this is a very smart strategy.

Main Comparison Table

The following table compares the three products across key dimensions for quick reference.

Dimension	Miaoyan	Shandianshuo	Zhipu AI Voice Input Method
Response Speed	Fast, nearly instant	Usually fast (local-first)	Slightly slower than Miaoyan
Continuous Stability	Stable	Depends on setup and environment	Very stable
Idle Misrecognition	Rare	Generally restrained (varies by version)	Obvious: outputs characters even if silent
Output Cleanliness/Control	High	More like an “input tool”	Occasionally messy
Developer Differentiator	Natural language → executable command	Local-first / optional enhancements	Ecosystem-attached capabilities
Subscription & Cost	Standalone, separate purchase	Basic usable; enhancements often require setup/subscription	Bundled free with programming package
My Current Preference	Best experience	More like a “foundation approach”	Easy to keep but not clean enough

Table 1: Core Comparison of Miaoyan, Shandianshuo, and Zhipu AI Voice Input Methods

User Loyalty to AI Voice Input Methods

The switching cost for voice input methods is actually low: just a shortcut key and a habit of output.

What really determines whether users stick around is:

Whether the output is controllable
Whether it keeps causing annoying minor issues
Whether it integrates into your existing workflow and payment structure

For me personally:

The best and smoothest experience is still Miaoyan
The one most likely to stick around is probably Zhipu
Shandianshuo is more of a “foundation approach” and worth watching for how its enhancements evolve

These points are not contradictory.

Summary

Miaoyan is more mature in engineering orientation, command capabilities, and input control
Zhipu has practical advantages in stability and subscription bundling
Shandianshuo takes a local-first + optional enhancement approach, with the key being how it balances “basic capability” and “enhancement cost”
Who truly becomes the “default gateway” depends on reducing distractions, fixing frequent minor issues, and treating voice input as true “infrastructure” rather than an add-on feature

The competition among AI voice input methods is no longer about recognition accuracy, but about who can own the shortcut key you press every day.

From Spatial Data to AI Open Source: Technical Standards, Data Sovereignty, and the Global Divide

Jimmy Song — Sun, 11 Jan 2026 03:29:28 +0000

The divide in technical standards and data sovereignty determines the global competitive landscape of infrastructure open source in the AI era.

In this article, I will use the differences in air quality data presentation in Apple Maps and Weather as a starting point to explore how technical standards and data sovereignty influence the open source paths of AI in different countries. I will further analyze why, in the AI era, infrastructure-level open source has become the key battleground for ecosystem dominance.

Author’s Note

This article originates from a very everyday observation: Why is air quality data in China shown as “points” in Apple Maps and Weather, while in other countries it is often displayed as “areas”?

Figure 1: Air quality map in Apple Weather, showing point-based data in China and area-based data in other countries

At first glance, it seems like a product experience difference. But when I reconsidered this issue in the context of engineering, standards, and system design, I realized it actually points to a much bigger question: how different countries understand the relationship between technology, standards, openness, and sovereignty.

As an engineer who has long worked in cloud native, AI infrastructure, and open source ecosystems, I gradually realized that this difference is not limited to air quality or map data. In the AI era, it is further amplified, directly affecting how we open source models, build infrastructure, and whether we can participate in the formulation of global rules.

Writing this article is not about judging right or wrong, but about using a concrete example to explain a structural difference and discuss the long-term impact and real opportunities this difference may bring in the AI era.

What is especially important: at the level of AI infrastructure and infra-level open source, the competition has just begun. China is not without opportunities, but the choice of path will become more critical than ever.

Differences in Air Quality Data Presentation: A Microcosm of Technical Standards and Sovereignty

The following image illustrates the divide between spatial data, AI open source, and technical standards. By comparing how air quality data is presented in Apple Maps and Weather in different countries, you can intuitively feel the differences in technical standards and sovereignty strategies behind the scenes.

Figure 2: The divide between spatial data, AI open source, and technical standards

If you regularly use global products such as maps, weather, traffic, or various data services, you may notice a recurring phenomenon that is rarely discussed seriously: the way data is presented in China often differs significantly from global mainstream standards.

A very intuitive example comes from the air quality display in Apple Maps or Weather. In China, air quality is usually shown as discrete points; in the US, Europe, Japan, and other countries, it is often rendered as continuous coverage areas.

At first glance, this seems like a product experience difference, and may even lead people to mistakenly believe that “China’s data is incomplete.” But if you treat it as an engineering or system design issue, you will find: this is not a matter of data capability, but a different choice in technical standards, data sovereignty, and openness strategies.

And this choice is not limited to air quality.

Air Quality Is Just a Slice: Greater Differences in Spatial Public Data

Air quality is just a highly visible and relatively low-risk example. Similar differences have long existed in broader spatial and public data domains.

Maps and coordinate systems
Surveying and high-precision spatial data
Real-time traffic and population movement
Remote sensing, environmental, and urban operation data

In global mainstream systems, such data is usually regarded as public information infrastructure. It is standardized, gridded, API-ified, allows interpolation, modeling, and redistribution, and is widely used in research, business, and product innovation.

In China, this data often takes another form: hierarchical, discrete, strictly defined, and with centralized interpretation authority.

This is not a technical preference in a single field, but a systemic logic of technology and governance.

Three Global Paths

Placing China in a global context, we can see that there are roughly three different paths worldwide regarding “how public data and technical standards are opened.”

Engineering-Open Type: Standards and Ecosystem First

Represented by the US and some European countries, the core features of this system are:

Public data prioritized as infrastructure
Standards and interfaces come first
Encourages engineering autonomy and ecosystem evolution
Tolerates model inference and uncertainty

This path directly shaped the global landscape of foundational software and infrastructure-level open source. Linux, Kubernetes, and the cloud native system are essentially products of openness at the rules layer.

Governance-Sovereignty Type: Control and Auditability First

Represented by China, this path emphasizes:

Sensitivity of spatial and public data
Data as part of governance capability
Standards, definitions, and release methods are highly bound
Emphasizes traceability, accountability, and controllability

In this system, “point data” is not a sign of technological backwardness, but a governable technical form. When a technical system is designed as a governance system, its primary goal is not reusability, but controllability.

Compromise-Coordinated Type: Cautious Openness, Engineering Internationalization

Some countries try to find a balance between the two, maintaining caution in spatial data while being highly internationalized in engineering and industry. This shows that the difference is not about being advanced or backward, but about different objective functions.

The following diagram compares the core characteristics, typical cases, and advantages/challenges of these three paths from a global perspective. The “Engineering-Open Type” on the left shapes the global infrastructure software landscape through standards and ecosystems; the “Governance-Sovereignty Type” in the middle emphasizes data sovereignty and security controllability but has limitations in influence at the rules layer; the “Compromise-Coordinated Type” on the right attempts to find a balance between security and openness. The divide between these three paths directly affects the infrastructure competition landscape of various countries in the AI era.

Figure 3: Global Perspective: Three Paths for Public Data and Technical Standards

The Essence of “Point” vs. “Area” in Air Quality

Among all spatial public data, air quality is an ideal observation window:

Does not directly involve military or core economic security
Highly visible, updated daily, and perceptible to everyone

China does not lack air quality data; on the contrary, the density of monitoring stations is among the highest in the world. The real difference lies in:

Whether interpolation is allowed
Whether model inference is allowed
Whether platforms are allowed to reinterpret the data

“Point” means authenticity and traceability; “area” means models, inference, and redistribution of interpretive authority. This is precisely the watershed between technical standards and data sovereignty.

The following diagram compares two different technical paths. The left side, “Governance-Sovereignty Type,” emphasizes data traceability and controllability, using discrete point-based data presentation. The right side, “Engineering-Open Type,” allows model interpolation and inference, providing more user-friendly experience through continuous area-based coverage. The essence of this difference lies not in the level of technical capability, but in the different choices made between data sovereignty, governance capability, and open ecosystems.

Figure 4: Technical Standards and Sovereignty Divide in Spatial Data Presentation

The Amplification Effect in the AI Era

With the above logic in mind, many phenomena in the AI era become less confusing.

For example:

Why are Chinese AI companies more willing to open source large language model (LLM) weights, while American companies have clearly shifted toward closed source in recent years?
Why is foundational software and infrastructure-level open source still mainly led by the US?

The key is not “whether to open source,” but “which layer is open sourced.”

Model weights are static, declarable assets
Infrastructure, runtimes, protocols, and standards are dynamic, evolving system rules

Open sourcing weights is essentially openness at the asset layer; infrastructure-level open source means relinquishing control over operating rules and interpretive authority.

The following diagram compares two different layers of AI open source. The left side shows “Model Weight Layer Open Source,” which is a typical feature of Chinese path—opening static digital assets with low cost and controllable risk, but not involving rule-making. The right side shows “Infrastructure Layer Open Source,” which is a core strategy of US path—by open sourcing development tools, protocol standards, runtimes, and compute scheduling and other infrastructure, defining how AI is used, thereby mastering ecosystem rules and interpretive authority. Key insight: Open sourcing model weights does not equal mastering AI ecosystem, and the real competitive focus is shifting to the infrastructure layer of “how AI runs.”

Figure 5: Two Layers of AI Era Open Source: Model Weights vs Infrastructure

The US Approach: Focusing on Rules and Runtime Layers

In the past year or two, US-led AI open source and ecosystem initiatives have shown a highly consistent direction: not rushing to open source the strongest models, but focusing on defining “how AI is used.”

The Linux Foundation established AAIF (Agentic AI Foundation), focusing on AI infrastructure, standards, and toolchain collaboration
Protocols like MCP (Model Context Protocol) aim to define common interaction methods between agents and tools/systems
Major tech companies are generally focusing on APIs, platforms, runtimes, and ecosystem binding

The commonality of these actions: competing in model capability, but controlling the usage rules.

China’s Shift: From Model-Oriented to Infrastructure-Oriented

It is important to emphasize that this difference does not mean China is unaware of the issue.

Whether in policy discussions or within industry and research institutions, the risk of “only open sourcing models without controlling infrastructure and standard dominance” has been repeatedly discussed.

The real challenge lies in how to achieve a directional shift within the existing governance logic and risk framework. This shift has already appeared in some concrete practices.

Exploration and Practice at the Infrastructure Layer

In the AI era, infrastructure often starts with the most engineering-driven problems.

HAMi Project

Projects like HAMi do not focus on model capability, but on:

Abstraction, allocation, and isolation of GPU resources
How multi-tenant AI workloads are run
How computing power transitions from hardware assets to governable system resources

The significance of such projects is not about being “SOTA,” but about entering the domain of “how AI runs.”

AI Runtime Reconstruction from a System Software Review

Exploration at the research institution level is also noteworthy. The FlagOS initiative by the Beijing Academy of Artificial Intelligence is a clear signal: AI is being redefined as a system software issue, not just a model or algorithm problem.

Long-Term Tech Stack Investment by Industry Players

In the industry, Huawei’s strategy reflects a similar direction: not simply open sourcing models, but attempting to build a complete, controllable AI tech stack, from computing power to frameworks, platforms, and ecosystems. This is a slower, heavier, but more infrastructure-competitive path.

Realistic Assessment: The Starting Point of AI Infrastructure Competition

Taking a longer view, we find an easily overlooked fact:

At the level of AI infrastructure and infra-level open source, there is no settled pattern between China and the US.

The US advantage lies in:

Mature engineering culture
Standard organizations and foundation mechanisms
High proficiency in openness at the rules layer

China’s variables include:

Huge AI application scenarios
Extreme demand for computing power and system efficiency
Ongoing directional adjustments

The real uncertainty is not “whether we can catch up,” but whether it is possible to gradually open up space for engineering autonomy and standard co-construction while maintaining governance bottom lines.

Summary

The “points” and “areas” of air quality, model weights and the world of operations—behind these appearances lies not a simple technical route dispute, but how a country finds its own balance between openness, standards, and sovereignty.

In the AI era, this issue will not disappear, but will become more concrete and more engineering-driven. And this is precisely where there are still opportunities for China’s AI infrastructure open source.

Joining Dynamia: Embarking on a New Journey in AI Native Infrastructure

Jimmy Song — Wed, 07 Jan 2026 07:49:21 +0000

Compute governance is the critical bottleneck for AI scaling. From hardware consumption to core asset, this long-undervalued path needs to be redefined.

Figure 1: Dynamia.ai

A New Beginning

I have officially joined Dynamia as Open Source Ecosystem VP, responsible for the long-term development of the company in open source, technical narrative, and AI Native Infrastructure ecosystem directions.

Why I Chose Dynamia

I chose to join Dynamia not because it’s a company trying to “solve all AI problems,” but precisely the opposite—it’s because Dynamia focuses intensely on one unavoidable, yet long-undervalued core issue in AI Native Infrastructure: compute, especially Graphics Processing Units (GPU), are evolving from “technical resources” into infrastructure elements that require refined governance and economic management.

Through years of practice in cloud native, distributed systems, and AI infrastructure (AI Infra), I’ve formed a clear judgment: as Large Language Models (LLM) and AI Agents enter the stage of large-scale deployment, the real bottleneck limiting system scalability and sustainability is no longer just model capability itself, but how compute is measured, allocated, isolated, and scheduled, and how a governable, accountable, and optimizable operational mechanism is formed at the system level. From this perspective, the core challenge of AI infrastructure is essentially evolving into a “resource governance and Token economy” problem.

About Dynamia and HAMi

Dynamia is an AI-native infrastructure technology company rooted in open source DNA, driving efficiency leaps in heterogeneous compute through technological innovation. Its leading open source project, HAMi (Heterogeneous AI Computing Virtualization Middleware), is a Cloud Native Computing Foundation (CNCF) sandbox project providing GPU, NPU and other heterogeneous device virtualization, sharing, isolation, and topology-aware scheduling capabilities, widely adopted by 50+ enterprises and institutions.

Dynamia’s Technical Approach

In this context, Dynamia’s technical approach—starting from the GPU layer, which is the most expensive, scarcest, and least unified abstraction layer in AI systems, treating compute as a foundational resource that can be measured, partitioned, scheduled, governed, and even “tokenized” for refined accounting and optimization—aligns highly with my long-term judgment on AI-native infrastructure.

This path doesn’t use “model capabilities” or “application innovation” as selling points in the short term, nor is it easily packaged into simple stories. However, with rising compute costs, heterogeneous accelerators becoming the norm, and AI systems moving toward multi-tenant and large-scale operations, these infrastructure-level capabilities are gradually becoming prerequisites for the establishment and expansion of AI systems.

Future Focus

As Dynamia’s Open Source Ecosystem VP, I will focus on technical narrative of AI-native infrastructure, open source ecosystem building, and global developer collaboration, promoting compute from “hardware resource being consumed” to governable, measurable, and optimizable AI infrastructure core asset, laying the foundation for the scaling and sustainable evolution of AI systems in the next stage.

Summary

Joining Dynamia is an important milestone in my career and a concrete action demonstrating my long-term optimism about AI-native infrastructure. Compute governance is not a short-term trend that yields quick results, but an infrastructure proposition that cannot be bypassed for AI large-scale deployment. I look forward to exploring, building, and landing solutions on this long-undervalued path with global developers.

References

Running Parallel AI Agents on My Mac: Hands-On with Verdent's Standalone App

Jimmy Song — Sun, 04 Jan 2026 02:25:48 +0000

I’ve been spending more time recently experimenting with vibe coding tools on real projects, not demos. One of those projects is my own website, where I constantly tweak content structure, navigation, and layout.

During this process, I started using Verdent’s standalone Mac app more seriously. What stood out was not any single feature, but how different the experience felt compared to traditional AI coding tools.

Figure 1: Verdent Standalone App UI

Verdent doesn’t behave like an assistant waiting for instructions. It behaves more like an environment where work happens in parallel.

A Different Starting Point: Tasks, Not Chats

Most AI coding tools begin with a conversation. Verdent begins with tasks.

When I opened my website repository in the Verdent app, I didn’t start with a long prompt. I created multiple tasks directly: one to rethink navigation and SEO structure, another to explore homepage layout improvements, and a third to review existing content organization.

Each task immediately spun up its own agent and workspace. From the beginning, the app encouraged me to think in parallel, the same way I normally would when sketching ideas on paper or jumping between files.

This framing alone changes how you work.

Built for Multitasking, Without Losing Context

Switching contexts is unavoidable in real development work. What usually breaks is continuity.

Verdent handles this well. Each task preserves its full context independently. I could stop one task mid-way, switch to another, and come back later without re-explaining the problem or reloading files.

For example, while one agent was analyzing my site’s navigation structure, another was exploring layout options. I moved between them freely. Nothing was lost. Each agent remembered exactly what it was doing.

This feels closer to how developers think than how chat-based tools operate.

Safe Parallel Coding with Workspaces

Parallel work only becomes truly safe when code changes are isolated. When parallelism moves from discussion to actual code modification, risk management becomes essential.

Verdent solves this with Workspaces. Each workspace is an isolated, independent code environment with its own change history, commit log, and branches. This isn’t just about separation—it’s about making concurrent code changes manageable.

What this means in practice:

Multiple tasks can write code simultaneously
Changes remain isolated from each other
If conflicts arise, they’re visible and cleanly resolvable

I intentionally let different agents operate on overlapping parts of my project: one modifying Markdown content and links, another adjusting CSS and layout logic. Both ran in parallel. No conflicts emerged. Later, I reviewed the diffs from each workspace and merged only what made sense.

This kind of isolation removes significant anxiety from AI-assisted coding. You stop worrying about breaking things and start experimenting more freely, knowing that each change exists in its own contained environment.

Parallel Agent Execution Feels Like Delegation

Parallelism doesn’t mean that all agents complete the same phase of work at the same time—instead, by isolating and overlapping phases, what was once a strictly sequential process is compressed into a more efficient, collaborative mode.

In Verdent, each agent runs in its own workspace, essentially an automatically managed branch or worktree. In practice, I often create multiple tasks with different responsibilities for the same requirement, such as planning, implementation, and review. But this doesn’t mean they all complete the same phase simultaneously.

These tasks are triggered as needed, each running for a period and producing clear artifacts as boundaries for collaboration. The planning task generates planning documents or constraint specifications; the implementation task advances code changes based on those documents and produces diffs; the review task, according to the established planning goals and audit criteria, performs staged reviews of the generated changes. By overlapping phases around artifacts, the originally strict sequential process is compressed into a workflow that more closely resembles team collaboration.

The value of splitting into multiple tasks is not parallel execution, but parallel cognition and clear collaboration boundaries.

While it’s technically possible to put multiple roles into a single task, this causes planning, implementation, and review to share the same context, which weakens role isolation and the auditability of results.

Configurability and Design Trade-offs

Beyond the workflow model itself, Verdent exposes a surprisingly rich set of configurable capabilities.

It allows users to customize MCP settings, define subagents with configurable prompts, and create reusable commands via slash (/) shortcuts. Personal rules can be written to influence agent behavior and response style, and command-level permissions can be configured to enforce basic security boundaries. Verdent also supports multiple mainstream foundation models, including GPT, Claude, Gemini, and K2. For users who prefer a lightweight coding experience without a full IDE, Verdent offers DiffLens as an alternative review-oriented interface. Both subscription-based and credit-based pricing models are supported.

Figure 2: Verdent Settings

That said, Verdent makes a clear set of trade-offs. It is not built around tab-based code completion, nor does it offer a plugin system. If it did, it would start to resemble a traditional IDE - which does not seem to be its goal. Verdent is not designed for direct, fine-grained code manipulation; most changes are mediated through conversational tasks and agent-driven edits. This makes the experience clean and focused, but it also means that for large, highly complex codebases, Verdent may function better as a complementary orchestration layer rather than a full-time development environment.

Where Verdent Fits Today

There are many AI-assisted coding tools emerging right now. Some focus on smarter editors, others on faster generation.

Verdent feels different because it focuses on orchestration, not just assistance.

It doesn’t try to replace your editor. It sits one level above, coordinating planning, execution, and review across multiple agents.

That makes it particularly suitable for exploratory work, refactoring, and early-stage design - exactly the kind of work I was doing on my website.

Final Thoughts

Using Verdent’s standalone app didn’t just speed things up. It changed how I structured work.

Instead of doing everything sequentially, I started thinking in parallel again - and letting the system support that way of thinking.

Verdent feels less like an AI feature and more like an environment that assumes AI is already part of how development happens.

For developers experimenting with AI-native workflows, that shift is worth paying attention to.

2025 Annual Review: The Transformation Journey from Cloud Native to AI Native

Jimmy Song — Wed, 31 Dec 2025 10:02:01 +0000

The waves of technology keep evolving; only by actively embracing change can we continue to create value. In 2025, I chose to move from Cloud Native to AI Native—this year marked a key turning point for personal growth and system reinvention.

2025 was a turning point for me. This year, I not only changed my technical direction but also the way I approach problems. Moving from Cloud Native infrastructure to AI Native Infrastructure was not just a migration of content, but an upgrade in mindset.

Figure 1: Farewell 2025!

This year, I conducted a large-scale refactoring of the website and systematically organized the content. Beyond the technical improvements, I want to share my thoughts and changes throughout the year.

A Bold Shift: Embracing the AI Native Era

At the beginning of 2025, I made an important decision: to reposition myself from a Cloud Native Evangelist to an AI Infrastructure Architect. This was not just a change in title, but a strategic transformation after careful consideration.

As I witnessed the surge of AI technologies and the rise of Agent-based applications reshaping software, I realized that clinging to the boundaries of Cloud Native might mean missing an era. So, I systematically adjusted the website’s content structure, shifting the focus toward AI Native Infrastructure.

This transformation was not about abandoning the past, but extending forward from the foundation of Cloud Native. Classic content like Kubernetes and Istio remains and is continuously updated, but new topics such as AI Agent and the AI Native Landscape have been added, forming a more complete knowledge map.

Content Creation: From Technical Details to Ecosystem Perspective

AI Agent: Building Systematic Knowledge

Agents represent a major evolution in software for the AI era. When I tried to understand Agent design principles, I found fragmented information everywhere but lacked a systematic knowledge base.

So I created content that analyzes the Agent context lifecycle and control loop mechanisms, summarizing several proven architectural patterns. To make complex knowledge easier to digest, I organized it into logical sections so readers can learn step by step.

AI Tool Ecosystem: Mapping the Open Source Landscape

AI tools and frameworks are emerging rapidly, with new projects appearing daily. To help readers quickly grasp the ecosystem, I built a comprehensive AI OSS database.

This database covers everything from Agent frameworks to development tools and deployment services. I not only included active projects but also established an archive mechanism, preserving detailed information on over 150 historical projects. More importantly, I developed a scoring system to objectively evaluate projects across dimensions like quality and sustainability, helping readers decide which tools are worth investing time in.

Blogging: Capturing Technology Trends Faster

In 2025, I wrote over 120 blog posts. Compared to previous years, these articles focused more on observing and reflecting on technology trends, rather than just technical tutorials.

I started paying attention to deeper questions: How will AI infrastructure evolve? What does Beijing’s open source initiative mean for the AI industry? What ripple effects might a tech acquisition trigger? These articles allowed me and my readers to not only see “what” technology is, but also “why” and “what’s next.”

User Experience: Making Knowledge Easier to Discover and Consume

No matter how good the content is, if it can’t be easily found and read, its value is greatly diminished. In 2025, I invested significant effort into website functionality, with one goal: to provide readers with a smoother reading experience.

Comprehensive Search Upgrade

As the volume of content grew, the original search function could no longer meet demand. I redesigned the search system to support fuzzy search and result scoring, and optimized index loading performance. More importantly, the new search interface is more user-friendly, supporting keyboard navigation and category filtering so users can find what they want faster.

Multi-Device Experience Optimization

Mobile reading experience has improved significantly. I refactored the mobile navigation and table of contents, making reading on phones much smoother. Dark mode is now more refined, fixing several display issues and ensuring images and diagrams look good on dark backgrounds.

Efficiency Revolution in Content Distribution

A major change was optimizing the WeChat Official Account publishing workflow. Previously, publishing website content to WeChat required manual handling of many details; now, it’s almost one-click export. This workflow automatically processes images, metadata, styles, and all details, reducing a half-hour task to just a few minutes.

Additionally, I added a glossary feature for technical term highlighting and tooltips; improved SEO and social sharing metadata; and cleaned up outdated content. These seemingly minor improvements quietly enhance the user experience.

Content Evolution: More Dimensional Knowledge Expression

Looking back at content creation in 2025, I found clear changes in several dimensions.

From Tutorials to Observations

Early content leaned toward technical tutorials and practical guides, showing “how to do.” This year, I focused more on “why” and “what are the trends.” I wrote more technology trend analyses, ecosystem maps, and in-depth case studies. These may not directly teach you how to use an API, but they help you understand the direction of technological evolution.

From Chinese to Bilingual

AI is a global wave and cannot be limited to the Chinese-speaking world. In 2025, I wrote bilingual documentation for almost all new AI tools, and important blog posts also have English versions. This increased the workload, but allowed the content to reach a broader audience.

From Text to Multimedia

Text is efficient, but not all knowledge is best expressed in words. This year, I used many architecture and schematic diagrams to explain complex concepts, adding 59 new charts. These visual elements lower the barrier to understanding, making abstract concepts more intuitive. I also optimized image display in dark mode to ensure consistent visual experience.

Development Approach: Embracing AI-Assisted Programming

2025 was not only a year of shifting content themes toward AI, but also a year of deep practice in AI-assisted programming.

I developed a VS Code plugin and created many prompts to automate repetitive tasks. I experimented with various AI programming tools and settled on a toolchain that suits me. I even migrated the website to Cloudflare Pages and used its edge computing services to develop a chatbot. These practices greatly improved development efficiency, giving me more time to focus on thinking and creating rather than mechanical coding.

This made me realize: AI will not replace developers, but developers who use AI well will replace those who do not. I also shared more insights to help others master AI-assisted programming.

Looking Ahead to 2026: Keep Moving Forward

Looking back at 2025, the site underwent a profound transformation—from a Cloud Native tech blog to an AI infrastructure knowledge base. But this is just the beginning, not the end.

Looking forward to 2026, I plan to continue deepening in several areas:

Enhancing the knowledge system: Continue to supplement GPU infrastructure and AI Agent content, especially practical cases and performance tuning knowledge.
Tracking ecosystem evolution: AI tools and frameworks iterate rapidly; I need to keep up with this fast-changing ecosystem and update content in a timely manner.
Deepening engineering practice: Share more practical AI engineering experience to help readers turn theory into practice.
Exploring knowledge connections: Consider building a knowledge graph to connect different content sections, providing smarter navigation and recommendations.

Summary

2025 was a year of change and growth. From Cloud Native to AI Native, from technical practice to ecosystem observation, both the content and functionality of the site have made qualitative leaps.

What makes me happiest is that this transformation allowed me and my readers to stand at the forefront of the technology wave. We are not just learning new technologies, but thinking about how technology changes the world and the way we write software.

The waves of technology keep evolving; only by actively embracing change can we continue to create value. Thank you to every reader for your companionship and support. I look forward to sharing more insights and practices in 2026.

Further Reading:

AI Native Landscape
2025 Blog Posts

The Butterfly Effect After Manus Was Acquired by Meta

Jimmy Song — Tue, 30 Dec 2025 03:30:51 +0000

The success or failure of AI applications often lies not in the technology itself, but in the ability to scale delivery and create a closed loop.

Figure 1: The Butterfly Effect After Manus Was Acquired by Meta

When “Those Who Discuss It” Are Not “Those Who Pay for It”

On December 30, 2025, a piece of news went viral: Manus was acquired by Meta for billions of dollars (Manus Joins Meta for Next Era of Innovation). This startup, founded in China and under pressure from tech giants since its inception, completed a whirlwind journey in less than a year—from explosive growth, relocating to Singapore, to being acquired by a global giant.

According to Manus’s official statement, its products and subscriptions will continue to be available via the app and website, and the company will remain operational in Singapore. The team will join Meta to provide general Agent capabilities for Meta’s consumer and enterprise products (including Meta AI).

Rather than focusing on “who won,” I’m more interested in the chain reaction this event triggered: it activated completely opposite judgment systems among different groups, and this split is reshaping the growth paths and strategies for AI applications and startups.

Two Public Opinion Arenas: Blessings and Doubts Coexist

After Manus was acquired, the mainstream sentiment in social circles was one of congratulations and excitement. Many saw it as a stellar example of a Chinese team going global—achieving remarkable results in the most competitive field in a very short time.

Meanwhile, the comment sections of public accounts became “venting valves for counter-narratives,” with skepticism centering on three main points:

Whether the technology has real barriers (e.g., “there are countless similar products,” “it’s not hard for big companies to build their own”).
Valuation and bubble concerns (e.g., “another case of the AI bubble”).
Distrust in the buyer’s judgment (e.g., “giants making desperate bets,” “history repeating itself”).

This divergence isn’t about who understands AI better, but about different evaluation frameworks: social circles focus on “trajectory and outcome,” while comment sections focus on “legitimacy and worthiness.”

Many people are impressed by Manus’s marketing buzz and controversies, which can lead to skepticism. But if it achieved a “strict $100M ARR” in 10 months, one fact is clear: its revenue doesn’t depend on broad consensus, but comes from a highly concentrated group of global users with strong willingness to pay.

Manus’s core user profile is closer to “individuals as production units,” including freelancers, indie developers, independent researchers, and key deliverers in small and medium businesses. They don’t care about debates over “wrapping” or not; they care about “can I deliver end-to-end tasks,” and “can this help me hire one less person, work fewer late nights, or avoid juggling ten tools.”

This leads to a counterintuitive phenomenon: those who discuss the most may not pay, while those who pay steadily are often silent.

For these users, tools are not identity badges—they are profit levers.

Three Lessons for Entrepreneurs: The Growth Paradigm in the AI Application Era Has Changed

Based on the above, the Manus case offers three lessons for entrepreneurs:

Growth No Longer Equals Positive Reviews

AI applications can commercialize first and build consensus later. Public opinion can remain divided for a long time, but cash flow doesn’t wait for unified recognition.

“Heavy Marketing” Is Becoming a Capability, Not a Stigma

As foundational models and capabilities spread rapidly, differentiation is quickly erased. Being seen, understood, and paid for is itself part of the moat. Not all marketing deserves respect, but “distribution and mindshare” have become unavoidable battlegrounds for AI applications.

Globalization Is No Longer a Bonus, but May Be a Survival Strategy

From payment willingness, compliance boundaries, talent density to valuation systems, market structure means many teams “can only complete the loop overseas.” It’s not romantic, but it’s reality.

A Personal Reflection

As someone long engaged in cloud native and AI infrastructure, I’m used to evaluating products by their “technical barriers.” But cases like Manus remind me: at the AI application layer, barriers may not first appear in models or code, but often in organizational speed, productization capability, delivery loop, and distribution efficiency.

When a system can reliably turn “capability” into “results,” it has built a commercial moat—even if its tech stack doesn’t meet outsiders’ ideals of “purity.”

The biggest butterfly effect of Manus being acquired by Meta may not be the deal itself, but making more entrepreneurs realize: in the AI era, the winning move is shifting from “what model you use” to “whether you can deliver results at scale.”

Summary

The acquisition of Manus by Meta is not just a convergence of capital and technology, but also a microcosm of the changing growth paradigm in the AI application era. For entrepreneurs, understanding and mastering “user structure,” “distribution capability,” and “global closed loops” will be key to future competition.

AI Infra Open Source in China: Analysis of Beijing and Shanghai's Plans

Jimmy Song — Thu, 25 Dec 2025 10:01:13 +0000

Institutionalized open source marks a new starting point for China’s AI Infra, but true breakthroughs and risks lie in the engineering and governance details.

Perspective on Beijing and Shanghai’s Open Source Plans

Using the simultaneous release of open source ecosystem plans by Beijing and Shanghai as a lens, and drawing on China’s past foundation practices and international open source governance experience, this article explores the real opportunities, structural constraints, and potential risks as AI Infrastructure (AI Infra, Artificial Intelligence Infrastructure) enters a new phase of institutionalized open source.

Figure 1: Beijing and Shanghai successively launch open source ecosystem construction plans

Why Compare Beijing and Shanghai Together

It is rare for me to write an article solely because of a local policy document. However, during Christmas, both Beijing and Shanghai’s Bureaus of Economy and Information Technology released their respective open source ecosystem construction plans:

This time, the fact that both cities released their plans on the same day sends a signal worth serious attention: China is attempting to advance open source in a more systematic and institutionalized way, especially regarding open source capabilities related to AI Infra.

If you only look at Beijing’s plan, it is easy to interpret it as a local industrial policy upgrade. But when you consider both Beijing and Shanghai’s plans together, it looks more like a clearly defined “dual-center structure.”

The question is no longer whether to develop open source, but:

In the AI era, what institutional forms, engineering paths, and governance models will open source take?

Open Source as “Industrial Infrastructure Engineering”

Both Beijing and Shanghai’s plans reflect a highly consistent judgment:

Open source is no longer seen as a spontaneous community activity, but as an industrial infrastructure capability that requires systematic construction.

This is especially evident in the field of AI Infra.

Issues such as computing power scheduling, model evaluation, toolchains, data elements, license compliance, and supply chain security—previously hidden in “engineering details”—are now systematically incorporated into policy language for the first time. This at least shows that decision-makers have realized:

AI competition is not only about model parameter scale
It is even more about toolchains, infrastructure, evaluation systems, and engineering capabilities
These capabilities are naturally more suitable for building public foundations through open source

In this respect, Beijing and Shanghai are highly aligned.

Two Open Source Paths: Infra vs. Platform

When we zoom in, the differences between the two plans become clear.

Beijing: “Foundation-Oriented” Open Source Path for AI Infra

Beijing’s plan focuses on:

Heterogeneous computing power scheduling
Model evaluation toolchains
Data elements and data governance
RISC-V software-hardware collaboration
SBOM, license compatibility, open source compliance
Supply chain security and industrial resilience

This is a typical perspective of “treating AI as an infrastructure problem.”

It is less concerned with the number of projects or community size, and more with:

Whether reusable engineering capabilities can be formed
Whether these can be trusted by industry and government over the long term
Whether they can stand up to scrutiny in terms of security, compliance, and governance

To some extent, Beijing is answering the question:

How can open source become a “governable, auditable, and scalable public capability”?

Shanghai: “Scale and Internationalization” Path for AI Platform

In contrast, Shanghai’s plan has a different focus:

Building an international open source community for artificial intelligence
Covering the entire platform chain from development, training, testing, hosting, to operation
Overseas sites, multilingual support, international activities
Resource linkage through computing vouchers and model vouchers
“Open source platform first release / global simultaneous release” dual-release mechanism
Clear targets for community, enterprise, and developer scale

Shanghai cares more about:

How open source can achieve scale effects
How it can support the growth of commercial enterprises
How it can be seen and adopted globally

This is a path of “treating open source as a global digital product and platform capability.”

Together: A Complete but Tension-Filled Structure

When viewed together, Beijing and Shanghai’s plans form a more complete picture:

Beijing is responsible for “making open source solid,” while Shanghai is responsible for “taking open source global.”

Structurally, this is a clear division of labor:

Beijing focuses on institutions, governance, and foundational capabilities
Shanghai focuses on community, commercialization, and international communication

These two paths are not in conflict; in theory, they are even complementary. The real question is whether they can form positive feedback in practice, rather than operating in silos.

Cautious Attitude Toward “Institutionalized, Platformized Open Source”

Precisely because both plans are so “systematic,” I am even more cautious.

The reason is simple: this is not China’s first attempt to promote open source through foundations, associations, or platforms.

Over the past decade, we have seen similar paths repeatedly, and recurring structural problems:

The difficulty of establishing neutrality and multi-party trust is extremely high
There is a huge gap between showcase metrics (quantity, activities, certifications) and ecosystem strength
Commercialization and long-term maintenance mechanisms are hard to sustain

These problems will not disappear just because the plans are more comprehensive.

Four Risks to Watch Under the Dual Plans

If we are to “listen to their words and watch their actions,” I would focus on the following four risks:

Will Metrics Hijack Engineering Reality

When “internationally influential projects,” “star projects,” and “first-release projects” become hard metrics, will this induce packaging, migration, and short-term hype, rather than truly solving engineering problems?

Will It Slide Toward Platform Centralism

The long-term pattern of AI Infra is closer to a model that prioritizes protocols, standards, and interoperability. If it eventually evolves into “a few platforms concentrating resources and discourse power,” it may be efficient in the short term but will suppress external participation and international collaboration in the long run.

Is Internationalization Underestimated as an “Operational Issue”

True international collaboration is never just about language, sites, or events; it also involves governance structures, compliance boundaries, and supply chain trust.

Will Application Demonstrations Become One-Off Projects

If “first plans” and “computing vouchers” are just procurement tactics without continuous iteration and community feedback mechanisms, the long-term benefit to the ecosystem will be very limited.

What Are the “Hard Results” of AI Infra Open Source After Three Years

If we review the success of this round of institutionalized open source after three years, I would look for three types of results:

Whether de facto standards and interoperable ecosystems have emerged, including scheduling interfaces, evaluation benchmarks, Agent tool invocation protocols, and observability semantics.
Whether compliance and supply chain security have become public capabilities—SBOM, license compatibility, vulnerability monitoring—truly productized and service-oriented.
Whether a sustainable maintenance business mechanism has been established, allowing core maintainers to stay long-term, rather than relying on passion and subsidies.

If I were to use a North Star metric to measure the success of these plans, it would be the emergence of several outstanding open source commercial companies rooted in China and serving the world.

Summary

The open source ecosystem plans of Beijing and Shanghai mark a new phase of institutionalization and engineering for AI Infra open source in China. Over the next three years, the real achievements will not be about meeting targets, but about forming sustainable engineering capabilities, de facto standards, and maintenance mechanisms. Only through continuous participation and practice can open source become the public foundation of AI infrastructure.

References

From 2025 Onwards, Software Engineering Shifts from Code-Centric to Runtime and Cost-Centric

Jimmy Song — Wed, 24 Dec 2025 14:59:11 +0000

In 2025, the core of software engineering is no longer just about code itself, but about runtime controllability and cost governance. This shift is fundamentally reshaping the industry’s underlying logic.

Looking back at 2025, I became increasingly aware that this year was not about “code becoming unimportant,” but rather that the value coordinates of engineering have shifted as a whole. For more than a decade, software engineering has focused on code quality, architectural evolution, and delivery efficiency. But starting in 2025, the key to system success is shifting—towards whether the runtime is controllable and whether costs are governable.

This is not just a slogan, but a conclusion repeatedly validated by my real-world experiences throughout the year.

My 2025: From “Platform Engineering” to “Runtime Challenges”

In my annual review, I noted a clear change: I spent less time on “how to write a good system,” and more time on “how to keep the system running stably, reliably, and affordably.”

This shift in focus is a natural extension of a decade of cloud native evolution.

The following timeline diagram illustrates how my focus has changed over recent years:

Figure 1: My Focus Shift Timeline

My focus shifted from cloud native platform engineering to LLM application engineering, then to AI infrastructure, and finally to Agentic Runtime with governance and cost control.

When AI workloads truly enter business scenarios, the core challenges engineers face also change:

Are inference, training, and evaluation competing for the same compute pool?
Is GPU utilization consistently below expectations?
Does cost scale linearly and uncontrollably with concurrency?
Does the system have failure isolation and replay capabilities?

These issues go far beyond the code level.

Industry Consensus: AI Is Shifting the Focus of Engineering

By 2025, an industry consensus is emerging: AI is rewriting software engineering. But the real change is not happening in the IDE or code completion speed—it is reflected in the migration of engineering complexity.

Previously, complexity was concentrated in code and interfaces, and problems were solved through abstraction, refactoring, and testing.

Now, complexity has shifted to the runtime, resource, and cost layers, and must be addressed through scheduling, isolation, observability, and governance.

This is why the same AI tools:

Serve as “accelerators” for junior engineers
But act as “magnifiers” for senior engineers

AI tools amplify whether you truly understand how systems run in production.

Why “Cost” Becomes a First Principle

In traditional cloud native systems, low CPU utilization is often just an efficiency issue; but in AI systems, low GPU utilization is often a cash flow problem.

In 2025, I repeatedly encountered scenarios like:

Resources “seem insufficient,” but utilization is not actually high
Scaling up to solve queuing issues ends up increasing unit costs
The system lacks clear budget and quota boundaries, so throttling becomes the only way to stop the bleeding

The root cause of these phenomena is not model selection, but the lack of a runtime and cost control plane tailored for AI workloads.

The following flowchart visually illustrates the cyclical relationship between GPU resources and cost pressures:

Figure 2: GPU Resource and Cost Cycle in AI Systems

In AI systems, limited GPU supply leads to queuing and waiting, which causes throughput to drop. Attempts to solve this through blind scaling only increase unit costs and create budget pressure, ultimately forcing the adoption of finer scheduling and governance strategies.

Engineering problems ultimately manifest as cost issues.

The Rise of Agents: The Real Challenge Is at Runtime

In 2025, Agent (Intelligent Agent, Agent, Intelligent Agent) became a hot topic; by 2026, it will enter the “can it actually run” stage.

The challenge for Agents has never been about “how smart they are,” but rather:

Whether there are clear permission and data boundaries
Whether they run in an isolated execution environment
Whether they can be observed, evaluated, and replayed
Whether they are subject to explicit cost and budget constraints

These capabilities form the outline of Agentic Runtime (Agentic Runtime, Intelligent Agent Runtime) that I have been trying to clarify throughout the year.

The following flowchart shows the core capability layers of Agentic Runtime:

Figure 3: Agentic Runtime Capability Layers

Agentic Runtime builds from the foundation of Agents and workflows, connecting through orchestration and tool protocols, with the runtime managing state, memory, and evaluation. It provides secure execution environments (Sandbox and Policy), and ultimately implements a resource and cost control plane that unifies GPU, quota, and billing management.

Without a runtime, an Agent is just a demo; without cost constraints, an Agent is just a risk amplifier.

Outlook for 2026: The “Foundation” of Engineering Matters Again

Looking ahead to 2026, I remain cautiously optimistic.

I do not believe the future belongs to “those who write the best prompts,” but more likely to:

Those who understand runtime boundaries
Those who can govern compute as a constrained resource
Those who design AI systems as long-running systems

From 2025 onwards, software engineering is no longer code-centric, but runtime and cost-centric. This is not a regression, but a return: a return to being responsible for the whole system and for real-world constraints.

For me personally, this is both a year-end summary and the direction I will continue to invest in for the coming years.

Summary

In 2025, the focus of software engineering has shifted from code itself to runtime and cost governance. The rise of AI and Agents has not diminished the value of engineering, but has pushed complexity to a higher level. In the future, understanding runtime, managing compute and cost will become the new core competencies for engineers. I hope this year-end review provides some inspiration and reflection for fellow professionals.

From Cloud Native to AI Native: Why Kubernetes Is the Foundation for Next-Gen AI Agents

Jimmy Song — Wed, 24 Dec 2025 12:25:52 +0000

As a long-time practitioner in the cloud native field, I am increasingly convinced of one thing: AI Agents are not just a change in application form, but a migration of infrastructure paradigms.

As artificial intelligence evolves from demos and copilots to systems that truly take on tasks and responsibilities, AI Agents are becoming the new execution units in enterprise IT architectures. They not only “think,” but also act: they can invoke tools, access systems, and collaborate to achieve goals.

This raises an important question:

What kind of infrastructure should such systems run on?

In my view, Kubernetes remains a solid choice for large-scale scenarios—but only if we reimagine Kubernetes in an AI-native way.

Cloud Native Challenges for Production-Grade AI Agents

In real production environments, AI Agents expose infrastructure needs that are fundamentally different from traditional microservices. Agents are not “just another HTTP service”; they have three distinct characteristics:

Behavior is non-deterministic (driven by model inference)
Execution paths are dynamic (tool invocation cannot be fully enumerated in advance)
Decisions must be auditable, constrained, and reviewable

If we simply apply existing cloud native infrastructure, we quickly hit bottlenecks.

The following table summarizes the main challenges and risks AI Agents face in cloud native environments:

Challenge Category	Real Needs of Agents	What Happens If Missing
Policy & Security	Dynamic control of tool and data access based on context, identity, and task	Agents have “superuser” privileges, risks are uncontrollable
Observability	Not just “did it succeed,” but also why was this decision made	Hard to debug, hard to review, hard to hold accountable
Governance & Consistency	Platform-level guardrails enforce organizational policies	Each Agent could become a “shadow AI”

Table 3: Challenges and Risks for AI Agents in Cloud Native Environments

All these issues point to one conclusion:

AI Agents must be treated as first-class citizens in Kubernetes, not just ordinary workloads.

Core Architecture: Making Agents Native Kubernetes Objects

Looking back at the evolution of cloud native technologies, we’ve gone through similar stages:

Physical machines → Virtual machines
Virtual machines → Containers
Containers → Microservices
Microservices → Declarative, governable platforms

AI Agents are simply the next step.

A production-ready AI Agent architecture requires at least three layers:

Agent Orchestration Layer: Declaratively define Agents
Tool Service-ization Layer (MCP Services): Turn capabilities into governable services
AI Native Data Plane / Gateway: Unify policy, security, and protocols

Agent Orchestration Layer: Declarative Agent Management

Agents should no longer be “runtime objects” inside an SDK—they should be managed like Pods or Deployments.

Key concepts:

Agents as Kubernetes Resources

Agents are defined using CRD (CustomResourceDefinition)
Lifecycle managed via kubectl or GitOps
Agent models, tools, and policies are all explicitly declared

A typical Agent definition includes:

Agent logic (inference loop)
Model configuration (specifying which large language model to use)
Callable toolset

This closely mirrors how we once decomposed “applications” into Deployments, Services, and ConfigMaps.

Tool Service-ization Layer: MCP Services Are Essential

In Agent architectures, tools are where real “actions” happen.

Early MCP tools were often:

Local processes
Tightly coupled to a single Agent
Lacking versioning, permissions, and auditing

This is unsustainable in enterprise environments.

The Essence of MCP Service-ization

Tools → Remote services
Services → Kubernetes native workloads
Capabilities → Reusable, governable, auditable

This step is fundamentally similar to how we once turned scripts into microservices.

AI Native Gateway: The “Control Plane Entry” for the Agent World

As the number of Agents grows and tools/models diversify, connectivity itself becomes a system risk.

Traditional API Gateways do not understand scenarios like:

MCP
Agent-to-Agent (A2A) communication
Model invocation context

Thus, we need an AI native gateway dedicated to mediation and governance.

It must understand at least three types of traffic:

A2T: Agent → Tool
A2L: Agent → LLM
A2A: Agent ↔ Agent

And enforce, across these paths:

Identity and authorization
Policy and guardrails
Auditing and rate limiting

Architecture Overview

The diagram below illustrates the core layers and traffic paths of an AI-native system on Kubernetes:

Figure 3: AI Native Architecture Layers and Traffic Paths

Summary

AI Agents do not negate cloud native; on the contrary:

AI Agents are the natural extension of cloud native in the era of intelligence.

Declarative → Agent definitions
Service → MCP Services
Service Mesh → AI Native Gateway

If Kubernetes is the “automated factory,” then AI Agents are the intelligent workers who actually get things done.

And the AI native gateway is the security and governance system tailored for these intelligent workers.

This is not an optional architecture—it is the only path for AI to reach production.

AI Open Source Landscape: A One-Stop Guide to AI Project Navigation and Scoring System

Jimmy Song — Tue, 23 Dec 2025 08:34:05 +0000

The AI Open Source Landscape is not just a project directory, but an innovative attempt to bring transparency and quantifiability to the AI open source ecosystem.

Note: This article is intended for general readers and focuses on platform features and usage scenarios. If you want to see the technical details and formulas behind the scoring, please refer to: AI Project Scoring and Inclusion Criteria.

Project Background and Positioning

The AI Open Source Landscape aims to provide developers, researchers, and enterprise users with a one-stop navigation and evaluation platform for AI open source projects. With the rapid development of large language models (LLM, Large Language Model), multimodal models (Multimodal Model), and other AI technologies, the open source community has seen a surge of innovative projects. However, information is scattered and quality varies, making it difficult for users to filter and make decisions.

Figure 1: AI Open Source Landscape

The AI Open Source Landscape systematically collects mainstream AI open source projects. As of the time of writing, it has included 851 open source projects. This landscape combines a multi-dimensional scoring system to help users efficiently discover, compare, and select the most suitable AI tools and frameworks for their needs. The platform not only focuses on models themselves, but also covers datasets, inference engines, evaluation tools, application frameworks, and the entire ecosystem chain, striving to promote transparency, quantifiability, and sustainable development in the AI open source ecosystem.

Main Interface and Feature Highlights

The platform homepage presents project distribution in both landscape and list views, supporting category filtering, keyword search, and tag navigation to help users quickly locate target projects.

Figure 2: Open Source Project Detail Page

For general readers, the main experience points include:

Card view: One-sentence overview, star rating, and overall score for quick browsing and comparison.
Health card: Displays overall health and key dimensions (activity, community, influence, sustainability) on the project page or sidebar, with the latest update marked for easy assessment of maintenance status.
Detail page: Provides more background information, project links, and application scenarios to help you evaluate suitability for your needs.
Smart badges: Visually display labels such as “Active”, “New Project”, “Popular”, “Archived” on cards, helping you quickly capture key project features.

If you are interested in the specific rules for badge determination or scoring, detailed explanations are available on the Scoring Rules Page.

Scoring and Ranking Mechanism

The platform uses multi-dimensional scores to reflect the overall health and popularity of projects. The main dimensions include: Activity, Community, Quality, Sustainability, and the comprehensive Health score. These scores help you quickly judge whether a project is suitable for production or experimentation.

Data Sources and Update Mechanism

The platform’s data mainly comes from GitHub, project lists, official documentation, and community recommendations. We regularly and automatically synchronize and update metrics to ensure that the “last updated” and scores displayed on the interface reflect the current maintenance status of projects. Projects that have not been updated for a long time or are determined to be “inactive” are moved to the Archived Page. Archived projects remain searchable and retain historical scores, but will not appear in the default view of active rankings, making it easier for readers to focus on projects that are still maintained and active.

For general readers, the key points are:

The page displays key metrics and “last updated” time, helping you quickly judge whether a project is still maintained.
The AI Open Source Landscape continuously iterates on the scoring model to improve fairness and differentiation.

How to Contribute and Correct Data

If you want a project to be included or its data updated, you can:

Submit an AI Open Source Project Inclusion Request
Keep the project’s README, License, documentation, and other information complete in the repository to facilitate our data collection and assessment.
For faster synchronization or if you encounter data issues, contact the maintainers via project issues or raise a request in the site discussion area.

Typical Use Cases or User Feedback

The AI Open Source Landscape has been widely used in various scenarios such as AI developer selection, enterprise technology research, and academic studies. For example:

Developers can quickly filter models or tools that meet their needs through the platform, saving significant research time.
Enterprise technical teams use the ranking lists for competitor analysis and technology planning.
Educational and research institutions refer to the landscape to understand trends in the AI open source ecosystem, supporting course design and topic selection.

Some users have commented that the platform is “comprehensive, well-structured, and fair in scoring,” greatly improving the efficiency of AI project selection and learning. Community suggestions continue to drive ongoing improvements in platform features and content.

Summary

The AI Open Source Landscape systematically and quantitatively organizes the AI open source ecosystem: the backend worker is responsible for reliable data collection and scoring calculations (supporting backfill and migration), while frontend components handle fast rendering and visualization (including smart badges, health cards, and metric explanations).

If you want to learn more about the scoring details or participate in improvements:

The community is welcome to join in evaluation, backfilling historical data, and refining scoring rules, working together to make the AI open source ecosystem more transparent and sustainable.

Jimmy Song – Jimmy Song's Blog

Core Model Overview

Four-Layer Model

Model Interactions

Model Application Value

The Yin-Yang Layer: Dynamic Balance of System States

Three Typical Pairs of Yin-Yang Tensions

Expansion ↔ Constraint

Innovation ↔ Governance

Speed ↔ Stability

The Art of Yin–Yang Balance

Practical Cases

Dynamic Transformation of Yin–Yang States

Five Elements Layer: Classification and Collaboration of System Roles

Engineering Mapping of Five Elements

Water – Data Flow and Quality

Wood – Model Growth and Capability Expansion

Fire – Compute Conversion and Work Efficiency

Earth – Platform Support and Orchestration Governance

Metal – Hardware Constraints and Physical Boundaries

Five Elements Generation Relationships

Five Elements Overcoming Relationships

Five Elements Balance Diagnosis

Common Imbalance Patterns

Balance Principles

The Yun Layer: Stages and Cycles of System Evolution

Exploration Period (Initial Stage)

Platform Period (Growth Stage)

Scale Period (Mature Stage)

Rebalancing/Substitution Period (Renewal Stage)

Evolutionary Cycle

The Art of Following the Momentum

Error Examples

Stage Assessment Checklist

Qi Layer: Effective System Flow and Pressure Fields

The Essence of Qi: Overall State of Affairs

Two States of Qi

Qi Flow: System Active

Qi Stagnation: System Pathological

Qi’s Flow Path

Two Forms of Qi

Healthy Flow

Pressure Propagation

Application of Qi Layer in Operations

Not Just Whether There’s a Problem, But How It’s Trending

Qi Disorder Precedes Major Incidents

Strategies for Guiding Qi Flow

Architecture Level

Strategy Level

Agent System Special Attention

Qi Layer Monitoring Practices

Summary

Dynamic Relationship Modeling: Five Elements Flow Under Yin-Yang Balance

Yin-Yang × Five Elements: Intrinsic Tension of Elements

Five Elements × Qi: Dynamic Network of Flow

Yun × Yin-Yang Five Elements: Boundary Conditions for Stage Evolution

Comprehensive Analysis Case: GPU Scheduling Scenario

Solutions

Value of Dynamic Modeling

Practical Recommendations

Engineering Practice Guide: Architecture Decisions Guided by Theory

Balance Yin and Yang, Avoid Extremes

Complete the Five Elements, Identify and Fill Weaknesses

Follow the Trend, Align with the Movement

Observe Qi Field, Optimize Flow

Dynamic Adjustment, Continuous Rebalancing

Practice Case: Large-Scale GPU Training Cluster Optimization

Scenario Application Quick Reference Table

Summary

System Diagnosis Principles: Criteria for Health Status

Five-Dimensional Diagnosis Framework

Five Elements Balance Check

Diagnosis Method

Identify Problems

Typical Symptoms

Qi Flow Smoothness Check

Diagnosis Method

Qi Smooth vs. Qi Not Smooth

Diagnosis Points

Yin-Yang Dynamics Check