For GPU sharing to become a governable infrastructure capability, what has been missing was never partitioning—it was a verifiable correspondence between scheduling decisions and runtime isolation.
Background: Where This Thread Came From
Since the beginning of this year, my writing has followed a single thread: how GPUs evolve from exclusive, scarce hardware into a governable infrastructure capability. That thread has passed through several dimensions. Early in the year, when discussing open GPU scheduling, the focus was on DRA, CDI, and the structural questions of standards and lock-in avoidance. Later, HAMi v2.9 showed Kubernetes becoming the GPU control plane, bringing the debate over resource models and control planes to the surface. Then came two deep dives: one toward observability, from GPU to token in eight layers; one toward measurement, with GPU utilization “failing” us and the notion of Productive GPU-Hours. Last month, in the article written after HAMi joined CNCF Incubating, the topic was consensus: when there is more than one technical route, how do a community and its ecosystem make the choice?
Put these dimensions together and a clear structure emerges: scheduling decides how resources are divided, observability makes them visible, measurement makes them accountable, and consensus decides who has the final say. But one dimension has remained hanging: enforcement. Once resources are allocated, what stops a workload from exceeding its share at runtime? Governance without enforcement is merely advice. This is precisely why multi-tenant users dare not share GPUs.
The adoption of HAMi-core by the NVIDIA KAI Scheduler lands squarely on this gap. This article discusses how the enforcement link is being closed, and what that means for the entire thread of compute governance.
Treating the GPU as a Layered Stack
To understand GPU governance, first decompose the GPU into a layered stack from the Kubernetes perspective. A GPU is not a single resource; it spans at least five layers: scheduling and allocation, Kubernetes device resources, container device injection, node accelerator software, and physical hardware. The essence of governance is to give every layer a clear, single owner. This layering is not tied to any one vendor: swap NVIDIA for Ascend or AMD and the structure holds—only the concrete components in each layer change.
The five-layer GPU software stack, from scheduling and allocation down to physical hardware:
Taking the most representative ecosystem, NVIDIA, as an example: the GPU Operator covers the middle three layers, turning a GPU-equipped node into a standardized container runtime node; Ascend’s NPU suite and AMD’s GPU Operator play the same role in their respective ecosystems. But this layer does not decide which queue a job should enter or whether gang scheduling applies, nor does it enforce memory caps at runtime—the same holds for NVIDIA’s CUDA layer as for Ascend’s CANN layer. Mixing layers together in discussion is the most common conceptual mistake in GPU governance.
The Long-Missing Link: Runtime Enforcement
Within this stack, the point where GPU sharing has long been stuck is clear: the scheduling layer can allocate, the device resource layer can declare, but nothing enforced those allocations at runtime. A container declares how much memory it needs, the scheduler accounts for it accordingly, yet the runtime cannot stop the container from blowing past its declaration and consuming the entire card’s memory.
The result is that multi-tenant users dare not truly share GPUs, and expensive accelerators can only be consumed exclusively. This is not a defect of any particular scheduler; it is a missing “isolated execution” across the entire chain, along with the observability to match. GPU sharing has therefore long stopped at “allocatable” without reaching “governable”.
Three Systems, Three Layers of Ownership
In practice, the most common confusion is treating GPU Operator, KAI Scheduler, and HAMi as three comparable products. In reality their responsibilities barely overlap—each owns a different layer of the stack. The following uses the NVIDIA ecosystem as the example, but the same ownership structure holds for Ascend, AMD, and other vendors.
| System | Layer owned | Role in GPU sharing |
|---|---|---|
| GPU Operator | Node GPU software lifecycle (Driver, Container Toolkit, CDI, DCGM) | Makes the node a usable NVIDIA container runtime node |
| KAI Scheduler | Scheduling policy, queues, fairness, and shared accounting | Decides how GPUs are allocated and shared |
| HAMi Core | Runtime memory and compute isolation | Turns “cooperative accounting” into “runtime enforcement” |
There is a governance principle here that is easy to overlook: a node must have one and only one owner of its device resources. The real risk is not a missing component, but two systems each believing they own the same layer. For example, if the NVIDIA Device Plugin and the HAMi Device Plugin both register nvidia.com/gpu on the same node, they conflict outright. Once ownership is clearly drawn, these three systems are not mutually exclusive—instead they form an AI infrastructure stack with clean boundaries.
Scheduling and Isolation Can Now Be Reconciled
HAMi-core entering the NVIDIA KAI Scheduler closes exactly the missing link. There is a technical fact often overlooked: KAI Scheduler’s GPU sharing does only cooperative accounting by default—the official documentation states plainly that it does not enforce memory limits, nor does it isolate the memory usage of different processes. What HAMi Core does is turn that “cooperative accounting” into runtime enforcement.
The scheduler makes allocation decisions; HAMi Core intercepts CUDA calls at runtime to enforce memory limits—for the first time, the two can be reconciled. I introduced this integration in the HAMi community here; I later verified it on GKE—allocations within quota run normally, requests beyond quota are rejected outright, and the reproduction steps are documented as a complete lab. Sharing thus shifts from a trust-based gentlemen’s agreement to a verifiable contract.
The four-layer verification of GPU sharing: after scheduling, device allocation, and visibility are all correct, isolation enforcement is the final layer:
Verify Hard Limits, Not Just Displayed Values
A common pitfall: seeing the container display only its partitioned memory via nvidia-smi (npu-smi on Ascend) and concluding that isolation works. Visibility is not enforcement. What is genuinely worth doing is a negative test: have the workload actively request more memory than its quota, and observe whether the request is rejected. Only an allocation failure (CUDA OOM) proves that isolation actually takes effect at runtime.
This is also why I place this layer in cluster acceptance criteria, rather than stopping at a one-off visibility check. Correct scheduling, correct device allocation, and correct visibility do not add up to correct enforcement—and enforcement is precisely the layer HAMi Core completes.
Software Isolation as a GPU Governance Layer
What carries more structural significance is NVIDIA’s choice: KAI Scheduler (which originated as Run:ai and now belongs to NVIDIA) did not build isolation in-house, but adopted HAMi-core directly. The significance is not that an open source project won an endorsement—it is that a path has been confirmed: GPU isolation and governance can exist as an independent software layer, without depending entirely on hardware virtualization such as MIG or SR-IOV.
For heterogeneous compute governance, this point is especially critical. Hardware virtualization capabilities vary wildly across vendors—some support fine-grained partitioning, others barely at all; a unified software isolation layer is the prerequisite for cross-vendor governance to exist at all. Over a longer horizon, the resource plane (Device Plugin, DRA) is gradually separating from the injection plane (CDI), and devices are evolving from a simple integer extended resource toward a model with attributes, declarations, and dynamic allocation.
The resource plane and injection plane are gradually separating, as devices evolve toward an attributable, declarative, dynamically allocatable model:
This direction is the continuation of the “GPU control plane” I proposed when discussing HAMi v2.9.
The Next Stop for Governance: Observability and Heterogeneous Unification
“Governable” does not stop at isolation. For a resource to be governed, it must also be observable, operable, and manageable uniformly across heterogeneous environments. This is precisely what HAMi 2.10, scheduled for release on August 21, focuses on completing:
- Ascend soft partitioning now ships utilization, memory, and Prometheus metrics, moving soft-partitioned resources from “allocatable” to “observable and operable”.
- A single cluster can now mix template-based hard partitioning with HAMi-core-based soft partitioning—another step toward unified heterogeneous governance.
- Heterogeneous device support extends to AMD MI300x and Biren, currently at the scheduling layer primarily; full virtualization capability still awaits verification.
This main thread itself also enters 2.10: through the standalone KAI Resource Isolator, the division of labor between scheduling and isolation is being formalized as a product. It should be noted that the relevant isolation capabilities are still maturing—actual enforcement of memory limits and directory permissions for non-root containers still have PRs pending before release—so the current stage is better described as “rapidly maturing” rather than “fully usable”.
Taken together, the emphasis of 2.10 is pushing GPU sharing from “usable” toward “observable, operable, and uniformly governable across heterogeneous clusters”—consistent with the governance closed loop discussed earlier.
Summary
Back to the starting point of this thread: this year, from open scheduling to the control plane, observability, efficiency measurement, and community consensus, every dimension of compute governance has been discussed—except enforcement, which remained hanging. HAMi-core entering the NVIDIA KAI Scheduler closes exactly that link: scheduling decisions and runtime isolation can, for the first time, be verifiably reconciled. Viewed as a layered stack, governance means clear ownership for every layer: GPU Operator owns the node software lifecycle, KAI owns scheduling and shared accounting, and HAMi Core owns runtime enforcement. And software isolation being adopted by a mainstream scheduler means GPU governance is shifting from reliance on hardware virtualization toward a software layer reusable across vendors. Isolation is only the starting point of governance; observability and heterogeneous unification come next, and HAMi 2.10 is advancing along exactly that direction. The road from scarce hardware to governable infrastructure capability is becoming concrete.
References
- HAMi-core Adopted by NVIDIA KAI Scheduler - project-hami.io
- KAI Scheduler and HAMi GPU Memory Hard Isolation - project-hami.io
- Deploying KAI Scheduler and HAMi Memory Isolation on GKE - project-hami.io
- KAI Scheduler GPU Sharing docs - GitHub
- HAMi and KAI Scheduler integration - project-hami.io
- HAMi - GitHub
- KAI Scheduler - GitHub
- NVIDIA GPU Operator - GitHub
- NVIDIA Container Toolkit - GitHub
- NVIDIA DCGM Exporter - GitHub
- Container Device Interface - GitHub
