Why GPU Scheduling Matters: From the Scheduling Landscape to Verifying HAMi mutex Semantics

GPU scheduling is not just “who gets which card”—it is answering three layered questions in sequence: which node, which card, and where the card sits inside the machine. This post walks from that three-layer framework to the HAMi v2.10 policy chain, then settles the most commonly misread policy semantic: mutex does not ask for exclusion from other mutex Pods; it asks for a zero-tenant card. The verdict comes from a decisive experiment anyone can reproduce on a laptop.

Why GPU Scheduling at All

An eight-GPU H100 server sells for more than a million yuan, and the GPUs account for the bulk of that cost. You buy it for compute, but the card has three properties a CPU does not: it is too expensive to leave idle; its granularity is too coarse—an entire card carries 80 GB of memory while an inference replica may need only two or three; and its performance is sensitive to neighbors—co-located tenants contend with each other for compute, memory bandwidth, and cache. Together, these three define the fundamental problem GPU scheduling must solve: place this expensive card, at the right granularity, in the right position, for the right workload, at the right time.

The Kubernetes default scheduler, meanwhile, was designed for stateless microservices. Its entire knowledge of a GPU is one line in the resource declaration:

resources:
  limits:
    nvidia.com/gpu: 1

Through its eyes, a GPU is an indivisible integer count. That creates three blind spots:

• Granularity blind spot: allocation is whole-card only. A small workload monopolizes a big card, and the idle memory and compute on it are pure waste;
• Co-location blind spot: it cannot tell which workloads should be packed onto the same card to amortize cost, and which must be kept apart to protect performance;
• Placement blind spot: it does not know which NUMA node a GPU hangs off, which PCIe switch it sits under, or which other GPUs and NICs are its close relatives.

The first two blind spots live inside the card, and GPU-sharing projects like HAMi exist precisely for them: slice the card by memory and compute, and let multiple Pods co-locate. The third blind spot is subtler and the most counterintuitive. Manan Paliwal’s article Why Kubernetes Is Slowing Down Your GPUs describes exactly this scenario: an H100 cluster that looks perfectly healthy, yet distributed training runs 30% to 40% slower than expected, and adding more cards barely helps. The cause is not CUDA, PyTorch, or NCCL—the scheduler placed the GPU and its RDMA NIC on different NUMA nodes, so every packet has to cross the inter-socket CPU link before reaching the network. Or consider a four-card machine where GPU 0 and GPU 1 hang under PCIe switch A and GPU 2 and GPU 3 under switch B: a workload requests two GPUs, and the scheduler is entirely capable of returning GPU 1 plus GPU 2—the count is right, the placement is wrong, and communication now crosses switches and the CPU interconnect. The most stinging line in that article sums it up: nothing crashes, no job fails, performance quietly evaporates.

Three Dimensions, One Policy Chain

Turn the blind spots into a question list, and a single GPU scheduling decision is really three layered questions answered in sequence:

The default scheduler answers only the first layer, crudely, with “are total resources sufficient?"—layers two and three are left entirely to luck. The Kubernetes ecosystem’s answers are layered: Kueue does hardware-aware whole-job placement at the queueing layer, kubelet’s Topology Manager aligns NUMA inside the node, and the NVIDIA Network Operator with Multus exposes RDMA networks directly to workloads. HAMi’s angle is different: it moves layers two and three into the scheduler itself, expressed as a language of Pod annotations.

Concretely, the v2.10 scheduling model looks like this:

• Card-level sharing is the home turf: HAMi-core soft-slices a card by memory and compute at the driver layer so multiple Pods co-locate on one card—the foundation of the sharing side of the “card” dimension;
• Two annotations govern two layers of preference: hami.io/node-scheduler-policy for the node level (a single value, binpack or spread), and hami.io/gpu-scheduler-policy for the card level (comma-separated policy chains as of v2.10);
• Policies play two roles: filters (mutex, NVIDIA’s topology-aware) remove unqualified cards from the candidate set before ranking; sort keys (binpack, spread, numa) rank the survivors. When only a filter is given, ranking falls back to spread.

Mapped against the table above: binpack and spread answer “how to pack”; numa and topology-aware answer “is the position right”; mutex answers “may it have neighbors.” A chain like mutex,binpack,numa translates to: give me a card nobody has used, pack as tightly as possible, and ideally keep it on the same NUMA node.

On this map, mutex is the only policy carrying exclusive semantics—and the newest addition in v2.10. The precise meaning of its “exclusivity” happens to be the murkiest part of both the documentation and community understanding, and that is precisely the question to settle next.

Background: Who Exactly Does mutex Exclude

Verdict first, answering the most direct question: does hami.io/gpu-scheduler-policy: "mutex" make Pods carrying the annotation mutually exclusive with one another? No. Its actual behavior is:

HAMi v2.10 brought the mutex policy and composable policy chains (such as mutex,binpack) to the hami.io/gpu-scheduler-policy annotation. Two readings of the mutex semantics have long coexisted in the community:

Most test reports (including one internal LWS test of ours) cannot distinguish between the two readings, because they set gpuSchedulerPolicy: mutex as a global default. When every Pod is mutex, the two semantics predict identical behavior: with any Pod already on a card, later mutex Pods get rejected. To separate A from B, you must design a mixed scenario: fill the card with ordinary Pods only, then submit a mutex Pod.

• Reading A predicts: rejected (the card already has a tenant)
• Reading B predicts: allowed (no mutex Pod is on the card)

The verdict has already been stated: HAMi implements Reading A. Four layers of evidence support it: the original request in issue #2009 says “no existing users”; the implementation in PR #2011 checks dev.Used > 0 (a count that covers every Pod on the card); observations from our real-GPU experiment on a four-card T4 node on GKE; and the local mock experiment in this post. This article documents the full reproduction of the fourth layer of evidence—it needs no GPU at all and runs in about twenty minutes on a laptop.

Why HAMi Needed mutex

To understand why this policy appeared, start from the fundamental tension of GPU sharing.

The value of GPU virtualization projects like HAMi lies in slicing: cut an 80 GB card into shares by memory and compute, co-locate multiple inference replicas on it, and utilization goes up while cost comes down. But slicing has an inherent price: co-located tenants share SMs, share memory bandwidth, and pollute each other’s L2 cache. Throughput-oriented batch jobs don’t care; two classes of workloads find it lethal: latency-sensitive online inference (uncontrollable tail latency), and benchmarking or training jobs that need a stable baseline (irreproducible results).

Before mutex, users who wanted a card to themselves had only two unsatisfactory options:

• Skip slicing and take the whole card: during troughs an entire card sits idle—in a shared cluster, the single largest source of waste;
• Use hardware partitioning such as MIG: fixed geometries, pre-configured on the node by an administrator, profiles that rarely align exactly with workload needs, and every change touches the node.

mutex fills the gap in between: a scheduler-layer, Pod-granularity, zero-hardware-dependency soft-exclusivity switch. It introduces no new isolation technology; it simply tells the scheduler “give me a card with zero tenants right now.” Combined with HAMi’s existing soft slicing, a cluster can for the first time serve both classes at once: shared slices amortize cost at peak, while mutex claims an idle card exclusively for overnight benchmarks or latency-sensitive services, released the moment the job finishes.

Its arrival in v2.10 is not isolated either. The same roadmap (#1889) advanced policy chains (#2010) in parallel, because real clusters stack their demands: pack tightly to free whole cards, keep NUMA affinity, and still hold a few exclusive cards for critical workloads. The chained expression mutex,binpack,numa is the complete form of those demands. Another typical case is replica-group scheduling with LeaderWorkerSet: GPU Pods within the same replica group should not share physical cards, and marking the group’s Pods as mutex satisfies that naturally.

There is one more design trade-off worth savoring: mutex exclusivity is one-way and binds only its own placement moment. Implementation-wise it merely reuses the scheduler’s existing Used counter (any Pod allocation on a card increments it), introducing no runtime locks or reservations. The choice keeps the implementation minimal and the semantics unambiguous, at the cost of not constraining later workloads: ordinary Pods scheduled afterward can still join the card. For end-to-end exclusivity, request a whole card’s resources or pin the card with use-gpuuuid. Where to draw the line of “guarantee” in a scheduling system is always a trade-off; HAMi draws it at placement time and leaves stronger constraints for users to express explicitly.

How mutex Works

The evaluation model of the policy chain was given in the previous section; here we zoom into the exact behavior of the mutex filter inside the scheduler. It stands guard at the entrance of the candidate card set with a single criterion: whether the card’s Used count is 0. That count covers every Pod on the card, mutex annotation or not.

The “one-way exclusivity” is clearest on a timeline. The same Used > 0 check executes only when the mutex Pod itself is being scheduled:

The three experiments below map to these two diagrams: Experiment 1 verifies that the filter condition is “any tenant” rather than “mutex tenants only”; Experiment 2 verifies the happy path of the first diagram; Experiment 3 verifies the one-way property of the second.

Lab Environment: kind + mock-device-plugin

The overall idea: bring up a single-node cluster with kind, install the HAMi scheduler (control plane only), then use HAMi’s official mock-device-plugin to register two fake Tesla T4s on the node. The HAMi scheduler runs its policy computation on them exactly as it would on real cards.

The experiment uses exactly the same build as HAMi’s official Lab 14 (the GKE experiment): the Helm chart comes from HAMi source commit 45b3d46769b44cfc1445728dfcb8e524939afba1 (master HEAD on 2026-08-17, i.e. the v2.10.0 release candidate), and the image is the per-commit tag HAMi CI published for that commit, projecthami/hami:45b3d46. Do not take the shortcut of latest: it is a moving tag, and by the time of writing it had already drifted to a newer master commit (the version embedded in the image changed from 45b3d46 to 949f78e), which would distort the reproduction.

Create the Cluster and Prepare the Images

kind nodes pulling images themselves is at the mercy of the network; pull on the host first, then load:

kind create cluster --name hami-mutex2 --image kindest/node:v1.36.1

# Pre-pull on the host (retry a few times on a flaky network)
docker pull projecthami/hami:45b3d46
docker pull projecthami/mock-device-plugin:latest
docker pull \
  registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.36.1
docker pull liangjw/kube-webhook-certgen:v1.1.1

kind load docker-image projecthami/hami:45b3d46 --name hami-mutex2
kind load docker-image projecthami/mock-device-plugin:latest --name hami-mutex2
kind load docker-image \
  registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.36.1 \
  --name hami-mutex2
kind load docker-image liangjw/kube-webhook-certgen:v1.1.1 --name hami-mutex2

The kube-scheduler image is the sidecar used by the scheduler embedded in the HAMi chart—its tag must match the cluster version; certgen is the image for the webhook certificate Job during installation.

Install the HAMi Scheduler and the Mock Plugin

# Fetch the same chart source as Lab 14
curl -fsSL https://codeload.github.com/Project-HAMi/HAMi/tar.gz/45b3d46769b44cfc1445728dfcb8e524939afba1 \
  -o hami-src.tar.gz
tar xzf hami-src.tar.gz

helm install hami HAMi-45b3d46769b44cfc1445728dfcb8e524939afba1/charts/hami \
  -n kube-system \
  --set global.imageTag=45b3d46 \
  --set devicePlugin.enabled=false \
  --set mockDevicePlugin.enabled=true \
  --set mockDevicePlugin.image.tag=latest \
  --set mockDevicePlugin.image.pullPolicy=IfNotPresent

Three parameters matter:

• devicePlugin.enabled=false: no real GPUs here, so HAMi’s own device plugin is disabled;
• mockDevicePlugin.enabled=true: the mock plugin takes over, registering virtual memory/compute extended resources on the node;
• mockDevicePlugin.image.tag=latest: latest is mandatory. Version 1.0.1 cannot parse the new vnpus config format in the current chart and crashes outright with cannot unmarshal !!map into []ascend.VNPUConfig—a pit I fell into personally.

Register Two Fake T4s

The mock plugin is designed as a three-piece set: a device-config block (already shipped in the chart), a count extended resource on the node (the health gate—only needs to be greater than 0), and the hami.io/node-nvidia-register annotation (describing the fake cards). The latter two must be provided by hand:

NODE=hami-mutex2-control-plane

# Health gate: nvidia.com/gpu = 2 cards x 10 slices
kubectl patch node $NODE --subresource=status --type=json -p '[
  {"op": "add", "path": "/status/capacity/nvidia.com~1gpu", "value": "20"}
]'

# Two fake T4s: GPU-MOCK-A and GPU-MOCK-B, 15360 MiB each
kubectl annotate node $NODE --overwrite \
  'hami.io/node-nvidia-register=[
    {"id":"GPU-MOCK-A","count":10,"devmem":15360,"devcore":100,
     "type":"NVIDIA-Tesla-T4","health":true,"numa":0,"mode":"hami-core"},
    {"id":"GPU-MOCK-B","index":1,"count":10,"devmem":15360,"devcore":100,
     "type":"NVIDIA-Tesla-T4","health":true,"numa":0,"mode":"hami-core"}
  ]'

Wait about 30 seconds, then confirm the mock plugin finished registering resources:

kubectl get node $NODE -o jsonpath='{.status.allocatable}' | python3 -c "
import json, sys
alloc = json.load(sys.stdin)
nvidia = {k: v for k, v in alloc.items() if 'nvidia' in k}
print(json.dumps(nvidia, indent=2))
"

{
  "nvidia.com/gpu": "20",
  "nvidia.com/gpucores": "200",
  "nvidia.com/gpumem": "30720",
  "nvidia.com/gpumem-percentage": "200"
}

Both cards are ready. All test Pods use the same template: request 1 vGPU and 1000 MiB of memory. The environment preloads every image, but note that imagePullPolicy: IfNotPresent is still worth writing explicitly, to avoid depending on the registry at runtime:

cat > plain-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: plain-N
spec:
  restartPolicy: Never
  containers:
    - name: app
      image: docker.io/projecthami/hami:45b3d46
      imagePullPolicy: IfNotPresent
      command: ["sh", "-c", "sleep 3600"]
      resources:
        limits:
          nvidia.com/gpu: 1
          nvidia.com/gpumem: 1000
EOF

The single tool for observing placement is this command—the HAMi scheduler writes the chosen card onto a Pod annotation:

kubectl get pods -o custom-columns=\
  'POD:.metadata.name,'\
  'CARD:.metadata.annotations.hami\.io/vgpu-devices-allocated'

Experiment 1: The Decisive Experiment

Step one: deploy two plain Pods with no policy annotations at all (sed renames them):

sed 's/plain-N/plain-1/' plain-pod.yaml | kubectl apply -f -
sed 's/plain-N/plain-2/' plain-pod.yaml | kubectl apply -f -
kubectl wait --for=condition=Ready pod/plain-1 pod/plain-2 --timeout=3m

POD       CARD
plain-1   GPU-MOCK-B,NVIDIA,1000,0:;
plain-2   GPU-MOCK-A,NVIDIA,1000,0:;

The default spread policy puts the two plain Pods on separate cards. At this moment the cluster contains not a single mutex Pod. Submit one:

cat > mutex-pod.yaml <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: mutex-1
  annotations:
    hami.io/gpu-scheduler-policy: "mutex"
spec:
  restartPolicy: Never
  containers:
    - name: app
      image: docker.io/projecthami/hami:45b3d46
      imagePullPolicy: IfNotPresent
      command: ["sh", "-c", "sleep 3600"]
      resources:
        limits:
          nvidia.com/gpu: 1
          nvidia.com/gpumem: 1000
EOF
kubectl apply -f mutex-pod.yaml
sleep 20
kubectl get pod mutex-1
kubectl describe pod mutex-1 | sed -n '/Events:/,$p' | tail -3

NAME      READY   STATUS    RESTARTS   AGE
mutex-1   0/1     Pending   0          20s

Warning  FailedScheduling  20s  hami-scheduler
  0/1 nodes are available: 1 2/2 ExclusiveDeviceAllocateConflict.
  no new claims to deallocate,
  preemption: 0/1 nodes are available: 1
  No preemption victims found for incoming pod.

2/2 ExclusiveDeviceAllocateConflict: both cards rejected. If Reading B (mutex-only exclusion) held, there would be no mutex Pod on either card right now, and this Pod should schedule immediately. It did not.

Step two: delete plain-1 to free GPU-MOCK-B, and watch where the mutex Pod goes:

kubectl delete pod plain-1 --force --grace-period=0
kubectl wait --for=condition=PodScheduled pod/mutex-1 --timeout=2m

POD       CARD
mutex-1   GPU-MOCK-B,NVIDIA,1000,0:;
plain-2   GPU-MOCK-A,NVIDIA,1000,0:;

The mutex Pod landed on the card that had been completely emptied, while plain-2 was still running on the other card. Zero-tenant semantics confirmed.

Experiment 2: mutex vs mutex (the Control Group)

Clear all Pods, then submit three mutex Pods in a row:

kubectl delete pod --all --force --grace-period=0
kubectl apply -f mutex-pod.yaml
sed 's/name: mutex-1/name: mutex-2/' mutex-pod.yaml | kubectl apply -f -
sed 's/name: mutex-1/name: mutex-3/' mutex-pod.yaml | kubectl apply -f -

POD       PHASE     CARD
mutex-1   Running   GPU-MOCK-B,NVIDIA,1000,0:;
mutex-2   Pending   GPU-MOCK-A,NVIDIA,1000,0:;
mutex-3   Pending   <none>

The first two mutex Pods take the two cards; the third is rejected with 2/2 ExclusiveDeviceAllocateConflict. Both readings predict the same outcome here, so this serves as the control group. As a side note, mutex-2 briefly emitted a node lock contention event before succeeding on retry—a known HAMi node-lock contention behavior that does not affect semantics.

Experiment 3: Exclusivity Is One-Way

One last question: once a mutex Pod has taken a card, is that card exclusive from then on? With mutex-1 running on GPU-MOCK-B, submit a plain sharing Pod annotated binpack:

POD        CARD
binpack-1  GPU-MOCK-B,NVIDIA,1000,0:;
mutex-1    GPU-MOCK-B,NVIDIA,1000,0:;
mutex-2    GPU-MOCK-A,NVIDIA,1000,0:;

The binpack Pod landed on the very same card the mutex Pod was using. Exclusivity applies only at the mutex Pod’s own placement moment and is one-way: it demands an empty target card at placement time, but does not reserve the card afterward.

The Verdict

All three experiments agree fully with the code, the original requirement, and the observations on real GPUs. HAMi’s mutex semantics fit in two sentences:

1. A mutex Pod can only land on a card that is currently zero-tenant, regardless of whether the Pods already on it carry the mutex annotation;
2. Exclusivity is one-way and placement-time-only: non-mutex Pods scheduled later can still join the card.

Practical advice for users: if your goal is “this workload owns the whole card until it exits,” mutex only gets you halfway. Either request the card’s full resources (saturate both memory and compute), or pin the card with nvidia.com/use-gpuuuid; both are also covered in the official HAMi blog.

Looking back at the motivation and the mechanism, the value of mutex is not any isolation technology per se, but that it adds the “I want to live alone” option to the soft-slicing world, hooked into the policy chain through the lightest possible implementation—a filter. Placed back on the three-dimension map from the beginning of this post, mutex occupies just one cell, “card-level exclusivity”; but precisely because policies compose, that cell can stack with packing preference and NUMA affinity in the same chain, each evolving on its own. The methodology for judging its behavior matters just as much: when documentation wording diverges, construct a minimal scenario where the two readings predict different outcomes, reproduce it locally with mock devices, and the verdict arrives within minutes. The same method can extend to verifying the ranking behavior of binpack,numa chains, the fallback rules of combined policies, and more—extensions welcome.

Cleanup

kubectl delete pod --all --force --grace-period=0
helm uninstall hami -n kube-system
kind delete cluster --name hami-mutex2

Summary

This post started from the question “why do GPU scheduling at all,” broke out the three scheduling dimensions—node, card, topology—and showed how HAMi v2.10 covers them with a policy chain of filters plus sort keys. It then zoomed into mutex, the only policy with exclusive semantics: using kind plus mock-device-plugin, we built a GPU-less local environment and settled its semantic dispute through three experiments—it demands a zero-tenant target card (not merely exclusion from other mutex Pods), and its exclusivity is one-way, binding only at placement time. The experiments use the same v2.10.0 release-candidate build as official Lab 14; every command and output comes from actual runs and is fully reproducible. The post also records three real-world pitfalls: mock plugin 1.0.1 cannot parse the new config format and requires the latest tag, the latest image should be explicitly IfNotPresent, and preloading images onto the kind node is essential.

References

• Composable GPU Scheduling Policies in HAMi v2.10 - project-hami.io: principles of the policy chain and the mutex semantics
• Lab 14: Composable GPU Scheduling Policies on GKE - project-hami.io: the full experiment on real T4s
• Why Kubernetes Is Slowing Down Your GPUs (And How Topology-Aware Scheduling Fixes It) - Medium: motivation for topology-aware scheduling—the default scheduler counts cards but ignores position
• Kueue - kueue.sigs.k8s.io: Kubernetes-native job queueing and hardware-aware scheduling
• Kubernetes Topology Manager - kubernetes.io: NUMA resource alignment at the kubelet layer
• feat: new mutex GPU scheduling policy (issue #2009) - GitHub: the original requirement, explicitly saying “no existing users”
• feat: new mutex GPU scheduling policy (PR #2011) - GitHub: the mutex implementation
• feat: support comma-separated gpu-scheduler-policy combinations (PR #2621) - GitHub: the composable policy-chain implementation
• mock-device-plugin - GitHub: HAMi’s official mock device plugin