git clone https://github.com/Project-HAMi/hami-workshop
cd hami-workshop/examples/containerdays-singapore
git clone https://github.com/fishman/hami-mock-monitor
docker pull kindest/node:v1.37.0 busybox:1.36 ghcr.io/llm-d/llm-d-inference-sim:v0.12.0

• Needs Docker (or OrbStack / Colima), 4 CPU and 8 GB free, plus kind, kubectl, helm, jq, git
• Go 1.27+ only to build HAMi or contribute code. A GitHub account for Lab 4
• Conference Wi-Fi is slow. Pull images at home

Kubernetes allocates GPUs atomically: one whole device, one task.

• A 1 GB task blocks an 80 GB device
• Most GPUs sit idle most of the time
• No advanced scheduling yet: no binpack, no spread, no topology

• Mutating webhook: routes GPU Pods to hami-scheduler
• Scheduler extender: Filter, score, Bind against per-card state
• Device plugin: registers cards, implements Allocate
• HAMi-core: libvgpu.so in the container, enforces limits
• vGPUmonitor: per-container usage metrics

Code entry points: cmd/scheduler, cmd/device-plugin, cmd/vGPUmonitor, pkg/device/<vendor>

./setup.sh

The script:

• Creates a kind cluster hami-lab and labels the node gpu=on
• Installs HAMi 2.10.0 with the real device plugin off and the mock plugin on
• Fakes two A100-40GB cards. Two pieces:the count resource nvidia.com/gpu=20: the mock plugin's health gate, 2 cards x 10 splits
• the node annotation hami.io/node-nvidia-register from fake-gpus.json, what the real device plugin writes after it reads NVML
• Builds and installs hami-mock-monitor with completeBinds=true

Expected at the end:

{ "nvidia.com/gpu": "20", "nvidia.com/gpucores": "200",
  "nvidia.com/gpumem": "81920", "nvidia.com/gpumem-percentage": "200" }

kubectl -n kube-system get pods | grep hami
kubectl -n kube-system get cm hami-scheduler-device -o yaml | less
kubectl get node hami-lab-control-plane \
  -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq

[{"id":"GPU-mock-0","index":0,"count":10,"devmem":40960,"devcore":100,
  "type":"NVIDIA-A100-SXM4-40GB","health":true,"numa":0,"mode":"hami-core"}, ...]

• count: how many Pods may share the card. devmem in MB, devcore in percent
• The card list lives on the node, written by the device plugin. The scheduler reads it at Filter

kubectl apply -f share.yaml -f too-big.yaml
kubectl get pods -o wide

share-1..3 each ask for 15000 MB and 30% of a card, and run. too-big asks for 45000 MB, more than any 40960 MB card, and stays Pending:

kubectl get events --field-selector involvedObject.name=too-big | tail -2
# 0/1 nodes are available: 1 2/2 CardInsufficientMemory.

45000 MB fits the node total, but no single 40960 MB card.

for p in share-1 share-2 share-3; do
  echo -n "$p "; kubectl get pod $p -o jsonpath='{.metadata.annotations.hami\.io/vgpu-devices-allocated}{"\n"}'
done
# share-1 GPU-mock-1,NVIDIA,15000,30:;
# share-2 GPU-mock-0,NVIDIA,15000,30:;
# share-3 GPU-mock-1,NVIDIA,15000,30:;

The format is uuid,type,memory_mb,cores: per device, ; per container. The device plugin reads it in Allocate.

• The admission webhook rewrites schedulerName to hami-scheduler
• The scheduler extender at Filter reads the node annotation hami.io/node-nvidia-register (the card list from Lab 1), checks memory and core headroom per card, and picks a physical card UUID
• At Bind it writes hami.io/vgpu-devices-allocated onto the Pod, sets hami.io/bind-phase=allocating and takes the node lock hami.io/mutex.lock
• The device plugin's Allocate mounts /dev/nvidia* and the vGPU files, reading the target from the Pod annotation, then marks success and releases the lock

On a real node, step 4 also injects libvgpu.so via ld.so.preload, and HAMi-core enforces the memory and core limits from that moment on.

• Both axes score cards by utilization ratios: binpack prefers the fuller card (consolidation), spread prefers the emptier one (dispersion)
• This lab has one node, so the node axis has a single candidate
• The card axis is fully observable on the two fake A100s

kubectl delete -f share.yaml -f too-big.yaml
kubectl apply -f binpack.yaml
for p in pack-1 pack-2; do
  echo -n "$p "; kubectl get pod $p -o jsonpath='{.metadata.annotations.hami\.io/vgpu-devices-allocated}{"\n"}'
done

Both Pods report the same GPU-mock-X: two workloads sharing one card. Frees whole cards for big jobs.

kubectl delete -f binpack.yaml
kubectl apply -f spread.yaml
for p in spread-1 spread-2; do
  echo -n "$p "; kubectl get pod $p -o jsonpath='{.metadata.annotations.hami\.io/vgpu-devices-allocated}{"\n"}'
done

One Pod per card. Same cluster, same requests, different policy, different placement.

metadata:
  annotations:
    hami.io/gpu-scheduler-policy: spread   # or binpack, per Pod

• The allocation annotation names an exact card UUID, and card choice is a topology decision: on a multi-GPU node it decides NVLink and PCIe locality
• fake-gpus.json carries a numa field per card, the same hint the real device plugin records from the driver
• Card-level spread keeps Pods on separate cards, binpack packs them onto one
• Explicit topology-aware placement across cards and nodes is what the schedulers in the next lab add

• A capacity model: the extended resources plus the scheduler extender you just used
• HAMi-core: the isolation library
• Other schedulers can replace the placement brain and keep the isolation layer

NVIDIA's scheduler for AI workloads (CNCF Sandbox): gang scheduling, hierarchical fair queues, fractional GPU, topology-aware placement.

helm install kai-scheduler oci://ghcr.io/nvidia/kai-scheduler --version v0.17.0 \
  --set global.gpuSharing=true --set binder.plugins.hamicore.enabled=true \
  -n kai-scheduler --create-namespace

• Since v0.16.4 the kai-resource-isolator DaemonSet plus a mutating webhook deliver the same libvgpu.so into Pods
• CUDA_DEVICE_MEMORY_LIMIT injected from gpu-fraction or gpu-memory annotations
• KAI makes the placement decisions, HAMi-core makes the memory cap hard

kind: Job                    # batch.volcano.sh/v1alpha1
spec:
  schedulerName: volcano
  minAvailable: 2            # both workers or none
  queue: vgpu-queue
  tasks:
  - replicas: 2
    template:
      spec:
        containers:
        - resources:
            limits:
              volcano.sh/vgpu-number: 1
              volcano.sh/vgpu-memory: 2000
              volcano.sh/vgpu-cores: 30

• Takes the vGPU path instead of the standard HAMi chart: Volcano deviceshare plugin plus the volcano-vgpu-device-plugin
• Queue capability caps vGPU memory per team, even when the node has room
• Never run the standard HAMi device plugin and a Volcano vGPU plugin on the same node

• HAMi-DRA is a mutating webhook that rewrites nvidia.com/gpu requests into DRA ResourceClaims
• The same Pod spec keeps working on Kubernetes 1.34+ with the DRA Consumable Capacity feature gate and CDI enabled in the runtime
• A ResourceClaimTemplate mode exists for Kueue admission
• Your Pod YAML does not change, the machinery underneath does

github.com/Project-HAMi/HAMi-DRA

HAMi exposes two metric sets. Keep them apart: one is a reservation, the other is usage.

./metrics.sh scheduler   # allocation, port 9395
./metrics.sh monitor     # usage, port 9394

kubectl apply -f sim.yaml      # llm-d-inference-sim, requests oscillate 0 to 5
watch -n 5 "./metrics.sh monitor | grep 'pod=\"sim\"'"

hami_container_device_utilization_ratio{pod="sim"} 26.4
hami_node_gpu_memory_allocated_ratio{device_uuid="GPU-mock-1"} 0.73
hami_host_gpu_utilization_ratio{device_uuid="GPU-mock-1"}      20.0

• sim is a vLLM-compatible inference simulator: utilization moves between 0 and 33 (load x allocated cores)
• Memory stays at 0.9 x the limit, because vLLM preallocates its KV cache

Question. Card GPU-mock-1 shows 73% memory allocated but 20% used. Which number goes on a capacity dashboard, and which on a cost report?

metrics.sh is fine for a terminal, but the HAMi WebUI turns the same numbers into dashboards. It needs a Prometheus in the cluster:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add hami-webui https://project-hami.github.io/HAMi-WebUI
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --version 75.15.1 -n monitoring --create-namespace --set grafana.enabled=false --wait
kubectl apply -f monitoring.yaml   # scrape :9395 and :9394
kubectl apply -f webui-rules.yaml  # translate metric names for the WebUI
helm upgrade --install my-hami-webui hami-webui/hami-webui \
  --set externalPrometheus.enabled=true --set dcgm-exporter.enabled=false \
  -n kube-system --wait
kubectl -n kube-system port-forward svc/my-hami-webui 3000:3000

Open http://localhost:3000.

• The WebUI queries vGPUmonitor's metric names (hami_memory_size, hami_core_util, hami_container_vgpu_allocated, ...)
• This lab's scheduler and mock monitor expose the newer byte/ratio names (hami_gpu_memory_limit_bytes, hami_host_gpu_utilization_ratio, ...)
• The PrometheusRule records one set as the other. Without it every allocation and usage panel stays empty

kind delete cluster --name hami-lab

git clone https://github.com/Project-HAMi/hami-workshop
cd hami-workshop/examples/containerdays-singapore
git clone https://github.com/fishman/hami-mock-monitor
docker pull kindest/node:v1.37.0 busybox:1.36 ghcr.io/llm-d/llm-d-inference-sim:v0.12.0

• Needs Docker (or OrbStack / Colima), 4 CPU and 8 GB free, plus kind, kubectl, helm, jq, git
• Go 1.27+ only to build HAMi or contribute code. A GitHub account for Lab 4
• Conference Wi-Fi is slow. Pull images at home

Kubernetes allocates GPUs atomically: one whole device, one task.

• A 1 GB task blocks an 80 GB device
• Most GPUs sit idle most of the time
• No advanced scheduling yet: no binpack, no spread, no topology

• Mutating webhook: routes GPU Pods to hami-scheduler
• Scheduler extender: Filter, score, Bind against per-card state
• Device plugin: registers cards, implements Allocate
• HAMi-core: libvgpu.so in the container, enforces limits
• vGPUmonitor: per-container usage metrics

Code entry points: cmd/scheduler, cmd/device-plugin, cmd/vGPUmonitor, pkg/device/<vendor>

./setup.sh

The script:

• Creates a kind cluster hami-lab and labels the node gpu=on
• Installs HAMi 2.10.0 with the real device plugin off and the mock plugin on
• Fakes two A100-40GB cards. Two pieces:the count resource nvidia.com/gpu=20: the mock plugin's health gate, 2 cards x 10 splits
• the node annotation hami.io/node-nvidia-register from fake-gpus.json, what the real device plugin writes after it reads NVML
• Builds and installs hami-mock-monitor with completeBinds=true

Expected at the end:

{ "nvidia.com/gpu": "20", "nvidia.com/gpucores": "200",
  "nvidia.com/gpumem": "81920", "nvidia.com/gpumem-percentage": "200" }

kubectl -n kube-system get pods | grep hami
kubectl -n kube-system get cm hami-scheduler-device -o yaml | less
kubectl get node hami-lab-control-plane \
  -o jsonpath='{.metadata.annotations.hami\.io/node-nvidia-register}' | jq

[{"id":"GPU-mock-0","index":0,"count":10,"devmem":40960,"devcore":100,
  "type":"NVIDIA-A100-SXM4-40GB","health":true,"numa":0,"mode":"hami-core"}, ...]

• count: how many Pods may share the card. devmem in MB, devcore in percent
• The card list lives on the node, written by the device plugin. The scheduler reads it at Filter

kubectl apply -f share.yaml -f too-big.yaml
kubectl get pods -o wide

share-1..3 each ask for 15000 MB and 30% of a card, and run. too-big asks for 45000 MB, more than any 40960 MB card, and stays Pending:

kubectl get events --field-selector involvedObject.name=too-big | tail -2
# 0/1 nodes are available: 1 2/2 CardInsufficientMemory.

45000 MB fits the node total, but no single 40960 MB card.

for p in share-1 share-2 share-3; do
  echo -n "$p "; kubectl get pod $p -o jsonpath='{.metadata.annotations.hami\.io/vgpu-devices-allocated}{"\n"}'
done
# share-1 GPU-mock-1,NVIDIA,15000,30:;
# share-2 GPU-mock-0,NVIDIA,15000,30:;
# share-3 GPU-mock-1,NVIDIA,15000,30:;

The format is uuid,type,memory_mb,cores: per device, ; per container. The device plugin reads it in Allocate.

• The admission webhook rewrites schedulerName to hami-scheduler
• The scheduler extender at Filter reads the node annotation hami.io/node-nvidia-register (the card list from Lab 1), checks memory and core headroom per card, and picks a physical card UUID
• At Bind it writes hami.io/vgpu-devices-allocated onto the Pod, sets hami.io/bind-phase=allocating and takes the node lock hami.io/mutex.lock
• The device plugin's Allocate mounts /dev/nvidia* and the vGPU files, reading the target from the Pod annotation, then marks success and releases the lock

On a real node, step 4 also injects libvgpu.so via ld.so.preload, and HAMi-core enforces the memory and core limits from that moment on.

• Both axes score cards by utilization ratios: binpack prefers the fuller card (consolidation), spread prefers the emptier one (dispersion)
• This lab has one node, so the node axis has a single candidate
• The card axis is fully observable on the two fake A100s

kubectl delete -f share.yaml -f too-big.yaml
kubectl apply -f binpack.yaml
for p in pack-1 pack-2; do
  echo -n "$p "; kubectl get pod $p -o jsonpath='{.metadata.annotations.hami\.io/vgpu-devices-allocated}{"\n"}'
done

Both Pods report the same GPU-mock-X: two workloads sharing one card. Frees whole cards for big jobs.

kubectl delete -f binpack.yaml
kubectl apply -f spread.yaml
for p in spread-1 spread-2; do
  echo -n "$p "; kubectl get pod $p -o jsonpath='{.metadata.annotations.hami\.io/vgpu-devices-allocated}{"\n"}'
done

One Pod per card. Same cluster, same requests, different policy, different placement.

metadata:
  annotations:
    hami.io/gpu-scheduler-policy: spread   # or binpack, per Pod

• The allocation annotation names an exact card UUID, and card choice is a topology decision: on a multi-GPU node it decides NVLink and PCIe locality
• fake-gpus.json carries a numa field per card, the same hint the real device plugin records from the driver
• Card-level spread keeps Pods on separate cards, binpack packs them onto one
• Explicit topology-aware placement across cards and nodes is what the schedulers in the next lab add

• A capacity model: the extended resources plus the scheduler extender you just used
• HAMi-core: the isolation library
• Other schedulers can replace the placement brain and keep the isolation layer

NVIDIA's scheduler for AI workloads (CNCF Sandbox): gang scheduling, hierarchical fair queues, fractional GPU, topology-aware placement.

helm install kai-scheduler oci://ghcr.io/nvidia/kai-scheduler --version v0.17.0 \
  --set global.gpuSharing=true --set binder.plugins.hamicore.enabled=true \
  -n kai-scheduler --create-namespace

• Since v0.16.4 the kai-resource-isolator DaemonSet plus a mutating webhook deliver the same libvgpu.so into Pods
• CUDA_DEVICE_MEMORY_LIMIT injected from gpu-fraction or gpu-memory annotations
• KAI makes the placement decisions, HAMi-core makes the memory cap hard

kind: Job                    # batch.volcano.sh/v1alpha1
spec:
  schedulerName: volcano
  minAvailable: 2            # both workers or none
  queue: vgpu-queue
  tasks:
  - replicas: 2
    template:
      spec:
        containers:
        - resources:
            limits:
              volcano.sh/vgpu-number: 1
              volcano.sh/vgpu-memory: 2000
              volcano.sh/vgpu-cores: 30

• Takes the vGPU path instead of the standard HAMi chart: Volcano deviceshare plugin plus the volcano-vgpu-device-plugin
• Queue capability caps vGPU memory per team, even when the node has room
• Never run the standard HAMi device plugin and a Volcano vGPU plugin on the same node

• HAMi-DRA is a mutating webhook that rewrites nvidia.com/gpu requests into DRA ResourceClaims
• The same Pod spec keeps working on Kubernetes 1.34+ with the DRA Consumable Capacity feature gate and CDI enabled in the runtime
• A ResourceClaimTemplate mode exists for Kueue admission
• Your Pod YAML does not change, the machinery underneath does

github.com/Project-HAMi/HAMi-DRA

HAMi exposes two metric sets. Keep them apart: one is a reservation, the other is usage.

./metrics.sh scheduler   # allocation, port 9395
./metrics.sh monitor     # usage, port 9394

kubectl apply -f sim.yaml      # llm-d-inference-sim, requests oscillate 0 to 5
watch -n 5 "./metrics.sh monitor | grep 'pod=\"sim\"'"

hami_container_device_utilization_ratio{pod="sim"} 26.4
hami_node_gpu_memory_allocated_ratio{device_uuid="GPU-mock-1"} 0.73
hami_host_gpu_utilization_ratio{device_uuid="GPU-mock-1"}      20.0

• sim is a vLLM-compatible inference simulator: utilization moves between 0 and 33 (load x allocated cores)
• Memory stays at 0.9 x the limit, because vLLM preallocates its KV cache

Question. Card GPU-mock-1 shows 73% memory allocated but 20% used. Which number goes on a capacity dashboard, and which on a cost report?

metrics.sh is fine for a terminal, but the HAMi WebUI turns the same numbers into dashboards. It needs a Prometheus in the cluster:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add hami-webui https://project-hami.github.io/HAMi-WebUI
helm upgrade --install prometheus prometheus-community/kube-prometheus-stack \
  --version 75.15.1 -n monitoring --create-namespace --set grafana.enabled=false --wait
kubectl apply -f monitoring.yaml   # scrape :9395 and :9394
kubectl apply -f webui-rules.yaml  # translate metric names for the WebUI
helm upgrade --install my-hami-webui hami-webui/hami-webui \
  --set externalPrometheus.enabled=true --set dcgm-exporter.enabled=false \
  -n kube-system --wait
kubectl -n kube-system port-forward svc/my-hami-webui 3000:3000

Open http://localhost:3000.

• The WebUI queries vGPUmonitor's metric names (hami_memory_size, hami_core_util, hami_container_vgpu_allocated, ...)
• This lab's scheduler and mock monitor expose the newer byte/ratio names (hami_gpu_memory_limit_bytes, hami_host_gpu_utilization_ratio, ...)
• The PrometheusRule records one set as the other. Without it every allocation and usage panel stays empty

kind delete cluster --name hami-lab