端到端实战：从裸机节点到训练作业
草稿
每一条命令都达到生产可用。这一章是全书的“接线图”，把软件栈、设备插件、Operator、NUMA 与监控串成一条时间线。
环境
| 项目 | 值 |
|---|---|
| 节点 OS | Ubuntu 24.04 LTS |
| GPU | 1× NVIDIA H100 80GB SXM5（DGX H100 节点） |
| Kubernetes | v1.34（DRA GA） |
| GPU Operator | v26.3.0 |
| NVIDIA 驱动目标 | 570.x（由 GPU Operator 管理） |
| 容器运行时 | containerd v2.0 |
第 1 步：准备节点（K8s 之前）
# 在 GPU 节点上（纯净 OS，尚未装驱动）
sudo apt-get update
sudo apt-get install -y curl apt-transport-https ca-certificates gnupg lsb-release
# 安装 containerd
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor \
-o /etc/apt/keyrings/docker.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] \
https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
| sudo tee /etc/apt/sources.list.d/docker.list
sudo apt-get update && sudo apt-get install -y containerd.io
# 配置 containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
# 加入集群
sudo kubeadm join <control-plane-ip>:6443 --token <token> \
--discovery-token-ca-cert-hash sha256:<hash>
# 此刻：节点在集群里，但没有 GPU 支持
kubectl describe node gpu-node-01 | grep 'nvidia.com'
# GPU 对 Kubernetes 不可见第 2 步：部署 GPU Operator
# 在控制平面 / 管理机上
curl -fsSL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update
helm install --wait --generate-name \
-n gpu-operator --create-namespace \
nvidia/gpu-operator \
--version=v26.3.0 \
--set driver.enabled=true \
--set driver.version='570.86.10' \
--set toolkit.enabled=true \
--set devicePlugin.enabled=true \
--set dcgmExporter.enabled=true \
--set dcgmExporter.serviceMonitor.enabled=true \
--set migManager.enabled=true \
--set nfd.enabled=true \
--set gfd.enabled=true第 3 步：观察自举顺序
watch kubectl get pods -n gpu-operator
# 预期顺序（与 GPU Operator 章的八步对应）：
# 1. node-feature-discovery 启动 → 打 OS/内核标签
# 2. nvidia-driver-daemonset 启动 → 编译并加载 nvidia.ko（首次 2–5 分钟）
# 3. nvidia-container-toolkit 启动 → 配置 containerd
# 4. gpu-feature-discovery 启动 → 打 GPU 型号标签
# 5. nvidia-device-plugin 启动 → 注册 nvidia.com/gpu
# 6. nvidia-dcgm-exporter 启动 → 暴露指标
# 7. nvidia-mig-manager 启动 → 管理 MIG（空闲）
# 8. nvidia-operator-validator 启动 → CUDA 冒烟测试
kubectl describe node gpu-node-01 | grep -A3 'Allocatable'
# Allocatable:
# nvidia.com/gpu: 1 ← 成功！GPU 已注册第 4 步：运行 GPU 验证 Pod
kubectl run nvidia-smi-test \
--image=nvcr.io/nvidia/cuda:12.6.0-base-ubuntu22.04 \
--restart=Never \
--limits='nvidia.com/gpu=1' \
-- nvidia-smi
kubectl logs nvidia-smi-test
# +---------------------------------------------------------------------+
# | NVIDIA-SMI 570.86.10 Driver Version: 570.86.10 CUDA Version: 13.1|
# +-------+---------------------+------+------+------------------------+
# | 0 | NVIDIA H100 80GB HBM3 On | 00000000:1E:00.0 |
# +-------+---------------------+------+------+------------------------+
# | 30W / 700W | 1023MiB / 81920MiB | 0% Default |
# +---------------------------------------------------------------------+
kubectl delete pod nvidia-smi-test第 5 步：配置 NUMA 感知调度
# 在 GPU 节点上配置 kubelet
sudo tee -a /var/lib/kubelet/config.yaml << 'EOF'
cpuManagerPolicy: static
topologyManagerPolicy: single-numa-node
topologyManagerScope: pod
memoryManagerPolicy: Static
reservedMemory:
- numaNode: 0
limits:
memory: 4Gi
EOF
# 策略变更后必须清理管理器状态
sudo systemctl stop kubelet
sudo rm -f /var/lib/kubelet/cpu_manager_state
sudo rm -f /var/lib/kubelet/memory_manager_state
sudo systemctl start kubelet
sudo journalctl -u kubelet | grep -i topology
# 'topology manager policy: single-numa-node'第 6 步：运行分布式训练作业
---
apiVersion: batch/v1
kind: Job
metadata:
name: pytorch-training
namespace: ml-team
spec:
template:
spec:
restartPolicy: Never
nodeSelector:
nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
containers:
- name: trainer
image: nvcr.io/nvidia/pytorch:25.03-py3
command:
- python
- -c
- |
import torch
print(f'CUDA available: {torch.cuda.is_available()}')
print(f'GPU count: {torch.cuda.device_count()}')
print(f'GPU name: {torch.cuda.get_device_name(0)}')
x = torch.randn(4096, 4096, device='cuda')
y = torch.mm(x, x)
print(f'MatMul result shape: {y.shape} -- GPU compute verified!')
resources:
limits:
nvidia.com/gpu: '1'
memory: 64Gi
cpu: '16'
requests:
nvidia.com/gpu: '1'
memory: 64Gi
cpu: '16'
env:
- name: NCCL_DEBUG
value: INFO
- name: CUDA_VISIBLE_DEVICES
value: '0'第 7 步：用 DCGM 监控
kubectl get svc -n gpu-operator | grep dcgm
# nvidia-dcgm-exporter ClusterIP 10.96.xxx.xxx 9400/TCP
kubectl port-forward svc/nvidia-dcgm-exporter 9400:9400 -n gpu-operator &
curl -s localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL
# DCGM_FI_DEV_GPU_UTIL{gpu='0',UUID='GPU-xxxx',...} 87.0
# 训练期间的关键指标（口径详见“可观测”部分）：
# DCGM_FI_DEV_GPU_UTIL -> 活跃 matmul 期间应为 85-99%
# DCGM_FI_DEV_MEM_COPY_UTIL -> 数据加载期间升高
# DCGM_FI_DEV_FB_USED -> 显存占用（警惕逼近上限的 OOM）
# DCGM_FI_DEV_POWER_USAGE -> H100 SXM 满载应为 600-700W
# DCGM_FI_DEV_GPU_TEMP -> 满载通常 65-80°C（健康）
# DCGM_FI_DEV_XID_ERRORS -> 必须为 0，非零立即排查
# DCGM_FI_DEV_SM_CLOCK -> 应处于加速频率（H100 约 1800 MHz）
# GPU 健康诊断
kubectl exec -n gpu-operator -it nvidia-dcgm-exporter-xxxxx -- dcgmi diag -r 1
# -r 1 短诊断（30 秒）/ -r 2 中等（90 秒）/ -r 3 完整 burn-in（约 10 分钟）总结
七步走完，恰好对应全书的机制链条：容器运行时与驱动（软件栈）→ 设备注册（设备模型）→ Operator 自举（GPU Operator）→ 拓扑对齐（NUMA）→ 负载（工作负载）→ 验收（可观测）。把这套流程套进实施顺序的阶段 1–3，就是一台 GPU 节点从上架到生产的完整剧本。