Most homelab k3s guides stop at "get your cluster running." Then you throw an LLM inference server or a Stable Diffusion pod at it, and the scheduler drops it on your Raspberry Pi. The GPU node sits idle. The CPU node chokes. Fun.
GPU-aware scheduling is not optional once you have mixed hardware. It’s the difference between a cluster that actually works and one that wastes your most expensive resources. This guide covers the full stack: labeling nodes so the scheduler knows what’s where, taints to keep GPU nodes reserved for GPU workloads, NVIDIA and AMD device plugins that expose hardware as proper Kubernetes resources, and pod specs that request them correctly.
Official k3s repo: https://github.com/k3s-io/k3s
NVIDIA device plugin: https://github.com/NVIDIA/k8s-device-plugin
The Problem With Naive Scheduling
Kubernetes has no idea what hardware your nodes have unless you tell it. Out of the box, the scheduler uses CPU and memory requests/limits. GPUs don’t exist to it. You can add nodeSelector with a node name as a workaround, but that’s hardcoding — it breaks the moment you rename a node or add a second GPU machine.
The right approach uses three layers:
- Labels — declare what a node has (e.g.,
nvidia.com/gpu.present=true) - Taints — repel workloads that don’t explicitly want GPU nodes
- Device plugins — expose
nvidia.com/gpuoramd.com/gpuas allocatable resources so pods can request them just like CPU/RAM
When all three are in place, the scheduler places GPU workloads automatically on nodes with capacity, and non-GPU workloads never land on your expensive hardware unless you explicitly allow it.
Step 1: Prep the Node — Drivers First
Before k3s cares about your GPU, the OS needs working drivers. The device plugin talks to the driver via the container runtime, not directly to hardware.
For NVIDIA, install the proprietary driver and the NVIDIA Container Toolkit:
# Verify driver is loaded
nvidia-smi
# Install NVIDIA Container Toolkit (Debian/Ubuntu)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt update && sudo apt install -y nvidia-container-toolkit
Now tell containerd (k3s’s default runtime) to use the NVIDIA runtime. k3s bundles containerd at /var/lib/rancher/k3s/agent/etc/containerd/config.toml — but don’t edit that file directly, it gets regenerated. Create the override instead:
# Create the config template that k3s won't overwrite
sudo mkdir -p /var/lib/rancher/k3s/agent/etc/containerd/
sudo tee /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl > /dev/null <<'EOF'
version = 2
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "nvidia"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
BinaryName = "/usr/bin/nvidia-container-runtime"
EOF
sudo systemctl restart k3s-agent
Gotcha: If you use k3s server on the GPU node (single-node or combined control-plane), restart k3s not k3s-agent. Confirm containerd picked up the change: sudo k3s crictl info | grep -i nvidia.
Step 2: Label Your GPU Nodes
Labels are just key-value metadata. Pick a consistent scheme and stick to it. I use the nvidia.com/ prefix because the NVIDIA device plugin later enriches these automatically — starting with it avoids a rename later.
# Apply labels manually (do this once per GPU node)
kubectl label node <node-name> \
nvidia.com/gpu.present=true \
nvidia.com/gpu.product=RTX-3090 \
hardware/gpu-vendor=nvidia \
hardware/gpu-memory=24Gi
Check that labels landed:
kubectl get node <node-name> --show-labels
# or for a cleaner view:
kubectl describe node <node-name> | grep -A 30 Labels
Once the NVIDIA device plugin is running (Step 4), it automatically adds richer labels like nvidia.com/gpu.count, nvidia.com/cuda.runtime.major, nvidia.com/memory.size, etc. via the GPU Feature Discovery component. You get those for free — the manual labels above are just your scaffolding before the plugin is up.
Step 3: Taint GPU Nodes
A taint tells the scheduler "don’t put workloads here unless they explicitly tolerate this." Without taints, a random Redis pod or a monitoring exporter can land on your GPU node and eat memory that the CUDA runtime needs.
# Taint the node — NoSchedule means new pods won't land here unless they tolerate it
kubectl taint node <node-name> nvidia.com/gpu=present:NoSchedule
The taint key/value is arbitrary — what matters is consistency. I use nvidia.com/gpu=present to mirror the label scheme.
Gotcha: Existing pods already running on the node are not evicted by NoSchedule. Use NoExecute if you want to clear it:
# Evicts existing pods that don't tolerate the taint
kubectl taint node <node-name> nvidia.com/gpu=present:NoExecute
Only do NoExecute if you’re okay with the disruption. For initial cluster setup, NoSchedule on a fresh node is fine.
Step 4: Deploy the NVIDIA Device Plugin
The device plugin is a DaemonSet that runs on every GPU node, discovers GPUs, and registers them as nvidia.com/gpu resources with the kubelet. Without it, you can use labels and taints but can’t actually limit which container gets GPU access — any container on the node could try to use CUDA.
The official Helm chart is the cleanest way:
helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
helm repo update
# Deploy — tolerations are critical so the DaemonSet lands on tainted nodes
helm upgrade --install nvdp nvdp/nvidia-device-plugin \
--namespace nvidia-device-plugin \
--create-namespace \
--set tolerations[0].key=nvidia.com/gpu \
--set tolerations[0].operator=Exists \
--set tolerations[0].effect=NoSchedule
Or if you prefer plain manifests (more transparent for homelabs):
# nvidia-device-plugin-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: nvidia-device-plugin-daemonset
namespace: kube-system
spec:
selector:
matchLabels:
name: nvidia-device-plugin-ds
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
name: nvidia-device-plugin-ds
spec:
# Must tolerate the taint we set on GPU nodes
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
# Only run on nodes that actually have an NVIDIA GPU
nodeSelector:
nvidia.com/gpu.present: "true"
priorityClassName: system-node-critical
containers:
- image: nvcr.io/nvidia/k8s-device-plugin:v0.17.0
name: nvidia-device-plugin-ctr
env:
- name: FAIL_ON_INIT_ERROR
value: "false"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
volumeMounts:
- name: device-plugin
mountPath: /var/lib/kubelet/device-plugins
volumes:
- name: device-plugin
hostPath:
path: /var/lib/kubelet/device-plugins
kubectl apply -f nvidia-device-plugin-ds.yaml
k3s-specific gotcha: k3s uses /var/lib/rancher/k3s/agent/kubelet/device-plugins/ as the kubelet socket path, not the standard /var/lib/kubelet/device-plugins/. The NVIDIA plugin v0.14+ detects this automatically, but older pinned versions will silently fail. If kubectl describe node shows nvidia.com/gpu: 0 after the plugin runs, check the pod logs:
kubectl logs -n kube-system -l name=nvidia-device-plugin-ds
Verify the GPU is actually exposed after the DaemonSet is healthy:
kubectl describe node <node-name> | grep -A 10 "Allocatable"
# You should see: nvidia.com/gpu: 1 (or however many you have)
Step 5: AMD GPU — ROCm Device Plugin
If you’re running AMD (RX 6000/7000 series, Instinct cards), the setup is conceptually identical but uses a different plugin.
kubectl apply -f https://raw.githubusercontent.com/ROCm/k8s-device-plugin/master/k8s-ds-amdgpu-dp.yaml
Label and taint AMD nodes with your chosen scheme:
kubectl label node <node-name> \
amd.com/gpu.present=true \
hardware/gpu-vendor=amd
kubectl taint node <node-name> amd.com/gpu=present:NoSchedule
The AMD plugin exposes amd.com/gpu as the resource. Pod specs request it the same way as NVIDIA (just swap the resource name).
Gotcha with AMD: ROCm requires the /dev/kfd and /dev/dri devices accessible in the container. The device plugin handles this, but SELinux or AppArmor profiles on some distros will block it. If your ROCm workloads fail with permission errors, check dmesg and your security policy before assuming a k8s config problem.
Step 6: Working Pod Specs
This is where it all comes together. A pod that correctly requests a GPU, tolerates the taint, and targets the right node:
# gpu-inference-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: llm-inference
namespace: default
spec:
# Must tolerate the taint on the GPU node
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
# Optional: force to GPU nodes even if other nodes have capacity
# (redundant when taints are set, but explicit is better than implicit)
nodeSelector:
nvidia.com/gpu.present: "true"
containers:
- name: inference
image: ollama/ollama:latest
ports:
- containerPort: 11434
resources:
limits:
# Request exactly 1 GPU — this is a hard limit, not a hint
nvidia.com/gpu: "1"
memory: "16Gi"
cpu: "4"
requests:
nvidia.com/gpu: "1"
memory: "8Gi"
cpu: "2"
env:
- name: OLLAMA_HOST
value: "0.0.0.0"
volumeMounts:
- name: ollama-models
mountPath: /root/.ollama
volumes:
- name: ollama-models
persistentVolumeClaim:
claimName: ollama-models-pvc
restartPolicy: Always
For a Deployment with GPU (more realistic for long-running inference services):
# gpu-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: stable-diffusion
namespace: ai-workloads
spec:
replicas: 1 # Can't exceed GPU count — don't set replicas > available GPUs
selector:
matchLabels:
app: stable-diffusion
template:
metadata:
labels:
app: stable-diffusion
spec:
tolerations:
- key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
# Use affinity instead of nodeSelector when you need more flexibility
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: nvidia.com/gpu.present
operator: In
values:
- "true"
# Only schedule on nodes with enough GPU memory
- key: hardware/gpu-memory
operator: In
values:
- "24Gi"
- "16Gi"
containers:
- name: sd-webui
image: ghcr.io/automatic1111/stable-diffusion-webui:latest
resources:
limits:
nvidia.com/gpu: "1"
memory: "20Gi"
requests:
nvidia.com/gpu: "1"
memory: "12Gi"
Critical gotcha with replicas: nvidia.com/gpu: "1" is a resource request and limit — Kubernetes will not schedule two replicas that each request 1 GPU onto a node that only has 1 GPU. But it also won’t reschedule them to a node that has capacity if no such node exists. If you scale beyond your GPU count, pods will stay in Pending. This is expected behavior, not a bug.
GPU Time-Slicing (Sharing One GPU Across Pods)
If you have one GPU and multiple workloads, the device plugin supports time-slicing — exposing one physical GPU as N logical resources. The scheduler can then fit N pods onto one node.
Create a ConfigMap before deploying the plugin:
# gpu-timeslicing-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: nvidia-device-plugin-config
namespace: nvidia-device-plugin
data:
config.yaml: |
version: v1
sharing:
timeSlicing:
resources:
- name: nvidia.com/gpu
replicas: 4 # Split one GPU into 4 logical units
kubectl apply -f gpu-timeslicing-config.yaml
# Re-deploy the plugin with the config reference
helm upgrade nvdp nvdp/nvidia-device-plugin \
--namespace nvidia-device-plugin \
--set config.name=nvidia-device-plugin-config
After this, kubectl describe node shows nvidia.com/gpu: 4. Each pod requests 1 and gets 25% of the GPU’s time. No memory isolation — all four pods share VRAM. Works fine for inference workloads that aren’t memory-hungry; terrible for training.
Debugging Checklist
When pods stay in Pending or CUDA errors appear at runtime, work through this list in order:
# 1. Is the GPU visible to the node?
kubectl describe node <node-name> | grep -A 5 "Allocatable"
# 2. Is the device plugin pod healthy?
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds
kubectl logs -n kube-system -l name=nvidia-device-plugin-ds --tail=50
# 3. Why is the pod pending?
kubectl describe pod <pod-name> | grep -A 20 "Events"
# Common outputs:
# "0/2 nodes are available: 1 Insufficient nvidia.com/gpu" — no GPU capacity
# "1 node(s) had untolerated taint" — missing toleration in pod spec
# 4. Can the container actually see CUDA?
kubectl exec -it <pod-name> -- nvidia-smi
# 5. Check containerd runtime config on the node
sudo k3s crictl info | python3 -m json.tool | grep -A 5 nvidia
Common trap: Nodes show nvidia.com/gpu: 0 after the plugin runs. This almost always means the containerd runtime isn’t configured or k3s was restarted without picking up the config.toml.tmpl. SSH to the node, run nvidia-container-cli info — if that fails, the container toolkit isn’t talking to the driver correctly, and no Kubernetes config will fix it.
Production-Ready Notes for Homelab
A few things that bite people once they move beyond "it works on my machine":
Pin your device plugin version. The NVIDIA plugin releases frequently and occasionally breaks API surface. Pinning to a tested version in your Helm values or manifest prevents surprise breakage after a helm upgrade.
Monitor GPU utilization. The device plugin exposes metrics at the kubelet level, but nvidia-smi in a DaemonSet or DCGM Exporter gives you Prometheus metrics you can alert on. You’ll know quickly if a pod is hogging all GPU memory and starving others.
Node drain before maintenance. GPU nodes running inference servers have long-lived connections. Always kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data before rebooting, so workloads reschedule cleanly instead of hard-failing.
Separate GPU and CPU workloads into namespaces with resource quotas. Set a ResourceQuota on your AI namespace limiting nvidia.com/gpu to prevent one runaway deployment from grabbing everything:
apiVersion: v1
kind: ResourceQuota
metadata:
name: gpu-quota
namespace: ai-workloads
spec:
hard:
requests.nvidia.com/gpu: "2"
limits.nvidia.com/gpu: "2"
Once you’ve gone through this setup, the scheduler does the right thing without babysitting. New GPU workloads land on GPU nodes, CPU workloads stay off them, and kubectl describe node gives you a live count of allocated versus available GPU resources. That’s the baseline for running anything serious — LLM inference, computer vision pipelines, or just Stable Diffusion without lag spikes because someone’s etcd scraper ate your VRAM.