You have a Kubernetes cluster. You have models — Mistral, Llama, Gemma, whatever. And right now, running them means a pile of hand-crafted Deployments, Services, PVCs, init containers that pull weights, and shell scripts duct-taped together to tear it all down. Every time you want a new model, you copy-paste, adjust names, forget to update a label somewhere, and spend 40 minutes debugging why the pod won’t schedule.
There’s a better way: a Kubernetes operator purpose-built for the LLM lifecycle. One kubectl apply to deploy a model. One kubectl delete to cleanly tear it down. Status conditions that tell you exactly what phase the model is in. It’s not magic — it’s just treating your models as first-class Kubernetes citizens.
This tutorial walks you through building that operator from scratch: the CRD, the reconciler loop, and all three lifecycle phases — deploy, serve, and destroy. We’ll use Go, kubebuilder, and controller-runtime.
What You’ll Actually Build
A custom resource called LLMModel. When you apply one, the operator:
- Deploys — creates a PVC and a init-container Job that pulls the model weights from a registry or object store.
- Serves — once weights are ready, spins up the inference server (vLLM, llama.cpp server, Ollama, your choice) behind a stable
Service. - Destroys — on deletion, uses a finalizer to cleanly shut down the server and optionally purge the weights volume before releasing the object.
You get status conditions, so kubectl get llmmodels actually tells you something useful.
Prerequisites
- Go 1.22+
- kubebuilder v3 CLI (
go install sigs.k8s.io/kubebuilder/cmd/kubebuilder@latest) - A running cluster (k3s, kind, or the real thing — doesn’t matter)
- Basic familiarity with Go and Kubernetes resource model
Scaffold the Project
mkdir llm-operator && cd llm-operator
kubebuilder init --domain ai.example.com --repo github.com/yourorg/llm-operator
kubebuilder create api --group inference --version v1alpha1 --kind LLMModel
Answer yes to both "Create Resource" and "Create Controller". This gives you the skeleton: api/v1alpha1/llmmodel_types.go for your types and internal/controller/llmmodel_controller.go for the reconciler.
Design the CRD
The types file is where you define what users can actually configure. Keep the spec tight — every field you add is a field you have to handle in the reconciler.
// api/v1alpha1/llmmodel_types.go
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/api/resource"
)
// ModelSource defines where to pull weights from.
type ModelSource struct {
// HuggingFace repo ID, e.g. "mistralai/Mistral-7B-Instruct-v0.3"
HuggingFaceRepo string `json:"huggingFaceRepo,omitempty"`
// Direct URL (S3, GCS signed URL, HTTP). Mutually exclusive with HuggingFaceRepo.
URL string `json:"url,omitempty"`
// Secret name containing HF_TOKEN or cloud credentials.
CredentialsSecret string `json:"credentialsSecret,omitempty"`
}
// ServeConfig controls the inference server container.
type ServeConfig struct {
// Container image for the inference server.
// e.g. "vllm/vllm-openai:latest" or "ghcr.io/ggerganov/llama.cpp:server"
Image string `json:"image"`
// Port the inference server listens on inside the container.
// +kubebuilder:default=8000
Port int32 `json:"port,omitempty"`
// Extra args passed verbatim to the server entrypoint.
Args []string `json:"args,omitempty"`
// GPU resource request. Omit for CPU-only.
GPUCount *int64 `json:"gpuCount,omitempty"`
}
// LLMModelSpec defines the desired state.
type LLMModelSpec struct {
Source ModelSource `json:"source"`
Serve ServeConfig `json:"serve"`
// Storage size for the model weights PVC.
// +kubebuilder:default="50Gi"
StorageSize resource.Quantity `json:"storageSize,omitempty"`
// StorageClass for the PVC. Empty means cluster default.
StorageClass string `json:"storageClass,omitempty"`
// Whether to delete the PVC when the LLMModel is deleted.
// +kubebuilder:default=false
PurgeWeightsOnDelete bool `json:"purgeWeightsOnDelete,omitempty"`
}
// LLMModelPhase is the high-level lifecycle phase.
// +kubebuilder:validation:Enum=Pending;Pulling;Serving;Terminating;Failed
type LLMModelPhase string
const (
PhasePending LLMModelPhase = "Pending"
PhasePulling LLMModelPhase = "Pulling"
PhaseServing LLMModelPhase = "Serving"
PhaseTerminating LLMModelPhase = "Terminating"
PhaseFailed LLMModelPhase = "Failed"
)
// LLMModelStatus reflects the observed state.
type LLMModelStatus struct {
Phase LLMModelPhase `json:"phase,omitempty"`
Conditions []metav1.Condition `json:"conditions,omitempty"`
// ServiceName is the stable DNS name for the inference endpoint.
ServiceName string `json:"serviceName,omitempty"`
// InferenceURL is the full base URL: http://<svc>.<ns>.svc.cluster.local:<port>
InferenceURL string `json:"inferenceURL,omitempty"`
}
// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Phase",type="string",JSONPath=".status.phase"
// +kubebuilder:printcolumn:name="Model",type="string",JSONPath=".spec.source.huggingFaceRepo"
// +kubebuilder:printcolumn:name="URL",type="string",JSONPath=".status.inferenceURL"
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
type LLMModel struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec LLMModelSpec `json:"spec,omitempty"`
Status LLMModelStatus `json:"status,omitempty"`
}
// +kubebuilder:object:root=true
type LLMModelList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []LLMModel `json:"items"`
}
func init() { SchemeBuilder.Register(&LLMModel{}, &LLMModelList{}) }
Run make generate && make manifests after any type change. This regenerates deepcopy functions and the CRD YAML in config/crd/.
Write the Reconciler
The reconciler is the heart of the operator. controller-runtime calls Reconcile() whenever the object changes or a watch event fires. The key insight: reconcile is idempotent, always. It looks at the world, compares it to desired state, and nudges toward that desired state. It never assumes it’s seeing a fresh object.
// internal/controller/llmmodel_controller.go
package controller
import (
"context"
"fmt"
batchv1 "k8s.io/api/batch/v1"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/api/resource"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/types"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
"sigs.k8s.io/controller-runtime/pkg/log"
inferencev1alpha1 "github.com/yourorg/llm-operator/api/v1alpha1"
)
const finalizerName = "inference.ai.example.com/cleanup"
type LLMModelReconciler struct {
client.Client
Scheme *runtime.Scheme
}
func (r *LLMModelReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
log := log.FromContext(ctx)
model := &inferencev1alpha1.LLMModel{}
if err := r.Get(ctx, req.NamespacedName, model); err != nil {
return ctrl.Result{}, client.IgnoreNotFound(err)
}
// --- Deletion path ---
if !model.DeletionTimestamp.IsZero() {
return r.reconcileDelete(ctx, model)
}
// --- Ensure finalizer is present ---
if !controllerutil.ContainsFinalizer(model, finalizerName) {
controllerutil.AddFinalizer(model, finalizerName)
if err := r.Update(ctx, model); err != nil {
return ctrl.Result{}, err
}
return ctrl.Result{Requeue: true}, nil
}
// --- Normal lifecycle ---
if err := r.ensurePVC(ctx, model); err != nil {
return ctrl.Result{}, r.setFailed(ctx, model, "PVCFailed", err)
}
// Check if pull Job is needed or already done.
jobDone, err := r.reconcilePullJob(ctx, model)
if err != nil {
return ctrl.Result{}, r.setFailed(ctx, model, "PullFailed", err)
}
if !jobDone {
// Still pulling. Re-queue and wait for the Job watch to trigger us.
log.Info("model weights still pulling, waiting")
return ctrl.Result{}, nil
}
// Weights are ready — ensure the inference server is up.
if err := r.ensureInferenceDeployment(ctx, model); err != nil {
return ctrl.Result{}, r.setFailed(ctx, model, "DeployFailed", err)
}
if err := r.ensureService(ctx, model); err != nil {
return ctrl.Result{}, r.setFailed(ctx, model, "ServiceFailed", err)
}
return ctrl.Result{}, r.setPhase(ctx, model, inferencev1alpha1.PhaseServing)
}
Phase 1: Deploy — Pull the Weights
Model weights can be hundreds of gigabytes. You don’t want to bake them into an image. The right approach is a one-shot Kubernetes Job with an init-style pull container that writes to a shared PVC. The inference server then mounts that PVC read-only.
func (r *LLMModelReconciler) ensurePVC(ctx context.Context, model *inferencev1alpha1.LLMModel) error {
pvc := &corev1.PersistentVolumeClaim{}
err := r.Get(ctx, types.NamespacedName{Name: pvcName(model), Namespace: model.Namespace}, pvc)
if err == nil {
return nil // already exists
}
if !apierrors.IsNotFound(err) {
return err
}
storageClass := model.Spec.StorageClass
desired := &corev1.PersistentVolumeClaim{
ObjectMeta: metav1.ObjectMeta{
Name: pvcName(model),
Namespace: model.Namespace,
},
Spec: corev1.PersistentVolumeClaimSpec{
AccessModes: []corev1.PersistentVolumeAccessMode{corev1.ReadWriteOnce},
StorageClassName: &storageClass,
Resources: corev1.VolumeResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceStorage: model.Spec.StorageSize,
},
},
},
}
_ = controllerutil.SetControllerReference(model, desired, r.Scheme)
return r.Create(ctx, desired)
}
func (r *LLMModelReconciler) reconcilePullJob(ctx context.Context, model *inferencev1alpha1.LLMModel) (bool, error) {
job := &batchv1.Job{}
err := r.Get(ctx, types.NamespacedName{Name: pullJobName(model), Namespace: model.Namespace}, job)
if apierrors.IsNotFound(err) {
// Create the pull Job and set phase to Pulling.
if setErr := r.setPhase(ctx, model, inferencev1alpha1.PhasePulling); setErr != nil {
return false, setErr
}
return false, r.createPullJob(ctx, model)
}
if err != nil {
return false, err
}
// Job exists — check completion.
for _, cond := range job.Status.Conditions {
if cond.Type == batchv1.JobComplete && cond.Status == corev1.ConditionTrue {
return true, nil
}
if cond.Type == batchv1.JobFailed && cond.Status == corev1.ConditionTrue {
return false, fmt.Errorf("pull job failed: %s", cond.Message)
}
}
return false, nil // still running
}
func (r *LLMModelReconciler) createPullJob(ctx context.Context, model *inferencev1alpha1.LLMModel) error {
backoffLimit := int32(2)
env := []corev1.EnvVar{
{Name: "MODEL_DEST", Value: "/model-weights"},
}
if model.Spec.Source.HuggingFaceRepo != "" {
env = append(env, corev1.EnvVar{Name: "HF_REPO", Value: model.Spec.Source.HuggingFaceRepo})
}
if model.Spec.Source.URL != "" {
env = append(env, corev1.EnvVar{Name: "MODEL_URL", Value: model.Spec.Source.URL})
}
if model.Spec.Source.CredentialsSecret != "" {
env = append(env, corev1.EnvVar{
Name: "HF_TOKEN",
ValueFrom: &corev1.EnvVarSource{
SecretKeyRef: &corev1.SecretKeySelector{
LocalObjectReference: corev1.LocalObjectReference{Name: model.Spec.Source.CredentialsSecret},
Key: "HF_TOKEN",
},
},
})
}
job := &batchv1.Job{
ObjectMeta: metav1.ObjectMeta{
Name: pullJobName(model),
Namespace: model.Namespace,
},
Spec: batchv1.JobSpec{
BackoffLimit: &backoffLimit,
Template: corev1.PodTemplateSpec{
Spec: corev1.PodSpec{
RestartPolicy: corev1.RestartPolicyOnFailure,
Containers: []corev1.Container{
{
Name: "pull",
// huggingface-cli pull or wget, depending on source.
// Use a thin image that has huggingface_hub installed.
Image: "ghcr.io/yourorg/model-puller:latest",
Env: env,
VolumeMounts: []corev1.VolumeMount{
{Name: "weights", MountPath: "/model-weights"},
},
Resources: corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("500m"),
corev1.ResourceMemory: resource.MustParse("1Gi"),
},
},
},
},
Volumes: []corev1.Volume{
{
Name: "weights",
VolumeSource: corev1.VolumeSource{
PersistentVolumeClaim: &corev1.PVCVolumeSource{
ClaimName: pvcName(model),
},
},
},
},
},
},
},
}
_ = controllerutil.SetControllerReference(model, job, r.Scheme)
return r.Create(ctx, job)
}
Phase 2: Serve — The Inference Deployment
Once the pull Job completes, the reconciler moves to creating the inference Deployment and Service. The weights PVC is mounted read-only — the inference server doesn’t need write access and you don’t want it accidentally corrupting weights.
func (r *LLMModelReconciler) ensureInferenceDeployment(ctx context.Context, model *inferencev1alpha1.LLMModel) error {
replicas := int32(1)
labels := map[string]string{"app": model.Name, "llm-operator/model": model.Name}
resources := corev1.ResourceRequirements{
Requests: corev1.ResourceList{
corev1.ResourceCPU: resource.MustParse("2"),
corev1.ResourceMemory: resource.MustParse("8Gi"),
},
}
if model.Spec.Serve.GPUCount != nil {
resources.Limits = corev1.ResourceList{
"nvidia.com/gpu": *resource.NewQuantity(*model.Spec.Serve.GPUCount, resource.DecimalSI),
}
}
container := corev1.Container{
Name: "inference-server",
Image: model.Spec.Serve.Image,
Args: model.Spec.Serve.Args,
Resources: resources,
Ports: []corev1.ContainerPort{
{ContainerPort: model.Spec.Serve.Port, Protocol: corev1.ProtocolTCP},
},
VolumeMounts: []corev1.VolumeMount{
{Name: "weights", MountPath: "/model-weights", ReadOnly: true},
},
// Basic liveness: the server must respond on its HTTP port.
LivenessProbe: &corev1.Probe{
ProbeHandler: corev1.ProbeHandler{
HTTPGet: &corev1.HTTPGetAction{
Path: "/health",
Port: intstr.FromInt32(model.Spec.Serve.Port),
},
},
InitialDelaySeconds: 30,
PeriodSeconds: 15,
},
}
desired := buildDeployment(model, replicas, labels, container)
_ = controllerutil.SetControllerReference(model, desired, r.Scheme)
existing := &appsv1.Deployment{}
err := r.Get(ctx, types.NamespacedName{Name: model.Name, Namespace: model.Namespace}, existing)
if apierrors.IsNotFound(err) {
return r.Create(ctx, desired)
}
if err != nil {
return err
}
// Patch only — don't replace. Avoids stomping on fields we don't own.
patch := client.MergeFrom(existing.DeepCopy())
existing.Spec = desired.Spec
return r.Patch(ctx, existing, patch)
}
func (r *LLMModelReconciler) ensureService(ctx context.Context, model *inferencev1alpha1.LLMModel) error {
svcName := model.Name
desired := &corev1.Service{
ObjectMeta: metav1.ObjectMeta{Name: svcName, Namespace: model.Namespace},
Spec: corev1.ServiceSpec{
Selector: map[string]string{"llm-operator/model": model.Name},
Ports: []corev1.ServicePort{
{Port: model.Spec.Serve.Port, TargetPort: intstr.FromInt32(model.Spec.Serve.Port)},
},
},
}
_ = controllerutil.SetControllerReference(model, desired, r.Scheme)
existing := &corev1.Service{}
err := r.Get(ctx, types.NamespacedName{Name: svcName, Namespace: model.Namespace}, existing)
if apierrors.IsNotFound(err) {
if err := r.Create(ctx, desired); err != nil {
return err
}
} else if err != nil {
return err
}
// Update status with the stable DNS name.
model.Status.ServiceName = svcName
model.Status.InferenceURL = fmt.Sprintf("http://%s.%s.svc.cluster.local:%d",
svcName, model.Namespace, model.Spec.Serve.Port)
return r.Status().Update(ctx, model)
}
Phase 3: Destroy — Finalizers Are Not Optional
This is where most tutorials hand-wave and say "Kubernetes garbage collection handles it." It doesn’t — not cleanly. If you want to optionally purge the weights PVC on deletion, you need a finalizer. Without one, the PVC can linger indefinitely because ownerReferences only cascade deletion of namespaced objects in the same namespace, and PVCs bound to running pods won’t be deleted anyway.
func (r *LLMModelReconciler) reconcileDelete(ctx context.Context, model *inferencev1alpha1.LLMModel) (ctrl.Result, error) {
if !controllerutil.ContainsFinalizer(model, finalizerName) {
return ctrl.Result{}, nil
}
_ = r.setPhase(ctx, model, inferencev1alpha1.PhaseTerminating)
// Scale down the inference Deployment to 0 before touching the PVC.
// This prevents the PVC from being stuck "in use".
if err := r.scaleDownInference(ctx, model); err != nil {
return ctrl.Result{}, err
}
if model.Spec.PurgeWeightsOnDelete {
pvc := &corev1.PersistentVolumeClaim{}
err := r.Get(ctx, types.NamespacedName{Name: pvcName(model), Namespace: model.Namespace}, pvc)
if err == nil {
if err := r.Delete(ctx, pvc); err != nil && !apierrors.IsNotFound(err) {
return ctrl.Result{}, err
}
}
}
controllerutil.RemoveFinalizer(model, finalizerName)
return ctrl.Result{}, r.Update(ctx, model)
}
func (r *LLMModelReconciler) scaleDownInference(ctx context.Context, model *inferencev1alpha1.LLMModel) error {
dep := &appsv1.Deployment{}
err := r.Get(ctx, types.NamespacedName{Name: model.Name, Namespace: model.Namespace}, dep)
if apierrors.IsNotFound(err) {
return nil
}
if err != nil {
return err
}
zero := int32(0)
dep.Spec.Replicas = &zero
return r.Update(ctx, dep)
}
Wire Up Watches and Register
In SetupWithManager, you want the controller to re-reconcile whenever a watched child resource changes — particularly the pull Job, so the reconciler wakes up the moment weights are ready without polling.
func (r *LLMModelReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&inferencev1alpha1.LLMModel{}).
Owns(&batchv1.Job{}). // wake on Job complete
Owns(&appsv1.Deployment{}).
Owns(&corev1.Service{}).
Owns(&corev1.PersistentVolumeClaim{}).
Complete(r)
}
The Owns() calls set up watches with an ownerReference filter. No polling. Pure event-driven.
Example CRD Manifest
Here’s what a user actually writes to deploy a model:
apiVersion: inference.ai.example.com/v1alpha1
kind: LLMModel
metadata:
name: mistral-7b
namespace: inference
spec:
source:
huggingFaceRepo: "mistralai/Mistral-7B-Instruct-v0.3"
credentialsSecret: hf-credentials # must have HF_TOKEN key
serve:
image: "vllm/vllm-openai:v0.4.2"
port: 8000
gpuCount: 1
args:
- "--model"
- "/model-weights"
- "--tensor-parallel-size"
- "1"
storageSize: "20Gi"
storageClass: "local-path"
purgeWeightsOnDelete: false
kubectl get llmmodels -n inference will show you Phase, Model, and URL columns thanks to the printcolumn markers you put on the type.
Gotchas
The PVC "in use" trap. If you try to delete a PVC while a pod is mounted to it, Kubernetes will leave it in Terminating indefinitely. Always scale the Deployment to zero and wait for the pod to die before deleting the PVC. The finalizer pattern above handles this, but you need to verify the pod actually terminated — consider adding a short ctrl.Result{RequeueAfter: 5 * time.Second} after scale-down instead of assuming it’s instant.
Job TTL vs. finalizers. Set ttlSecondsAfterFinished on your pull Job if you don’t want completed jobs cluttering the namespace. But don’t set it so low that the reconciler never sees the Complete condition. 300 seconds is a safe floor.
ownerReferences and cross-namespace objects. Kubernetes doesn’t allow cross-namespace owner references. If you want to share a weight PVC across namespaces (e.g., a cluster-scoped model registry), you’ll need to manage cleanup manually — the garbage collector won’t help you there.
Inference server startup time. A 7B model loading into GPU VRAM can take 60-90 seconds. Your liveness probe’s initialDelaySeconds needs to reflect that or Kubernetes will kill the pod in a restart loop before it ever serves a request. Tune this per model size.
Status update races. After r.Status().Update(), the object version in your local variable is stale. If you then try to r.Update() the main object (e.g., to add a finalizer), you’ll hit a conflict error. Use r.Get() to re-fetch before any second update, or use retry.RetryOnConflict.
Production-Ready Additions
RBAC from the start. kubebuilder generates //+kubebuilder:rbac markers. Pin them to exactly what the controller needs — Jobs, Deployments, Services, PVCs, and your CRD. Don’t use cluster-admin for an operator that only touches one namespace.
Metrics. controller-runtime exposes Prometheus metrics at /metrics by default. Add a ServiceMonitor CR if you’re running Prometheus Operator. Track reconcile duration and error rates per model.
Webhook for defaulting and validation. kubebuilder create webhook --kind LLMModel --defaulting --programmatic-validation. Reject gpuCount: 0 (meaningless), enforce that exactly one of huggingFaceRepo or url is set, default port to 8000. Catching bad specs at admission is far cheaper than debugging a failed Job at 2am.
Multi-replica serving. The current design uses replicas: 1. For production, expose replicas in the spec, add HorizontalPodAutoscaler management in the reconciler, and switch the PVC to ReadWriteMany if your storage class supports it — or restructure to use a read-only volume from a pre-populated snapshot.
Leader election. Always run with --leader-elect=true in production. The manager.Options has LeaderElection: true and LeaderElectionID: "llm-operator-leader". Without it, running two replicas of your controller for HA will result in split-brain reconciliation.
The full operator from here compiles with make build and deploys with make deploy IMG=yourrepo/llm-operator:latest. The CRD installs with make install. Once it’s running, your LLM deployments are a single YAML file and a kubectl apply away — no more shell scripts, no more copy-pasted Deployments, and a clean kubectl delete that actually cleans up after itself.