If you’re still provisioning Kubernetes nodes on Ubuntu or CentOS like it’s 2019, you’re doing it the hard way. Package drift, manual updates, config snowflakes — each node slowly becomes its own special problem over time. Flatcar Container Linux solves this at the OS level, and after years of running it in production, I think it’s one of the most underrated infrastructure choices in the Kubernetes world.
This article is a complete walkthrough: what Flatcar is, why it wins for K8s nodes, how to provision it with Ignition, and where to avoid the landmines.
A Quick History Lesson (Worth Reading)
CoreOS shipped the idea of an immutable, container-first Linux distribution. Red Hat bought CoreOS in 2018, merged it into RHEL/OpenShift land, and killed the standalone Container Linux product in 2020. Kinvolk forked it as Flatcar Container Linux before the shutdown. Microsoft acquired Kinvolk in 2021, and — surprisingly — kept Flatcar genuinely open-source, donating it to the CNCF Sandbox in 2023.
The GitHub repo is at github.com/flatcar/flatcar.
As of 2026 the project is healthy: regular releases across Stable, Beta, and Alpha channels, a real governance model, and broad cloud/bare-metal support. It is not abandonware. That concern is dead.
Why Flatcar for Kubernetes Nodes
The core pitch is simple: immutable root filesystem, atomic OS updates, zero package manager.
There is no apt, no yum, no pip on the system. The root partition is read-only. The OS image ships as a complete A/B partition pair, and updates are applied by writing the new image to the inactive partition and rebooting into it. If the new version fails health checks, it rolls back automatically.
For Kubernetes nodes this is exactly what you want:
- No configuration drift. Every node is byte-for-byte identical unless you explicitly wrote a difference into its Ignition config.
- Reproducible provisioning. Boot, Ignition runs, node joins the cluster. That’s it.
- Reduced attack surface. No compiler, no package manager, no Python interpreter sitting around. The OS surface is tiny.
- Updates that don’t surprise you.
update_enginepluslocksmith(orFLUO— more on that shortly) coordinate reboots across the fleet so you don’t lose quorum.
The tradeoff is that you can’t apt install debugging tools. You run them as containers, or you use the toolbox utility that Flatcar ships, which drops you into a Fedora container with full access to the host namespace. Once you accept this mental model, it’s liberating rather than limiting.
What Flatcar Gives You Out of the Box
systemd(init, network, timers — everything)containerd+docker(both present, containerd is the right one to use for K8s)rktis gone — nobody misses itetcdis not bundled anymore; manage it separately- SSH with key-based auth (password auth disabled)
update_enginefor OS updates from official channelslocksmithfor coordinated reboot windows
The kernel is current and ships with all the relevant cgroup v2, eBPF, and io_uring patches. You will not fight with outdated kernel features on Flatcar.
Ignition: The Configuration Layer
Flatcar uses Ignition for first-boot provisioning. This is not cloud-init. Ignition runs exactly once, before systemd even starts, and it is declarative JSON (or Butane YAML that transpiles to JSON). No shell scripts running ad-hoc — everything is expressed as filesystem writes, systemd unit drops, and user creation.
You write Butane configs (human-readable YAML), compile them to Ignition JSON with the butane CLI, and pass the JSON to your VM/bare-metal provisioning tool.
Install butane on your workstation:
# Linux amd64
curl -sSLo butane https://github.com/coreos/butane/releases/latest/download/butane-x86_64-unknown-linux-gnu
chmod +x butane && sudo mv butane /usr/local/bin/
Step-by-Step: Provisioning a Flatcar K8s Node
1. Base Butane Config
Here’s a production-starting-point Butane config for a Kubernetes worker node using k3s. Adapt for kubeadm if you’re running full Kubernetes.
# node.bu — Butane config for a Flatcar k3s worker
variant: flatcar
version: 1.0.0
passwd:
users:
- name: core
# Generate with: openssl passwd -6 'yourpassword' (or use SSH keys only)
ssh_authorized_keys:
- "ssh-ed25519 AAAA...your-public-key core@k8s-nodes"
storage:
files:
# Kernel parameter tuning required by Kubernetes
- path: /etc/sysctl.d/90-k8s.conf
mode: 0644
contents:
inline: |
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288
# br_netfilter needs to be loaded before sysctl runs
- path: /etc/modules-load.d/k8s.conf
mode: 0644
contents:
inline: |
br_netfilter
overlay
# k3s installer script — baked in, not downloaded at boot
# Replace K3S_URL and K3S_TOKEN with your actual values
- path: /opt/bin/k3s-install.sh
mode: 0755
contents:
inline: |
#!/bin/bash
set -euo pipefail
export INSTALL_K3S_SKIP_DOWNLOAD=false
export K3S_URL="https://your-k3s-server:6443"
export K3S_TOKEN="your-node-token"
# Pin the k3s version — never use INSTALL_K3S_CHANNEL=stable in prod
export INSTALL_K3S_VERSION="v1.31.5+k3s1"
curl -sfL https://get.k3s.io | sh -
systemd:
units:
# Run k3s install on first boot only, then start the agent
- name: k3s-agent-install.service
enabled: true
contents: |
[Unit]
Description=Install and start k3s agent
After=network-online.target
Wants=network-online.target
ConditionPathExists=!/opt/bin/.k3s-installed
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/bin/k3s-install.sh
# Mark installed so we skip on subsequent reboots
ExecStartPost=/bin/touch /opt/bin/.k3s-installed
[Install]
WantedBy=multi-user.target
Compile to Ignition JSON:
butane --pretty --strict node.bu > node.ign
2. Provision on Hetzner Cloud (Example)
Hetzner supports passing Ignition configs directly as user-data. Use hcloud:
hcloud server create \
--name k8s-worker-01 \
--image flatcar-linux-stable \
--type cpx21 \
--ssh-key your-key \
--user-data-from-file node.ign \
--location nbg1
For bare metal with iPXE, the Flatcar documentation covers network booting directly from stable.release.flatcar-linux.net. The Ignition JSON gets served via HTTP and passed with ignition.config.url= on the kernel cmdline. This works reliably and is how I run it on physical clusters.
For VMware/Proxmox: use the OVA/QCOW2 images from the official release page and pass Ignition as a guestinfo.ignition.config.data base64-encoded VM property.
3. Verify the Node Joined
# On your workstation with kubectl configured
kubectl get nodes -o wide
# Should show the new node in Ready state within ~90 seconds
# NAME STATUS ROLES AGE VERSION
# k8s-worker-01 Ready <none> 2m v1.31.5+k3s1
OS Updates: Do Not Wing This
This is where most people get burned. Flatcar will auto-update and reboot nodes. If you do nothing, it might reboot three nodes simultaneously and take down your cluster.
Option 1: Locksmith (built-in, simpler)
Locksmith uses etcd to hold a reboot lock. Only one node reboots at a time. Configure it in your Butane:
storage:
files:
- path: /etc/coreos/update.conf
mode: 0644
contents:
inline: |
# Options: best-effort, etcd-lock, reboot, off
REBOOT_STRATEGY=etcd-lock
LOCKSMITHD_REBOOT_WINDOW_START=02:00
LOCKSMITHD_REBOOT_WINDOW_LENGTH=1h
LOCKSMITHD_ETCD_ENDPOINTS=https://your-etcd:2379
Option 2: FLUO — Flatcar Linux Update Operator (preferred for K8s clusters)
FLUO is a Kubernetes operator that manages Flatcar node reboots natively, respecting PodDisruptionBudgets and draining nodes before rebooting. This is what you want if you’re running a real cluster.
kubectl apply -f https://github.com/flatcar/flatcar-linux-update-operator/releases/latest/download/fluo.yaml
It watches the update_engine D-Bus signal on each node and coordinates reboots through the Kubernetes API. Nodes get cordoned and drained automatically. Zero manual intervention.
Option 3: Disable auto-updates, handle it yourself
Sometimes appropriate for air-gapped environments. Set GROUP=disabled in update.conf and run update_engine_client -update on a maintenance schedule. Fine, but don’t forget to actually do it.
Gotchas
Gotcha: /usr is read-only
You cannot install binaries into /usr/bin. Everything custom goes into /opt/bin. The k3s installer handles this natively (it writes to /usr/local/bin which is actually a symlink to /opt/bin on Flatcar). Other tools you drop in need to land in /opt/bin and be declared executable in Ignition. Do not fight the filesystem layout.
Gotcha: Ignition only runs once
Ignition is not cloud-init. If you change your Butane config after provisioning, the existing node does not reprovision. You need to either reprovision the node entirely or manage post-boot changes through another mechanism (Ansible on /opt paths, a fleet management tool, etc.). This is a feature, not a bug — but it surprises people.
Gotcha: SELinux context on /opt
Flatcar ships with SELinux in permissive mode by default. If you harden to enforcing, binaries in /opt/bin need proper labels. Run restorecon -R /opt/bin in a startup script or accept permissive for now. Most people leave it permissive and rely on Kubernetes RBAC + network policies for access control.
Gotcha: The toolbox is your debugger, not your package manager
When something breaks, you reflexively want to apt install strace. You can’t. Run toolbox instead — it gives you a full Fedora environment with access to the host. But do not use it to install permanent tooling. It’s ephemeral. For permanent auxiliary tools (CNI plugins, custom node agents), use systemd units that run containers or drop binaries into /opt/bin via Ignition.
Gotcha: containerd is the right runtime, Docker is legacy
Both are available, but Kubernetes uses containerd. The Docker socket is there for convenience but Kubernetes does not talk to it. Don’t configure your cluster to use Docker. The kubelet / k3s agent talks to containerd’s socket at /run/containerd/containerd.sock. Verify:
crictl --runtime-endpoint unix:///run/containerd/containerd.sock info
Gotcha: The OEM partition and AWS/GCP metadata
On cloud providers Flatcar reads Ignition from the instance metadata service, not from user-data in the traditional cloud-init format. AWS passes Ignition via user-data correctly, but some older automation tools still try to push cloud-init YAML. It will silently do nothing. Always verify Ignition applied by checking journalctl -u ignition-files.service on first boot.
Production-Ready Pattern: Immutable Node Fleet
The pattern I use in production:
-
One Butane template per role (control plane, worker, storage node). Variables (cluster endpoint, token, node labels) are substituted with
envsubstor a simple sed pass before runningbutane. No Helm here — this is OS config, keep it simple. -
Pin the Flatcar version. In your VM image reference or iPXE boot URL, use the specific version string (
3850.2.1) not the channel alias. Channels move. Pin the image in your CI, test upgrades explicitly, then bump the pin. -
GitOps the Butane configs. Every node config change is a PR. The compiled
.ignfile is committed alongside it (or generated in CI). You know exactly what every node class is running. -
Use FLUO, not locksmith, if you’re on Kubernetes. The Kubernetes-native drain behavior is worth it. Locksmith is fine for non-K8s Flatcar deployments.
-
Ship logs off-node immediately. Nodes can be wiped. Don’t rely on
journaldas your log store. Vector or Promtail as a DaemonSet, logs to Loki or Elasticsearch. -
Keep
/varfor stateful data, not OS state. Flatcar persists/varacross updates. Your kubelet state, container storage, and custom certificates live there. Don’t put things in paths that get wiped on update.
Flatcar vs. Alternatives in 2026
People always ask about Talos Linux in this conversation. Talos is more opinionated: the API is its only interface, no SSH at all, Kubernetes-only. Talos is excellent if Kubernetes is literally everything you run. Flatcar is better if you want a general-purpose immutable Linux that happens to run K8s really well — you might also run it for standalone workloads, edge devices, or mixed environments.
Bottlerocket (Amazon’s take) is fine if you live in AWS and never leave. The moment you go multi-cloud or bare metal it becomes awkward.
Ubuntu with cloud-init is what you use when you haven’t convinced your team to change yet. It works. It just accumulates entropy over time.
Flatcar sits in the sweet spot: immutable and container-first, but without Talos’s "you will do it our way" energy.
Wrapping Up
Flatcar is mature, CNCF-backed, and genuinely production-ready in 2026. The learning curve is real — mostly unlearning the assumption that you can SSH in and apt install your way out of problems. Once that mental model shifts, you end up with a cluster where every node is identical, updates don’t keep you up at night, and your attack surface is minimal.
Start with a single worker node, get comfortable with the Ignition workflow, then roll out FLUO and pin your versions. The whole stack is open-source and the community in the #flatcar channel on Kubernetes Slack is responsive.
The days of treating your Kubernetes nodes like pets are over. Flatcar is how you run cattle at scale.