Bottlerocket OS: The Container-Only Linux You’re Probably Ignoring (And Shouldn’t)

When AWS open-sourced Bottlerocket back in 2020, most people filed it away under "cool thing I’ll never use outside of EKS." That instinct is understandable — the OS was literally born inside Amazon’s data centers to serve Amazon’s needs. But spend a weekend with it on bare metal or a VMware cluster and you realize something: the architecture choices that make it good for AWS also make it genuinely interesting for anyone running a container-heavy homelab or a small production Kubernetes cluster.

This is not a vendor puff piece. We’re going to look at what Bottlerocket actually is, where it shines, where it frustrates you, and whether the trade-offs make sense if you don’t have an AWS account in sight.

Official repo: https://github.com/bottlerocket-os/bottlerocket

What Bottlerocket Actually Is

The elevator pitch is deceptively simple: it’s a Linux distribution that only runs containers. There is no package manager. There is no apt, yum, or pacman. You cannot SSH in and run arbitrary software. The root filesystem is read-only and integrity-checked with dm-verity. The only way to modify the system is through a controlled API surface.

If that sounds limiting, that’s because it is — deliberately. Most security incidents on container hosts happen through the host OS itself: a misconfigured SSH daemon, a forgotten debug tool with a CVE, someone running pip install directly on a worker node. Bottlerocket’s answer to that threat model is to remove the attack surface entirely rather than harden it.

The kernel is a custom build based on the latest stable series, built with a minimal config that only includes drivers and subsystems actually needed for container workloads. Userspace is similarly stripped. You get containerd (or Docker, depending on the variant), a small Bottlerocket API server called apiserver, and not much else.

Updates use an A/B partition scheme. The running OS sits on partition A. An update writes to partition B in the background. On the next reboot, the bootloader flips to B. If B fails to come up cleanly, the bootloader rolls back to A automatically. This is the same pattern as Android, ChromeOS, or CoreOS — and it makes zero-downtime node rolling updates in Kubernetes trivially safe.

The Variants — Which One Do You Want?

Bottlerocket ships multiple variants tuned for different runtimes and platforms. This matters because the variants have different kernel configs, different pre-installed software, and different default settings.

Variant	Container Runtime	Target Platform
`aws-k8s-1.30`	containerd	AWS EC2 + EKS
`aws-ecs-1`	containerd	AWS ECS
`metal-k8s-1.30`	containerd	Bare metal
`vmware-k8s-1.30`	containerd	vSphere
`metal-dev`	containerd	Bare metal (debug)

For self-hosters, metal-k8s-* and vmware-k8s-* are the ones to look at. The metal-dev variant is useful when you’re figuring things out — it includes the admin container enabled by default and has slightly looser security policies.

The Kubernetes version in the variant name is literal: metal-k8s-1.30 ships with the kubelet and kube-proxy binaries for Kubernetes 1.30. You pick the variant that matches your cluster version, which means you need to track updates as Kubernetes versions cycle.

Getting It Running: Bare Metal Setup

The Bottlerocket team publishes OVA files for VMware and raw disk images for bare metal. For a physical machine, the process looks like this:

Step 1: Download the image

# Replace with current version from https://github.com/bottlerocket-os/bottlerocket/releases
VERSION="1.20.4"
VARIANT="metal-k8s-1.30"
ARCH="x86_64"

wget "https://cache.bottlerocket.aws/${VARIANT}/${VERSION}/${ARCH}/bottlerocket-${VARIANT}-${ARCH}-${VERSION}.img.lz4"
wget "https://cache.bottlerocket.aws/${VARIANT}/${VERSION}/${ARCH}/bottlerocket-${VARIANT}-${ARCH}-${VERSION}.img.lz4.sha512sum"

# Verify integrity — do not skip this
sha512sum -c "bottlerocket-${VARIANT}-${ARCH}-${VERSION}.img.lz4.sha512sum"

Step 2: Write it to the target disk

# Decompress and write directly to disk
# WARNING: /dev/sda will be completely overwritten
lz4 -d "bottlerocket-${VARIANT}-${ARCH}-${VERSION}.img.lz4" - | sudo dd of=/dev/sda bs=512K status=progress

sudo sync

Bottlerocket images are pre-partitioned. After writing, you’ll see several GPT partitions: the two OS partitions (A and B), a boot partition, and a data partition. The data partition is where container images, kubelet state, and persistent configuration live. This is the only writable partition under normal operation.

Step 3: Inject user data

This is where Bottlerocket diverges from every other Linux distro you’ve used. There is no installer, no first-boot wizard. Configuration happens through user data — a TOML file that gets written to the boot partition before first boot. On EC2, this comes from instance metadata. On bare metal, you write it yourself.

# Mount the BOTTLEROCKET-SETTINGS partition and write config
sudo mount /dev/sda12 /mnt/bottlerocket-settings

sudo tee /mnt/bottlerocket-settings/user-data <<'EOF'
[settings.kubernetes]
cluster-name = "my-homelab"
cluster-certificate = "BASE64_ENCODED_CLUSTER_CA"
api-server = "https://10.0.0.10:6443"

[settings.kubernetes.node-labels]
"node.kubernetes.io/role" = "worker"
"environment" = "homelab"

[settings.host-containers.admin]
enabled = true
superpowered = true

[settings.host-containers.control]
enabled = true
EOF

sudo umount /mnt/bottlerocket-settings

The cluster-certificate and api-server values come from your existing Kubernetes cluster. kubeadm gives you these during cluster init. Get the CA cert with:

kubectl get cm kube-root-ca.crt -n kube-system -o jsonpath='{.data.ca\.crt}' | base64 -w0

Accessing the System: The Admin Container

Here’s the part that surprises people the most. After boot, you cannot SSH directly into Bottlerocket. There is no SSH daemon on the host. What you get instead is a concept called host containers — privileged containers that run alongside your workload containers and can access the host namespace.

The admin host container is essentially a busybox-style environment with nsenter baked in. You SSH into the container, then break out into the host namespace:

# The admin container exposes its own SSH daemon
# You need to inject your public key via user data first:

[settings.host-containers.admin]
enabled = true
superpowered = true
user-data = "BASE64_ENCODED_JSON"

# The user-data JSON looks like:
# {"ssh": {"authorized-keys": ["ssh-ed25519 AAAA... your-key"]}}

echo '{"ssh":{"authorized-keys":["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAI... your-key-here"]}}' | base64 -w0

Once you’re in:

# SSH to the admin container (port 22 on the node IP)
ssh [email protected]

# Enter the host namespace
sudo sheltie

# Now you're in a real shell on the host
# You can run apiclient, inspect state, etc.
apiclient get all

sheltie is the Bottlerocket-specific tool that drops you into the host OS namespace. It’s not a back door — it’s an intentional, auditable escalation path. The fact that you had to enable the admin container explicitly in user data means it’s opt-in.

Gotcha: The admin container’s SSH host keys regenerate on every boot unless you persist them. If you’re jumping between admin containers, you’ll get "host key changed" warnings constantly. Pre-generate the keys, base64-encode them, and inject them in user data to fix this.

Configuring the System with `apiclient`

Bottlerocket’s API server exposes all system configuration through a structured settings tree. You talk to it with apiclient. This replaces editing config files directly.

# Inside sheltie — read current settings
apiclient get settings

# Set a DNS server
apiclient set settings.network.hosts="10.0.0.1"

# Update NTP servers
apiclient set settings.ntp.time-servers='["10.0.0.1", "pool.ntp.org"]'

# Add a kernel sysctl
apiclient set 'settings.kernel.sysctl."net.ipv4.ip_forward"=1'

# Check pending changes before applying
apiclient get settings.pending

# Apply pending changes
apiclient apply

The settings are validated before applying. You can’t accidentally set an invalid value and break something — the API server rejects it. Changes to most settings take effect immediately. Changes requiring a reboot (kernel parameters, disk configuration) are flagged as such.

This model feels weird at first. You want to just edit /etc/resolv.conf and be done with it. But after you’ve had a worker node drift because someone SSH’d in and changed something by hand three months ago, the appeal of "you can’t do that here" becomes obvious.

Update Management

Bottlerocket includes updog, the update client, and the bottlerocket-update-operator for Kubernetes-aware rolling updates. On bare metal outside Kubernetes, updates are managed manually or via apiclient:

# Inside sheltie — check for available updates
apiclient update check

# If an update is available
apiclient update apply

# The update downloads to the inactive partition
# Reboot to activate
apiclient reboot

If the updated partition fails to mount correctly after reboot, the bootloader automatically falls back to the previous version. This is not optional and not configurable — it’s baked into the partition scheme.

For Kubernetes, the bottlerocket-update-operator watches your nodes, applies updates one at a time, drains workloads before rebooting, and waits for the node to rejoin before moving to the next. Deploy it into your cluster:

# Install the update operator
kubectl apply -f https://raw.githubusercontent.com/bottlerocket-os/bottlerocket-update-operator/main/deploy/bottlerocket-update-operator.yaml

# The operator uses node labels to identify Bottlerocket nodes
# These are set automatically by the kubelet on Bottlerocket
kubectl get nodes --show-labels | grep bottlerocket

Gotcha: The update operator only handles Bottlerocket nodes. If your cluster has mixed OS nodes (some Ubuntu, some Bottlerocket), the operator ignores the non-Bottlerocket nodes. You still need a separate strategy for them.

Running Kubernetes on Bottlerocket: The Real Experience

Joining a Bottlerocket bare metal node to an existing kubeadm cluster requires a few things the user data must handle correctly.

# Full working user-data example for a kubeadm cluster worker node

[settings.kubernetes]
cluster-name = "prod-homelab"
cluster-certificate = "LS0tLS1CRUdJTi..." # your base64 CA cert
api-server = "https://10.0.0.10:6443"
cluster-dns-ip = "10.96.0.10"

# Bootstrap token from kubeadm token create --print-join-command
bootstrap-token = "abcdef.0123456789abcdef"

# Node labels and taints
[settings.kubernetes.node-labels]
"node-role.kubernetes.io/worker" = ""
"bottlerocket.aws/updater-interface-version" = "2.0.0"

[settings.kubernetes.node-taints]
# Leave empty unless you want to taint this node

# containerd configuration
[settings.container-runtime]
# Max container log size per file before rotation
max-container-log-line-size = 16384

# Host containers
[settings.host-containers.admin]
enabled = true
superpowered = true
user-data = "BASE64_OF_SSH_KEY_JSON"

[settings.host-containers.control]
enabled = true

# Network
[settings.network]
hostname = "worker-01"

[settings.ntp]
time-servers = ["10.0.0.1"]

One important detail: Bottlerocket uses containerd, not Docker, as the container runtime. If your cluster was set up assuming Docker (old tutorials, old tooling), you may need to adjust. Modern kubeadm defaults to containerd anyway, so this is rarely an issue on clusters built in the last two years.

Gotcha: Bottlerocket does not support dockershim. If you have custom tooling that talks to the Docker socket directly — old CI pipelines, monitoring agents that use docker stats — those will break. You need to switch them to the containerd CRI endpoint at /run/containerd/containerd.sock or the CRI socket at /run/containerd/containerd.sock.

Container Image Pulls and Storage

Bottlerocket allocates the data partition for container image storage. By default this is whatever’s left on disk after the OS partitions. On a 50GB disk with default partition sizes, you typically get around 44GB for data.

If you’re running image-heavy workloads (ML inference containers, fat Java services), you can configure a separate data volume:

# In user-data — external data disk
[settings.kubernetes]
# ... other settings

# Point kubelet to a separate, larger disk for image storage
# This requires the disk to be pre-formatted ext4 or xfs
[settings.host-volumes."kubernetes"]
source-device-path = "/dev/sdb"

For content-addressable storage (pulling the same base image across many containers), containerd’s image sharing already handles deduplication. You won’t store the same layer twice for containers sharing a base image.

Networking and CNI

Bottlerocket does not include a CNI plugin. You bring your own, same as you would on any Kubernetes node OS. Tested and commonly used:

Cilium — works well, BPF programs load correctly, no issues
Flannel — simple, works fine
Calico — works, the eBPF mode requires some kernel config verification first
Weave — works but the kernel module loading needs the right sysctl settings in user data

The host-level firewall is managed by Bottlerocket’s API and is intentionally minimal. You’re expected to handle network policy at the Kubernetes layer, not the host level.

The Self-Hoster Verdict

Where Bottlerocket earns its keep:

Kubernetes worker nodes. Full stop. If you’re running a home or small production cluster with 3-10 nodes that only run containers, Bottlerocket removes an entire category of operational overhead. No distro updates to manage, no "what did someone install on this node" drift, no wondering why a worker is behaving oddly because someone apt upgraded something three weeks ago.

The A/B update model is genuinely superior to in-place OS upgrades. Rolling back from a bad update is automatic and instant. You’ll never have a node stuck in a half-upgraded state.

Where it hurts:

Debugging is harder. When something goes wrong below the container level — a flaky NIC driver, a weird kernel panic, a storage issue — you’re in sheltie with a minimal toolset. You can’t install strace or tcpdump without pulling them as a container. The admin container has some tools, but it’s not a full Linux environment.

Not having a package manager is fine for production but annoying for experimentation. If you’re the type who likes to try things on nodes (a habit you should break anyway), Bottlerocket will fight you constantly.

The configuration-via-user-data model means you need to get config right before boot, or reimage. There’s no "just change this line in /etc/kubelet.conf" escape hatch.

The bottom line:

Bottlerocket is not for people who want to "manage servers." It’s for people who want servers to manage themselves while they manage applications. If that’s your model — and for container-only workloads, it should be — it’s genuinely good, and the bare metal and VMware support is more capable than the documentation implies.

Gotchas Summary

SSH goes to the admin container, not the host. Enable it in user data before boot or you’re locked out.
No package manager means debugging tools must be containerized or pre-baked into the admin container image.
Variant selection is hard-pinned to a Kubernetes version. When you upgrade K8s, you’re reimaging the node, not upgrading in place.
The data partition is automatically sized but not resized. Plan your disk layout before writing the image.
Update operator in mixed-OS clusters only handles Bottlerocket nodes. Don’t forget your Ubuntu nodes.
Container runtime is containerd only. Any tooling assuming a Docker socket needs to be updated.
The metal-dev variant is useful for learning but has relaxed security settings. Don’t run it in production.

Production-Ready Checklist

Before you put a Bottlerocket node in front of real traffic:

Pre-generate and inject SSH host keys for the admin container — eliminates host key warnings and TOFU attacks.
Set explicit NTP servers in user data — the default pools work but explicit internal servers are faster and more reliable on private networks.
Deploy the bottlerocket-update-operator — manual updates are fine for a homelab, not acceptable for anything that needs uptime guarantees.
Configure node labels and taints in user data, not with kubectl label after the fact — they’ll disappear on node replacement.
Keep a golden user-data template in version control. When you replace a node (and with Bottlerocket’s model, you will replace rather than repair), that template is your entire node configuration.
Monitor the inactive partition’s update state. apiclient update check can be scraped into your monitoring stack — knowing "there’s an update available" before it matters is worth the 30 minutes to wire it up.

The project moves quickly. Watch the releases page — new variants for new Kubernetes versions drop on a predictable cadence and support windows are explicit. Don’t let a variant age out under you.