You’ve been there. The CI pipeline passes green. You pull the branch locally, run the build, and it fails in a completely different way. Your colleague’s machine builds fine. Yours doesn’t. You spend two hours debugging PATH differences, wrong Node version, missing system library. The build works — just not here, not now.
This is the problem Earthly was built to solve. Not another YAML-soup CI abstraction, not a glorified wrapper around npm run build. Earthly borrows the containerized execution model from Docker and the dependency graph model from Makefile, combines them with BuildKit’s caching engine, and gives you builds that run identically on a developer laptop, in GitHub Actions, or on a bare-metal Jenkins node. Same inputs, same outputs, every time.
The official repo: https://github.com/earthly/earthly
Let’s skip the theory and get into it.
Why Existing Tools Fall Apart
Makefiles are fine for C projects from 2003. The moment you have a polyglot monorepo with Node, Python, and Go sharing a build pipeline, they turn into an unmaintainable mess of .PHONY targets, implicit environment assumptions, and shell quoting nightmares.
Dockerfiles give you isolation, but they’re not build orchestrators. They build images — they don’t run test suites, generate artifacts, coordinate between services, or produce multiple outputs from a single build graph.
CI YAML (GitHub Actions, GitLab CI, CircleCI) solves the orchestration problem but locks your build logic to a specific platform. You can’t run your GitHub Actions workflow locally without jumping through Docker hoops or installing a third-party runner emulator. You can’t test it without pushing a commit. You can’t share steps between workflows without custom actions.
Earthly sits between all of these. It looks like a Dockerfile. It runs like Make. It executes entirely in containers. And it runs the same on your laptop as it does in CI.
Installing Earthly
# Linux / macOS (the one-liner is official, source is on GitHub)
/bin/sh -c "$(curl -fsSL https://get.earthly.dev/earthly.sh)"
# Or via brew on macOS
brew install earthly/earthly/earthly
# Confirm Docker (or Podman) is running — Earthly needs a container daemon
earthly bootstrap
Earthly runs on top of BuildKit. The bootstrap command installs a BuildKit satellite daemon and configures Docker to use it. You only need this once per machine.
Gotcha: If you have Docker Desktop with its own BuildKit version, there can be conflicts. Run
earthly bootstrap --with-buildkit-imageto let Earthly manage the BuildKit version explicitly instead of inheriting Docker’s.
The Earthfile: Core Concepts
An Earthfile lives in your project root (or any subdirectory for monorepos). The syntax will feel familiar if you’ve written Dockerfiles, but the mental model is closer to Make targets.
VERSION 0.8
# This is a "base" target — shared foundation for other targets
build-base:
FROM golang:1.22-alpine
WORKDIR /app
# Copy only the dependency files first — cache layer trick
COPY go.mod go.sum .
RUN go mod download
build:
FROM +build-base
COPY . .
RUN go build -o output/myapp ./cmd/myapp
SAVE ARTIFACT output/myapp AS LOCAL ./dist/myapp
test:
FROM +build-base
COPY . .
RUN go test ./... -cover
docker:
FROM alpine:3.19
COPY +build/myapp /usr/local/bin/myapp
ENTRYPOINT ["/usr/local/bin/myapp"]
SAVE IMAGE myapp:latest
Run a target:
earthly +test
earthly +build
earthly +docker
Three concepts to internalize:
Targets are prefixed with +. They’re like Make targets but always containerized.
FROM +target lets you inherit the filesystem state from another target. +build-base downloads dependencies once; +build and +test both start from that cached layer.
SAVE ARTIFACT extracts files from the container into your local filesystem. SAVE IMAGE pushes an image to your local Docker daemon or a registry.
How the Cache Actually Works
This is where Earthly earns its keep. It uses BuildKit’s layer caching — same mechanism as docker build — but applied to the entire build graph, not just a single Dockerfile.
Local Cache
By default, every target’s layers are cached on your local machine. If you change only main.go and re-run earthly +build, the go mod download layer hits cache. Only the compilation step re-runs.
The cache key is computed from:
- The base image digest
- The contents of every
COPY-ed file - The exact
RUNcommand string
This is why splitting dependency installation from source compilation matters. The classic pattern:
# Good — deps cached separately from source
build:
FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json .
RUN npm ci # cached unless package-lock.json changes
COPY src/ .
RUN npm run build # re-runs when src/ changes
vs the naive version that re-downloads node_modules on every source change.
Remote Cache
For CI, local cache is useless — each runner is stateless. Earthly supports remote cache via a container registry:
# Push cache after a build
earthly --remote-cache=registry.example.com/myapp/cache +build
# Pull cache before a build (on fresh CI runner)
earthly --remote-cache=registry.example.com/myapp/cache +build
When the --remote-cache flag is set, Earthly pushes intermediate layer manifests to your registry after a successful build and pulls them at the start of subsequent builds. Cache hits on a cold CI runner are common for the dependency-heavy early layers.
Gotcha: Remote cache storage costs money. Each layer gets pushed as a registry manifest. On a large monorepo with many targets, this can get large fast. Use registry lifecycle policies to expire old cache tags. A
cache-tag prefix makes them easy to identify.
Cache Mounts (Advanced)
For package managers with their own disk caches (pip, cargo, maven), Earthly supports BuildKit cache mounts:
build:
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
pip install -r requirements.txt
COPY . .
RUN python -m pytest
The --mount=type=cache instruction reuses a named cache volume across builds without committing it to the image layer. Cargo builds go from 8 minutes to 45 seconds once the dependency cache is warm.
Parallelism: Build Multiple Targets Concurrently
Earthly’s dependency graph enables automatic parallelism. Targets with no dependency on each other run concurrently:
VERSION 0.8
test-unit:
FROM +deps
RUN go test ./internal/...
test-integration:
FROM +deps
RUN go test ./integration/... -tags integration
lint:
FROM +deps
RUN golangci-lint run
all:
BUILD +test-unit
BUILD +test-integration
BUILD +lint
Run earthly +all and all three targets execute in parallel. You get the same wall-clock time as your slowest target, not the sum.
This is a big deal. On a typical CI pipeline where lint, unit tests, and integration tests run sequentially, Earthly often cuts total build time by 40-60% just by running them concurrently.
Explicit Parallelism with WAIT
If you need to collect artifacts from parallel targets before proceeding:
docker-release:
WAIT
BUILD +build-linux-amd64
BUILD +build-linux-arm64
END
# Both artifacts are now available locally
RUN ./scripts/create-multiarch-manifest.sh
WAIT blocks until all nested BUILD commands complete, then continues. This lets you fan out, then fan in.
Monorepo Setup
Earthly shines in monorepos. Each service gets its own Earthfile in its subdirectory. The root Earthfile coordinates them:
.
├── Earthfile # root orchestrator
├── services/
│ ├── api/
│ │ └── Earthfile
│ ├── worker/
│ │ └── Earthfile
│ └── frontend/
│ └── Earthfile
└── shared/
└── Earthfile # shared build utilities
Root Earthfile:
VERSION 0.8
test-all:
BUILD ./services/api+test
BUILD ./services/worker+test
BUILD ./services/frontend+test
docker-all:
BUILD ./services/api+docker
BUILD ./services/worker+docker
BUILD ./services/frontend+docker
# Cross-service artifact sharing
api-client:
FROM +./services/api+generate-client
SAVE ARTIFACT ./client AS LOCAL ./services/frontend/src/api-client
Cross-directory target references use relative paths. ./services/api+test runs the +test target from services/api/Earthfile.
Gotcha: Avoid putting secrets in
EarthfileARGdefaults. Use--secretflag at invocation time andRUN --secret=id=mysecretinside the Earthfile. BuildKit secret mounts are never committed to layers and never appear indocker history.
Migrating from an Existing Build System
Migration is almost always incremental. You don’t rewrite everything on day one.
Phase 1: Wrap Existing Scripts (Low Risk)
The lowest-effort entry point. Take your existing Makefile targets and wrap them in an Earthfile without changing the scripts themselves:
VERSION 0.8
deps:
FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
test:
FROM +deps
COPY . .
RUN npm test # same command as your Makefile
build:
FROM +deps
COPY . .
RUN npm run build
SAVE ARTIFACT dist AS LOCAL ./dist
This gets you containerized, reproducible execution immediately. The build scripts themselves are untouched.
Phase 2: Add CI Integration
Replace run: npm test in your GitHub Actions with run: earthly +test. Your CI pipeline now runs exactly what developers run locally:
# .github/workflows/ci.yml
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Earthly
uses: earthly/actions-setup@v1
with:
version: latest
- name: Build and Test
run: earthly --remote-cache=${{ secrets.REGISTRY }}/cache +all
env:
EARTHLY_TOKEN: ${{ secrets.EARTHLY_TOKEN }}
Gotcha: The
earthly/actions-setupaction handles BuildKit daemon startup automatically on GitHub Actions runners. Don’t try to manually startearthly bootstrapin CI — it causes race conditions.
Phase 3: Migrate the Slow Parts
Identify your slowest CI steps. Dependency installation, compilation, and test suites are the usual suspects. Migrate those targets to use Earthly’s cache mounts and layer caching natively. Measure before and after.
Phase 4: Retire the Makefile
Once all Makefile targets have Earthfile equivalents, add thin shims to your Makefile for developer convenience:
.PHONY: test build docker
test:
earthly +test
build:
earthly +build
docker:
earthly +docker
Developers who muscle-memoried make test keep working without retraining. Earthly handles everything behind the shim.
Production-Ready Patterns
Version-pin your base images. Never use :latest or :lts in an Earthfile. Use digests or date-pinned tags (golang:1.22.3-alpine3.19). Otherwise your build inputs change silently when upstream pushes an update.
Use EARTHLY_ALLOW_PRIVILEGED only when you genuinely need it (e.g., running Docker-in-Docker for building nested images). Most builds don’t need privileged mode, and adding it widens your attack surface on CI runners.
Cache busting with ARG: When you need to force a cache miss — say, after a security advisory changes a base image — pass an ARG:
build:
ARG CACHEBUST=0
FROM golang:1.22-alpine
...
earthly --build-arg CACHEBUST=$(date +%s) +build
Test locally with --no-cache before releasing a new Earthfile version. What builds on your warm cache may fail on a cold CI runner. Run earthly --no-cache +all to simulate a cold start.
Keep targets small and composable. The temptation is to dump everything into one big build target. Resist it. Small targets compose, parallelize, and cache better. If a target takes more than five minutes when warm, it probably does too much.
Gotchas Summary
File ownership issues. Files copied into a container run as root by default. SAVE ARTIFACT can produce root-owned files on your local disk. Add RUN chown -R 1000:1000 /output before SAVE ARTIFACT or use --chown on the COPY command.
.earthignore is your friend. Earthly sends a build context to BuildKit. Without an .earthignore (same syntax as .dockerignore), you’re sending your entire repo for every target. Add node_modules, .git, dist, and any large generated directories.
Secrets in logs. If a RUN command echoes a secret — even accidentally via a shell error message — it ends up in the build log. Use RUN --secret mounts and validate that your scripts don’t echo them.
VERSION 0.8 matters. Earthly’s Earthfile syntax has evolved significantly across versions. Always declare the version at the top. Omitting it defaults to 0.5 compatibility mode and disables several useful features including WAIT/END blocks and improved ARG scoping.
When Earthly Is Not the Right Tool
Earthly is not a deployment tool. Don’t try to use it to kubectl apply manifests or SSH into servers. It builds artifacts and images. Deployment belongs to Helm, Flux, ArgoCD, or a plain shell script in CI.
It’s also overhead for genuinely simple projects. A single-language service with one test command and one Docker image doesn’t need a multi-target Earthfile. A Dockerfile and a docker build command is fine. Earthly pays off when you have multiple targets, multiple artifacts, multiple services, or the "works on my machine" problem has actually burned you.
If your team uses Bazel, Earthly is probably not a replacement — Bazel’s hermetic sandbox and remote execution are more rigorous for very large repos. Earthly is the practical middle ground between "raw Docker commands" and "full Bazel adoption."
The build reproducibility problem is older than containers and probably older than most readers’ careers. Earthly doesn’t fully solve it — no tool does — but it gets you closer than anything else that doesn’t require a dedicated build engineer to maintain. The cache model is correct, the parallelism is real, and the learning curve for anyone who’s written a Dockerfile is about an afternoon. That’s a reasonable trade for builds that stop lying to you.