Every team eventually hits the same wall: staging is broken again. Someone pushed half-finished auth changes, a QA session is blocked, and the release is tomorrow. You’ve been here. Everyone has.
The problem isn’t staging itself — it’s the model. A single shared staging environment is a shared toilet. Works fine when one person uses it. Falls apart the moment two squads start pushing simultaneously.
There are three dominant models for pre-production: shared, ephemeral, and branch-based. Each solves a different problem and creates its own set of headaches. This article breaks down all three with real configs, tells you exactly when to use what, and — more importantly — tells you what will break when you try.
Why Pre-Prod Matters More Than You Think
Pre-production is where confidence is built or destroyed before code touches users. The goal isn’t "test everything" — it’s catch regressions early, validate integrations, and give non-engineers (QA, product, stakeholders) something real to poke at.
Where teams go wrong is treating pre-prod as a single responsibility: one environment, one config, one purpose. That never scales beyond a team of three.
This is the default for most teams that haven’t thought hard about environments. There’s one staging server, pointing to one database, behind one domain. Everyone deploys to it.
When it works: Small teams, single product, low deploy frequency. If you’re shipping once a week, a shared staging is perfectly fine.
When it falls apart: The moment two features are in flight simultaneously. One branch resets the DB schema, another is mid-QA, and now you’re playing "who broke staging" Slack archaeology.
A shared staging environment with Docker Compose looks like this:
# docker-compose.staging.yml
version: "3.9"
services:
app:
image: myapp:${APP_VERSION:-latest}
environment:
- DATABASE_URL=postgres://app:secret@db/staging
- ENVIRONMENT=staging
- REDIS_URL=redis://cache:6379
ports:
- "8080:8080"
depends_on:
- db
- cache
restart: unless-stopped
db:
image: postgres:16-alpine
environment:
POSTGRES_DB: staging
POSTGRES_USER: app
POSTGRES_PASSWORD: secret
volumes:
- pg_data:/var/lib/postgresql/data
cache:
image: redis:7-alpine
volumes:
- redis_data:/data
volumes:
pg_data:
redis_data:
Simple, fast to set up, cheap to run. The ${APP_VERSION} env var is the only moving piece — your CI sets it on deploy.
The silent killer of shared staging is persistent state. When dev A runs a migration and dev B’s branch hasn’t applied it yet, you get a mismatched schema. The app may not crash immediately — it’ll just behave wrongly in subtle ways.
Fix: Never let shared staging run long-lived state without a migration gate. Automate the migration step into your deploy pipeline, and treat the staging DB as disposable. Script a nightly reset from a sanitized production snapshot if you need realistic data.
Ephemeral Environments
Ephemeral means: spun up on demand, destroyed when done. No persistent environment, no shared state, no "who broke staging" problem. The environment lives exactly as long as it needs to.
This is the model that scales. Every PR gets its own environment. QA reviews a feature in isolation. When the PR closes, the environment disappears.
The tradeoff: cost and complexity. Running dozens of isolated environments simultaneously requires orchestration, good resource cleanup, and either cloud budget or a beefy self-hosted cluster.
Kubernetes Namespace per PR
The cleanest implementation of ephemeral environments in Kubernetes uses a namespace per pull request. Each namespace is self-contained: its own DB, its own app pods, its own ingress.
Here’s a GitHub Actions workflow that creates and destroys namespaces:
# .github/workflows/ephemeral-env.yml
name: Ephemeral Environment
on:
pull_request:
types: [opened, synchronize, reopened, closed]
env:
# sanitize the branch name: replace slashes and special chars with dashes
ENV_NAME: pr-${{ github.event.pull_request.number }}
jobs:
deploy:
if: github.event.action != 'closed'
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up kubectl
uses: azure/setup-kubectl@v3
- name: Configure kubeconfig
run: echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > ~/.kube/config
- name: Create namespace
run: |
kubectl create namespace ${{ env.ENV_NAME }} --dry-run=client -o yaml \
| kubectl apply -f -
- name: Deploy app
run: |
helm upgrade --install myapp ./helm/myapp \
--namespace ${{ env.ENV_NAME }} \
--set image.tag=${{ github.sha }} \
--set ingress.host="${{ env.ENV_NAME }}.preview.example.com" \
--set db.name=${{ env.ENV_NAME }} \
--wait --timeout 5m
- name: Post preview URL to PR
uses: actions/github-script@v7
with:
script: |
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `Preview environment ready: https://${{ env.ENV_NAME }}.preview.example.com`
})
cleanup:
if: github.event.action == 'closed'
runs-on: ubuntu-latest
steps:
- name: Configure kubeconfig
run: echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > ~/.kube/config
- name: Destroy namespace
run: kubectl delete namespace ${{ env.ENV_NAME }} --ignore-not-found=true
The helm upgrade --install pattern is idempotent — re-running on each push updates the existing environment. No double-creating.
Helm Values for an Ephemeral Environment
# helm/myapp/values-ephemeral.yaml
replicaCount: 1 # single replica — this is preview, not prod
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 512Mi
# Ephemeral DB — cheap PostgreSQL sidecar, not an RDS
postgresql:
enabled: true
auth:
database: myapp
username: myapp
password: insecure_preview_only # not a real secret — env is throwaway
ingress:
enabled: true
annotations:
cert-manager.io/cluster-issuer: letsencrypt-staging # don't burn prod cert quota
tls:
- hosts:
- "{{ .Values.ingress.host }}"
secretName: "{{ .Release.Name }}-tls"
Gotcha: Wildcard DNS and TLS
You need *.preview.example.com resolving to your ingress controller and a wildcard TLS cert (or cert-manager issuing per-subdomain certs on demand). Wildcard certs are easy with Let’s Encrypt DNS-01 challenge. Cert-manager handles this well. Where teams stumble: using HTTP-01 challenge for ephemeral subdomains — it requires port 80 to respond for each subdomain before the cert is issued. DNS-01 is the correct choice here.
Gotcha: Resource Leaks
Ephemeral means you commit to cleaning up. If your PR closes but the workflow fails mid-cleanup, you leak a namespace, a running DB, and compute. Add a scheduled job that sweeps for stale namespaces:
#!/bin/bash
# cleanup-stale-preview-envs.sh
# Run as a CronJob in Kubernetes or a cron on your CI server.
# Delete any preview namespace not updated in 48+ hours
kubectl get namespaces -o json \
| jq -r '.items[] | select(.metadata.name | startswith("pr-"))
| select(
(now - (.metadata.creationTimestamp | fromdateiso8601)) > 172800
)
| .metadata.name' \
| xargs -I{} kubectl delete namespace {} --ignore-not-found=true
Branch-Based Environments
Branch-based sits between shared and fully ephemeral. Instead of one staging and one-per-PR, you get one environment per long-lived branch: main → staging, release/v2 → UAT, feature/big-rewrite → its own environment until it merges.
This model fits teams working on multiple parallel release tracks — think mobile apps with versioned API backends, or SaaS products with enterprise customers on different release channels.
Docker Compose + Traefik for Branch Environments
On a self-hosted server, Traefik is your best friend for branch-based routing. It picks up services dynamically via labels — no config reload required.
# docker-compose.branch.yml
# Deploy with: BRANCH_NAME=feature-payments docker compose -f docker-compose.branch.yml up -d
version: "3.9"
services:
app:
image: myapp:${GIT_SHA:-latest}
labels:
- "traefik.enable=true"
# Route by subdomain: feature-payments.preview.example.com
- "traefik.http.routers.${BRANCH_NAME}.rule=Host(`${BRANCH_NAME}.preview.example.com`)"
- "traefik.http.routers.${BRANCH_NAME}.entrypoints=websecure"
- "traefik.http.routers.${BRANCH_NAME}.tls=true"
- "traefik.http.routers.${BRANCH_NAME}.tls.certresolver=letsencrypt"
- "traefik.http.services.${BRANCH_NAME}.loadbalancer.server.port=8080"
environment:
- DATABASE_URL=postgres://app:secret@db-${BRANCH_NAME}/app
- BRANCH=${BRANCH_NAME}
networks:
- traefik_public
- internal
db:
image: postgres:16-alpine
container_name: db-${BRANCH_NAME}
environment:
POSTGRES_DB: app
POSTGRES_USER: app
POSTGRES_PASSWORD: secret
volumes:
# Branch-scoped volume — survives deploys, dies when explicitly removed
- pgdata_${BRANCH_NAME}:/var/lib/postgresql/data
networks:
- internal
volumes:
pgdata_${BRANCH_NAME}:
networks:
traefik_public:
external: true # shared Traefik network
internal:
driver: bridge
The BRANCH_NAME variable drives everything: container names, volume names, router names. This means you can run this compose file multiple times with different BRANCH_NAME values and they won’t collide.
Traefik Static Config
# /etc/traefik/traefik.yml
entryPoints:
web:
address: ":80"
http:
redirections:
entryPoint:
to: websecure
scheme: https
websecure:
address: ":443"
certificatesResolvers:
letsencrypt:
acme:
email: [email protected]
storage: /etc/traefik/acme.json
dnsChallenge:
provider: cloudflare # adjust to your DNS provider
resolvers:
- "1.1.1.1:53"
providers:
docker:
exposedByDefault: false
network: traefik_public
Gotcha: Env Name Sanitization
Branch names like feature/user-auth contain slashes. Docker volumes, container names, and DNS labels cannot. Sanitize early and consistently — everywhere. In CI, do this once and export:
# In your CI script, before any docker compose calls:
BRANCH_NAME=$(echo "${GITHUB_HEAD_REF:-$(git rev-parse --abbrev-ref HEAD)}" \
| tr '/' '-' \
| tr '[:upper:]' '[:lower:]' \
| sed 's/[^a-z0-9-]/-/g' \
| cut -c1-40)
export BRANCH_NAME
If you skip this, you’ll get docker: invalid character errors at 2am before a demo.
Choosing the Right Model
Here’s the honest comparison:
| Shared | Branch-Based | Ephemeral | |
|---|---|---|---|
| Cost | Lowest | Medium | Highest |
| Setup complexity | Low | Medium | High |
| Isolation | None | Per branch | Per PR |
| Feedback loop | Slow | Medium | Fast |
| Best for | Solo/tiny teams | Parallel release tracks | Product teams, QA automation |
| Main failure mode | State collision | Branch sprawl | Orphaned resources |
There’s no universally correct answer. The mistake I see most often is teams jumping to full ephemeral before they’ve solved the basics — and then spending more time maintaining preview infrastructure than shipping features.
My rule of thumb:
- Under 5 engineers: shared staging is fine, add discipline instead of tooling.
- 5–20 engineers with a QA function: branch-based is the sweet spot.
- 20+ engineers or heavy PR review workflow: ephemeral pays for itself.
Hybrid: The Model Nobody Talks About
Real production-grade setups often combine approaches. A common pattern:
- Ephemeral per PR for unit-testable, isolated features.
- Long-lived
stagingthat mirrors production for integration testing and stakeholder demos — updated by merges tomain. uatenvironment locked to release candidates, updated manually by the release manager.
You don’t have to pick one. Layering works fine as long as the purpose of each environment is clearly documented and enforced by your pipeline.
Database Strategy Across All Three Models
The hardest part of pre-prod environments isn’t the app — it’s state. Three patterns worth knowing:
1. Seed scripts only (ephemeral DBs)
For ephemeral environments, never restore a prod dump. Use seed scripts that generate realistic synthetic data. Faster, GDPR-compliant, reproducible.
-- seeds/base.sql
INSERT INTO users (email, role) VALUES
('[email protected]', 'admin'),
('[email protected]', 'user');
2. Production snapshot with PII scrubbing (shared/branch envs)
For shared staging or branch environments that need realistic data volume:
#!/bin/bash
# scrub-and-restore.sh
pg_dump $PROD_DATABASE_URL \
| sed 's/[a-zA-Z0-9._%+-]*@[a-zA-Z0-9.-]*\.[a-zA-Z]*/[email protected]/g' \
| psql $STAGING_DATABASE_URL
Crude but effective for most cases. For serious PII scrubbing, look at pgAnonymizer or faker-based migration scripts.
3. Schema-only + fixtures (fast CI)
When you need speed above all: restore schema only, insert just enough fixture data for tests to pass. 10 seconds instead of 10 minutes.
Production-Ready Checklist
Before shipping any pre-prod environment strategy to your team, verify these:
- Cleanup is automated. Manual cleanup will never happen consistently.
- Costs are visible. Tag all preview resources. Set alerts. Ephemeral environments that run overnight or over weekends are silent budget killers.
- Secrets are isolated. Never share production secrets with any pre-prod environment. Use separate credentials, even if the values are weaker.
- DNS propagation is handled. If you’re spinning up new subdomains per branch, account for DNS TTL. Either use a low TTL on the wildcard record or use IP-based routing within your ingress.
- The pipeline fails clearly. If environment creation fails, the PR should be blocked with a useful error, not silently succeed.
- Developers can self-service. If someone needs to file a Jira ticket to get a preview environment, the system has already failed.
Final Take
The pre-prod environment question is fundamentally a question about what your team values: simplicity, speed, or isolation. Shared is simple. Ephemeral is fast feedback. Branch-based is isolation without full complexity.
Most teams don’t fail because they chose the wrong model. They fail because they chose a model and never maintained it — let the staging DB grow stale, stopped cleaning up orphaned environments, let the pipeline drift from reality.
Pick the model that matches your team size and workflow today. Build the cleanup and maintenance in from day one. And when the model stops fitting — which it will — migrate deliberately, not in a panic at 11pm the night before a release.