You don’t need Cloudflare’s Enterprise tier or Fastly’s invoice to get globally fast static assets. If your site isn’t doing millions of concurrent requests, a two- or three-node DIY setup will beat a commercial CDN for latency in your key markets — and cost you less than a Netflix subscription.
The catch is that nobody explains how CDNs actually work at the small scale. They either sell you the $500/month product or give you a 200-page academic paper on Anycast routing. This article is neither. It’s a working blueprint you can deploy today with $10 Hetzner or DigitalOcean droplets.
Official repo for Nginx: https://github.com/nginx/nginx
What "small-scale CDN" actually means
Before laying any concrete, define the scope. A self-hosted CDN makes sense when:
- You have a static site, media-heavy blog, SaaS frontend, or software downloads.
- You want assets served from a location physically close to your users.
- Your traffic is predictable enough that 2-3 VPS nodes comfortably absorb it.
- You want cache control without filing a support ticket.
What it doesn’t replace: massive-scale DDoS mitigation, BGP Anycast, or true multi-continent PoP infrastructure. If you need those, pay Cloudflare. They’ve earned it.
For everything else — read on.
The Architecture in Plain Terms
A CDN is just an HTTP caching proxy that sits geographically close to the user. The request hits the edge node first. If the edge has a cached copy, it returns it immediately. If not, it fetches from your origin, caches it, and serves it. That’s 90% of the concept.
Your self-hosted version has three logical layers:
- Origin server — your actual application, WordPress, static file server, or object storage. Lives wherever you already have it.
- Edge nodes — cheap VPS instances in different regions, running Nginx or Varnish.
- DNS routing — routes users to the nearest edge node. This is the part most tutorials skip.
The diagram in your head: User → DNS → Nearest Edge Node → (cache hit?) → Origin.
Choosing Your Edge Locations
Pick nodes based on where your traffic actually comes from. Check your existing analytics. Three PoPs cover most small-scale scenarios:
- Western Europe — Hetzner Falkenstein or Nuremberg (~3.5€/month, 20TB transfer)
- US East or West — DigitalOcean NYC3 or SFO3, Vultr, Linode (~$6/month)
- Asia-Pacific — Hetzner Singapore or DigitalOcean SGP1 (~$6/month)
Total cost for 3 nodes: around $15–18/month. You get dedicated bandwidth, no cold-start latency, and you own the infrastructure.
Setting Up the Edge Node (Nginx as Caching Proxy)
Every edge node runs the same config. Nginx handles TLS termination, proxying to origin, and caching. Install it however you prefer — package manager or Docker. I’ll show the Docker Compose approach because it’s repeatable.
# docker-compose.yml — Edge Node
version: "3.9"
services:
nginx:
image: nginx:1.27-alpine
container_name: cdn-edge
restart: unless-stopped
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./sites:/etc/nginx/sites:ro
- ./cache:/var/cache/nginx # cache lives on host, survives container restarts
- ./certs:/etc/nginx/certs:ro # TLS certs from certbot on the host
environment:
- TZ=UTC
Now the Nginx config itself. The proxy_cache_path directive is doing the heavy lifting here — pay attention to the parameters.
# nginx.conf — Edge Node
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 2048;
use epoll;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
# --- Cache Zone Definition ---
# levels=1:2 creates a two-level directory hierarchy (avoids inode exhaustion)
# keys_zone=CDN:20m — 20MB of shared memory for cache keys (~160k entries)
# inactive=7d — purge entries not accessed in 7 days
# max_size=10g — cap disk usage at 10GB per node
proxy_cache_path /var/cache/nginx
levels=1:2
keys_zone=CDN:20m
inactive=7d
max_size=10g
use_temp_path=off;
# Upstream: your origin server
upstream origin {
server YOUR_ORIGIN_IP:443;
keepalive 16;
}
# Logging with cache status (HIT/MISS/EXPIRED)
log_format cdn '$remote_addr - [$time_local] "$request" '
'$status $body_bytes_sent '
'"$upstream_cache_status" "$http_referer"';
access_log /var/log/nginx/access.log cdn;
# TLS performance
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 1d;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
include /etc/nginx/sites/*.conf;
}
# sites/cdn.conf — Virtual Host for Edge
server {
listen 80;
server_name cdn-eu.yourdomain.com;
return 301 https://$host$request_uri;
}
server {
listen 443 ssl http2;
server_name cdn-eu.yourdomain.com;
ssl_certificate /etc/nginx/certs/fullchain.pem;
ssl_certificate_key /etc/nginx/certs/privkey.pem;
# --- Cache Behavior ---
proxy_cache CDN;
proxy_cache_valid 200 302 24h; # cache successful responses for 24h
proxy_cache_valid 404 1m;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503;
proxy_cache_background_update on; # refresh stale content in background, don't block user
proxy_cache_lock on; # collapse multiple cache-miss requests into one origin fetch
# Add cache status header so you can debug
add_header X-Cache-Status $upstream_cache_status always;
add_header X-Served-By $hostname always;
location / {
proxy_pass https://origin;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Strip cookies from cacheable requests — cookies bypass cache by default
proxy_ignore_headers Set-Cookie;
proxy_hide_header Set-Cookie;
proxy_cache_key "$scheme$host$request_uri";
}
# Static assets: cache longer, they rarely change
location ~* \.(jpg|jpeg|png|gif|ico|css|js|woff2|svg|webp)$ {
proxy_pass https://origin;
proxy_cache CDN;
proxy_cache_valid 200 7d;
expires 7d;
add_header Cache-Control "public, immutable";
add_header X-Cache-Status $upstream_cache_status always;
}
}
Deploy this on each VPS. The only thing that changes between nodes is the server_name (cdn-eu, cdn-us, cdn-ap).
Setting Up Cache Headers on Your Origin
Your edge won’t cache anything if your origin sends Cache-Control: no-store or doesn’t send cache headers at all. Fix this before debugging anything else.
For Nginx origin:
# On your origin server — add to your location blocks
location ~* \.(jpg|jpeg|png|gif|ico|css|js|woff2|svg|webp)$ {
expires 7d;
add_header Cache-Control "public, max-age=604800, immutable";
}
location ~* \.html$ {
expires 1h;
add_header Cache-Control "public, max-age=3600, must-revalidate";
}
For WordPress with WP Super Cache or W3 Total Cache — enable "Serve stale content" and make sure the plugin isn’t stripping Cache-Control from static files. Half the WordPress CDN setups I’ve debugged failed because a plugin was nuking headers.
DNS Routing: The Part That Makes It a CDN
This is where most tutorials wave their hands and say "use GeoDNS." Here’s what that actually means in practice.
Option A: DNS providers with GeoDNS (free tier available)
- Cloudflare — use their free plan purely for DNS. Create three A records for the same hostname (
cdn.yourdomain.com) pointing to different IPs, then use Cloudflare’s Load Balancing feature with geographic steering. Free tier doesn’t include this, but the $5/month Load Balancer add-on works. - Bunny DNS — $1/month, proper GeoDNS built in. Underrated. You define regions per record.
- Route53 — AWS, geolocation routing policies. Overkill for this use case unless you’re already in AWS.
Option B: Simple latency-based routing with multiple CNAMEs
Cheaper and simpler: give each region its own subdomain and reference them by hand in your code/CDN URL patterns.
cdn-eu.yourdomain.com → Hetzner EU node
cdn-us.yourdomain.com → DO NYC3 node
cdn-ap.yourdomain.com → Hetzner SGP node
Then in your app, serve assets with the right subdomain based on the user’s detected region. Not elegant, but it works and costs nothing extra.
Option C: Use Bunny.net as a thin CDN layer over your edges
Bunny.net charges ~$0.005/GB and pulls from your origin (which could be your Nginx edge nodes). You get their global PoP network as a free bonus tier without managing DNS routing yourself. For most small sites, this is the pragmatic answer.
Cache Invalidation
The hardest problem in distributed systems, famously. Your options from simple to complex:
1. Version your URLs. The gold standard. style.b3f2a1c.css never needs to be invalidated — you just ship a new filename. Use asset fingerprinting in your build tool (Webpack, Vite, etc.).
2. Purge via Nginx proxy_cache_purge module. Needs the Nginx commercial module or the ngx_cache_purge open-source fork. A POST request to /_purge/path removes that cache entry.
# Purge a specific URL from all edge nodes
for NODE in cdn-eu.yourdomain.com cdn-us.yourdomain.com cdn-ap.yourdomain.com; do
curl -X PURGE "https://$NODE/static/style.css" \
-H "X-Purge-Token: your-secret-token"
done
# In your Nginx config, restrict purge to trusted IPs
location ~ /purge(/.*) {
allow YOUR_MANAGEMENT_IP;
deny all;
proxy_cache_purge CDN "$scheme$host$1";
}
3. Short TTL + stale-while-revalidate. Set your HTML to a 5-minute TTL and let the edge serve stale content while revalidating in the background. Not instant, but avoids the operational complexity of a purge API.
Gotchas
Cookies kill caching. Nginx skips the cache by default if the request has cookies or the response sets them. If your CMS sets cookies even for anonymous users (WordPress does this), you’ll get 0% cache hit rate. Fix: strip cookies for anonymous requests at the edge before the cache key is computed.
# Bypass cache for logged-in WordPress users only
if ($http_cookie ~* "wordpress_logged_in") {
set $skip_cache 1;
}
proxy_cache_bypass $skip_cache;
proxy_no_cache $skip_cache;
HTTPS to origin requires valid certs. If your origin is behind a self-signed cert, proxy_pass https://origin will fail. Either use proxy_ssl_verify off (acceptable for internal network) or — better — terminate TLS at the edge and use plain HTTP on the private network between edge and origin.
Cache stampedes on cold start. When your edge node restarts with an empty cache and a burst of traffic arrives, every request misses and hammers the origin simultaneously. proxy_cache_lock on in the config above collapses those into a single upstream request. Don’t skip it.
Large files and proxy_buffering. For binary downloads (ISOs, videos), buffering the entire response in memory before sending to the client is a bad idea. Turn off buffering for these routes or set proxy_max_temp_file_size 0.
Clock skew breaks If-Modified-Since. Keep NTP sync active on all nodes (systemd-timesyncd or chrony). A 5-second clock skew causes edge nodes to serve stale content or re-fetch unnecessarily.
Split-brain cache. Each edge node has its own cache. A purge to EU doesn’t purge US. Your purge script must hit every node — automate this, don’t rely on memory.
Production-Ready Additions
Health checks and failover. Put a simple health endpoint on your origin (/healthz returning 200) and configure your DNS provider to failover to a secondary origin if the primary goes down.
Rate limiting at the edge. Your edge nodes absorb traffic before it hits your origin, which makes them the right place to rate-limit abusive IPs.
limit_req_zone $binary_remote_addr zone=per_ip:10m rate=30r/s;
location / {
limit_req zone=per_ip burst=60 nodelay;
...
}
Monitoring cache hit rate. Scrape the Nginx log for X-Cache-Status values. A hit rate below 70% usually means your TTLs are too short or cookies are bypassing cache.
# Quick cache hit rate check
awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn
# Output: number of HIT, MISS, BYPASS, EXPIRED
Let’s Encrypt wildcard cert with DNS challenge. Use certbot with the DNS-01 challenge so you can issue a single *.yourdomain.com cert and deploy it to all edge nodes without per-node HTTP challenges.
Ansible for config synchronization. Once you have two or more nodes, managing them by hand becomes painful fast. A 30-line Ansible playbook that ships your nginx.conf and restarts the container on change is worth every minute it takes to write.
# ansible/deploy-edge.yml (minimal example)
- hosts: cdn_edges
tasks:
- name: Sync nginx config
copy:
src: "{{ item }}"
dest: /opt/cdn-edge/
with_items:
- nginx.conf
- sites/
- name: Reload nginx
community.docker.docker_container:
name: cdn-edge
state: started
restart: yes
When to Upgrade to a Real CDN
Self-hosting makes sense until it doesn’t. Hand it off when:
- Your traffic peaks exceed what 2-3 small VPS nodes can absorb without queue buildup.
- You need DDoS mitigation beyond IP rate limiting.
- You’re serving users in regions where cheap VPS options don’t exist (sub-Saharan Africa, parts of South America).
- Operational overhead exceeds what you can justify — nobody should spend four hours debugging a cache stampede at 2am when $5/month on Bunny.net solves it.
The architecture you’ve built here is also a solid foundation for understanding what Cloudflare, Fastly, and Bunny actually do internally. That knowledge pays off when you’re reading their docs, billing reports, or debugging their behavior.
A two-node setup for a European blog with heavy image content can realistically cut p95 latency from 400ms (single-region origin) to under 80ms for your core audience. That’s not a marginal win — it’s the difference between users noticing and users not noticing.
Quick Start Checklist
- Spin up 2-3 VPS nodes in your target regions
- Copy the Docker Compose + Nginx config to each node, update
server_name - Issue TLS certs (certbot DNS-01 wildcard recommended)
- Fix cache headers on your origin
- Set up GeoDNS or per-region subdomains
- Verify
X-Cache-Status: HITappears after first request - Write purge script, test it
- Set up NTP, monitoring, and a basic Ansible playbook
- Watch your origin server’s CPU drop
The whole thing takes an afternoon. The maintenance overhead once it’s stable is close to zero. Good infrastructure is boring — build it once, forget about it.