You’ve been there. You deploy at 3 PM on a Friday, everything looks green in staging, then three minutes after going live your error rate spikes and you’re rolling back while people are Slack-pinging you. The problem isn’t that you deployed — it’s that you deployed to 100% of users instantly.
Canary deployments solve exactly this. You shift a small slice of real traffic to the new version, watch the metrics, and only proceed when you’re confident it won’t blow up. If it does blow up, only a fraction of your users experience it, and you roll back with one command.
This article covers three distinct canary strategies — percentage-based, header-based, and geo-based — with real, working configurations. We’ll go from a simple Nginx setup to Kubernetes with NGINX Ingress, then up to Argo Rollouts and Istio for serious production workloads.
What Makes a Canary Different from Blue-Green
Blue-green swaps traffic all at once between two environments. It’s clean and easy to understand, but you still expose every user to the new version the moment you flip the switch.
A canary is a gradual rollout. You run v1 and v2 simultaneously, and the routing layer decides which users hit which version. You start at 5%, watch your dashboards, go to 20%, watch again, and eventually reach 100%.
The "canary" name comes from coal miners who brought canaries into mines as an early warning system for toxic gas. Your new release is the canary. Your monitoring is the miner watching whether it survives.
Strategy 1: Percentage-Based Splitting
The most straightforward approach. Route X% of requests to the new version, the rest to stable.
Nginx: The Simple Way
If you’re not on Kubernetes yet, Nginx’s split_clients module handles this natively.
# /etc/nginx/conf.d/canary.conf
# split_clients uses the client IP (or any variable) to deterministically
# assign the same user to the same upstream across requests.
split_clients "${remote_addr}AAA" $upstream_pool {
5% canary; # 5% of users hit the new version
* stable; # everyone else stays on stable
}
upstream stable {
server 127.0.0.1:8080;
}
upstream canary {
server 127.0.0.1:8081;
}
server {
listen 80;
location / {
proxy_pass http://$upstream_pool;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
# Tell the app which version it's running as — useful for logging
proxy_set_header X-Deployment-Stage $upstream_pool;
}
}
The AAA salt in split_clients matters. It’s mixed with the variable to compute the hash. If you use just $remote_addr, every other configuration on this Nginx instance using split_clients will split at the same boundary. Change the salt, change the distribution.
Gotcha: split_clients uses a hash of the variable, not random assignment per request. This means a user always lands on the same upstream — which is what you want for session consistency — but it also means if you change the percentage, users get redistributed. Don’t change percentages mid-session on stateful apps without thinking about it.
Kubernetes: NGINX Ingress Controller
The NGINX Ingress Controller has first-class canary support via annotations. You define two Ingress resources for the same host: one for stable, one for canary.
# stable-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-app-stable
namespace: production
spec:
ingressClassName: nginx
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-app-stable
port:
number: 80
# canary-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-app-canary
namespace: production
annotations:
nginx.ingress.kubernetes.io/canary: "true"
# Route 10% of traffic to this ingress
nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
ingressClassName: nginx
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-app-canary
port:
number: 80
Apply both, and the controller handles the split transparently. To increase the canary percentage:
kubectl annotate ingress my-app-canary -n production \
nginx.ingress.kubernetes.io/canary-weight=25 \
--overwrite
Gotcha: canary-weight is approximate. The NGINX Ingress Controller implements this via Lua and random number generation per request, not deterministic hashing. Users may switch upstreams between requests. If your app has server-side sessions stored in memory, this will cause authentication issues. Use sticky sessions or externalize session storage (Redis, a database) before running canaries.
Strategy 2: Header-Based Routing
Percentage-based canaries are blind — users don’t know they’re on the canary, and you can’t opt specific users in or out. Header-based routing fixes this.
You manually set a header (via browser extension, internal tooling, or a feature flag service) and the routing layer sends all requests with that header to the canary. This is how you do internal testing on production: your team hits the new version, external users don’t.
Nginx: Header-Based Routing
# /etc/nginx/conf.d/canary-header.conf
map $http_x_canary $backend {
"enabled" canary;
default stable;
}
upstream stable {
server 127.0.0.1:8080;
}
upstream canary {
server 127.0.0.1:8081;
}
server {
listen 80;
location / {
proxy_pass http://$backend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
Send requests with X-Canary: enabled and they hit the new version. Anyone without the header goes to stable. Simple, predictable, auditable.
Kubernetes: NGINX Ingress Header Canary
# canary-header-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-app-canary-header
namespace: production
annotations:
nginx.ingress.kubernetes.io/canary: "true"
# Route requests where X-Canary header equals "enabled"
nginx.ingress.kubernetes.io/canary-by-header: "X-Canary"
nginx.ingress.kubernetes.io/canary-by-header-value: "enabled"
spec:
ingressClassName: nginx
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-app-canary
port:
number: 80
You can also combine header routing with percentage routing. Requests with the header always go to canary; requests without it get the percentage split treatment. This is useful when you want your QA team to always test the new version while general traffic gets a 5% slice.
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "X-Canary"
nginx.ingress.kubernetes.io/canary-by-header-value: "enabled"
nginx.ingress.kubernetes.io/canary-weight: "5"
Gotcha: Header-based routing is trivially bypassable. Don’t use it as a security boundary. It’s a convenience tool for developers and testers, not access control. Anyone can add that header to their requests. If you’re using it to gate a genuinely sensitive feature, pair it with authentication middleware.
Strategy 3: Geo-Based Routing
Roll out to users in one country or region before going global. This is especially valuable for localization changes (currency formats, date formats, legal notices) or when you want a smaller blast radius during off-peak hours in a specific timezone.
Nginx with GeoIP2
First, install the GeoIP2 module and download the MaxMind GeoLite2 database:
# Install the module (Debian/Ubuntu)
apt-get install libnginx-mod-http-geoip2
# Download the GeoLite2 database (requires free MaxMind account)
# After registering, download via:
# https://www.maxmind.com/en/geolite2/signup
# Place it at:
mkdir -p /usr/share/GeoIP
# mmdbinspect can verify the file after download
# /etc/nginx/conf.d/canary-geo.conf
# Load GeoIP2 module at the http block level
geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
auto_reload 5m;
$geoip2_metadata_country_build metadata build_epoch;
$geoip2_data_country_code country iso_code;
}
map $geoip2_data_country_code $canary_target {
# Route users from Germany and Austria to canary for EU rollout
DE canary;
AT canary;
# Everyone else gets stable
default stable;
}
upstream stable {
server 127.0.0.1:8080;
}
upstream canary {
server 127.0.0.1:8081;
}
server {
listen 80;
location / {
proxy_pass http://$canary_target;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
# Log which version served the request — critical for debugging
add_header X-Served-By $canary_target always;
}
}
Gotcha: GeoIP databases are probabilistic, not authoritative. VPNs, Tor exit nodes, and corporate proxies will all produce wrong country codes. Don’t rely on geo-routing for compliance or GDPR enforcement. Use it for controlled rollouts, not legal segmentation.
Kubernetes: Geo-Based with NGINX Ingress
The NGINX Ingress Controller supports canary routing by cookie value. This pairs nicely with a geo-detection middleware: detect the user’s country in a small auth/redirect service, set a cookie, and then let the ingress route on that cookie.
# canary-cookie-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-app-canary-cookie
namespace: production
annotations:
nginx.ingress.kubernetes.io/canary: "true"
# Route requests where the "canary-region" cookie is set to "eu"
nginx.ingress.kubernetes.io/canary-by-cookie: "canary-region"
spec:
ingressClassName: nginx
rules:
- host: app.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-app-canary
port:
number: 80
When canary-region=always cookie is present, traffic goes to canary. When canary-region=never, it always goes to stable. Any other value falls through to the weight-based rules.
Going Serious: Argo Rollouts
For teams running Kubernetes at any real scale, Argo Rollouts is worth the investment. It’s a Kubernetes controller that manages progressive delivery as a first-class resource type, replacing standard Deployments with a Rollout CRD.
# rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: my-app
namespace: production
spec:
replicas: 10
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: my-app:v2.0.0
ports:
- containerPort: 8080
strategy:
canary:
# Which ingress to update with canary annotations automatically
canaryService: my-app-canary-svc
stableService: my-app-stable-svc
trafficRouting:
nginx:
stableIngress: my-app-stable
annotationPrefix: nginx.ingress.kubernetes.io
steps:
# Step 1: Send 5% to canary, pause for manual promotion
- setWeight: 5
- pause: {}
# Step 2: After manual approval, go to 20% for 5 minutes
- setWeight: 20
- pause:
duration: 5m
# Step 3: 50% for another 5 minutes
- setWeight: 50
- pause:
duration: 5m
# Step 4: Full rollout — no pause, auto-completes
- setWeight: 100
# Automatically roll back if error rate exceeds threshold
analysis:
templates:
- templateName: success-rate
startingStep: 2
args:
- name: service-name
value: my-app-canary-svc
Promote through steps manually:
# Approve and move to next step
kubectl argo rollouts promote my-app -n production
# Check current status
kubectl argo rollouts status my-app -n production
# Abort and roll back immediately
kubectl argo rollouts abort my-app -n production
The Analysis template lets you tie rollout progression to actual metrics — Prometheus, Datadog, CloudWatch. If your error rate spikes during the canary phase, the rollout aborts automatically without a human needing to notice.
# analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: success-rate
namespace: production
spec:
args:
- name: service-name
metrics:
- name: success-rate
# Check every 30 seconds
interval: 30s
# Fail the analysis if this metric fails 3 times
failureLimit: 3
provider:
prometheus:
address: http://prometheus.monitoring.svc.cluster.local:9090
query: |
sum(rate(http_requests_total{
service="{{args.service-name}}",
status!~"5.."
}[2m])) /
sum(rate(http_requests_total{
service="{{args.service-name}}"
}[2m]))
# Require at least 95% success rate
successCondition: result[0] >= 0.95
failureCondition: result[0] < 0.90
Istio: When You Need the Full Picture
If you’re already running a service mesh, Istio’s VirtualService and DestinationRule give you traffic splitting at the sidecar level — no ingress annotations required, works for east-west (service-to-service) traffic as well as north-south.
# destination-rule.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: my-app
namespace: production
spec:
host: my-app
subsets:
- name: stable
labels:
version: v1
- name: canary
labels:
version: v2
# virtual-service.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: my-app
namespace: production
spec:
hosts:
- my-app
http:
# Header-based routing takes priority
- match:
- headers:
x-canary:
exact: "enabled"
route:
- destination:
host: my-app
subset: canary
# Geo-based: if a header set upstream identifies EU users
- match:
- headers:
x-user-region:
prefix: "eu-"
route:
- destination:
host: my-app
subset: canary
# Default: 10% canary, 90% stable
- route:
- destination:
host: my-app
subset: stable
weight: 90
- destination:
host: my-app
subset: canary
weight: 10
The match rules are evaluated top-to-bottom, first match wins. This lets you layer strategies: explicit header opt-in first, then geo-targeting, then probabilistic splitting as the fallback. You get the best of all three approaches in a single routing rule.
Production Checklist
Before you run any canary in production:
Observability first. You need separate metrics per version. Tag every metric, log line, and trace with the version and deployment stage. If you can’t tell v1 from v2 in your dashboard, you’re flying blind.
Session stickiness decision. Decide upfront: do you need users to stay on the same version for the duration of their session? If yes, use cookie-based stickiness. If no, document why and accept that some flows may break across a version boundary during the rollout.
Define rollback criteria before deploying. Write down: "If error rate exceeds X% or p99 latency exceeds Yms for Z minutes, we roll back." Do this before the pressure is on and you’re watching the dashboard wondering if that spike is normal.
Database migrations. Canary deployments and destructive database migrations are a brutal combination. Never run a migration that breaks v1 while v1 is still serving traffic. Expand-contract pattern: add new columns/tables first, then deploy the new code, then remove old schema later.
Test the rollback. Run a canary rollout in staging and actually abort it mid-way. The worst time to discover your rollback procedure is broken is during a production incident.
Picking the Right Strategy
Use percentage-based when you want a gradual, automated rollout with no special client behavior required. Works for most cases.
Use header-based when you want internal teams to test on production before any external users see the change. Combine it with percentage-based once internal testing passes.
Use geo-based when you’re doing a regional launch — new market, localized feature, regulatory compliance timing. It’s also a useful blast-radius limiter: rolling out to users in UTC+9 at 2 PM their time means your team in UTC+0 is awake and watching during off-peak hours.
All three strategies can coexist. The order of precedence is up to you and your routing layer, but the pattern that works well in practice is: header opt-in → geo segment → percentage fallback.
Start with whatever your current stack already supports. The best canary strategy is the one you’ll actually use.