Three weeks into any serious on-call rotation, you learn to hate your phone. Not because incidents are frequent — because the alerts are garbage. The CPU alert fires for 40 seconds, self-heals, fires again. The memory alert trips at 2:58 AM, you wake up, SSH in, and watch it recover on its own. You snooze the alert. You snooze it again. Then one day you snooze the real incident.
That’s alert fatigue, and it’s an engineering problem with an engineering solution.
Prometheus gives you everything you need to build alerting rules that behave. The issue is most people only use 20% of the alerting model — a threshold and a for clause — and then wonder why their on-call is miserable. This article covers the rest: signal quality, hysteresis, and burn rate alerting based on SLOs.
Why Your Current Alerts Are Lying to You
The most common Prometheus alerting rule looks like this:
- alert: HighCpuUsage
expr: rate(process_cpu_seconds_total[5m]) * 100 > 80
for: 1m
labels:
severity: warning
This fires if CPU stays above 80% for one continuous minute. Sounds reasonable. But in practice, CPU load is spiky. A batch job runs, CPU hits 85% for 90 seconds, the alert fires, the job finishes, CPU drops, the alert resolves — all before you even unlock your phone. You got woken up for a scheduled cron task.
The core problem: you’re alerting on a momentary state, not on a meaningful condition. High CPU for 90 seconds is not a problem. High CPU that doesn’t go away is a problem. And a service degrading its error budget over six hours is an existential problem that a CPU threshold will never catch.
There are two distinct failure modes to solve for:
- Transient spikes: conditions that are technically above threshold but resolve immediately
- Slow burns: conditions that never breach your threshold loudly but quietly destroy your SLO
Solving them requires different tools.
Getting Signal Right First
Before touching burn rate, fix your signal. A noisy input produces a noisy alert regardless of how sophisticated your alerting logic is.
Choose Your Rate Window Deliberately
rate() smooths over the given window. irate() uses only the last two samples. For alerting, you almost never want irate() — it’s too reactive, perfect for dashboards but terrible as an alert input.
# Bad: irate picks up individual spikes
expr: irate(http_requests_total{status=~"5.."}[1m]) > 5
# Better: rate over a meaningful window
expr: rate(http_requests_total{status=~"5.."}[5m]) > 5
The right window depends on how fast your system legitimately spikes. If deployments routinely cause 30-second error spikes that you don’t want to page on, make the window longer than 30 seconds. The window is your noise filter.
Use Ratios, Not Absolute Counts
Absolute counts break under traffic changes. 100 errors/second during peak traffic means something different than 100 errors/second at 4 AM when your traffic is 10x lower.
# Bad: absolute error count doesn't scale
expr: rate(http_requests_errors_total[5m]) > 10
# Good: error ratio is meaningful at any traffic level
expr: (
rate(http_requests_errors_total[5m])
/
rate(http_requests_total[5m])
) > 0.05
Always divide. Ratios survive load changes, deployments, and scaling events. Absolute counts don’t.
The for Clause Is Not Hysteresis
This is where most people stop. They add for: 5m and think they’ve solved the spike problem. They haven’t — not completely.
The for clause requires the condition to be continuously true for the duration before the alert fires. A 4m59s spike doesn’t page. Good. But here’s what people miss: the recovery is instantaneous. The moment your condition evaluates to false, the alert resolves immediately. No grace period.
This creates alert flap: fire, recover, fire, recover. If your metric oscillates around the threshold, you get a pager that behaves like a strobe light.
Hysteresis: Different Thresholds for Firing and Recovery
Hysteresis is a control systems concept. A thermostat with hysteresis turns the heat on at 19°C and turns it off at 21°C — not at 20°C for both. This dead zone prevents constant on/off cycling.
Prometheus doesn’t have native hysteresis support, but you can build it using recording rules and careful threshold design.
Approach 1: Recording Rules as State
Create a recording rule that tracks a smoothed version of your metric, then alert on that:
groups:
- name: recording.error_rate
interval: 30s
rules:
# Long-window smoothed error ratio
- record: job:http_error_ratio:rate5m
expr: |
rate(http_requests_errors_total[5m])
/
rate(http_requests_total[5m])
# Extra-smooth version for hysteresis
- record: job:http_error_ratio:rate15m
expr: |
rate(http_requests_errors_total[15m])
/
rate(http_requests_total[15m])
- name: alerts.error_rate
rules:
- alert: HighErrorRate
# Fires when 5m rate is bad AND 15m trend confirms it
expr: |
job:http_error_ratio:rate5m > 0.05
and
job:http_error_ratio:rate15m > 0.02
for: 2m
labels:
severity: warning
team: backend
annotations:
summary: "Error rate elevated: {{ $value | humanizePercentage }}"
description: >
{{ $labels.job }} error ratio is {{ $value | humanizePercentage }}.
This has been confirmed by both 5m and 15m windows.
The two-window and condition is your hysteresis. The alert fires when the short window shows a spike and the long window confirms it’s not a momentary blip. Recovery requires both windows to return to normal, which takes longer because of how rate() windows work.
Approach 2: Explicit Dead Band With unless
You can define separate firing and clearing thresholds directly:
- alert: HighErrorRate
# Fires at 5% error rate
expr: job:http_error_ratio:rate5m > 0.05
for: 3m
labels:
severity: warning
# This label creates a distinct alert identity
threshold: fire
- alert: HighErrorRateResolved
# This alert being absent means the first alert resolved cleanly
# Or use inhibition rules to silence the warning once dropped below 2%
expr: job:http_error_ratio:rate5m < 0.02
labels:
severity: info
Honestly, the two-window and approach from the first example is cleaner in practice. The inhibition route gets complicated fast.
Gotcha: and Requires Both Sides to Have Matching Labels
When you join two metrics with and, Prometheus does label matching. If the label sets don’t match exactly, the and silently drops the result and your alert never fires. Always verify with a PromQL query in the Prometheus UI before deploying.
# This will silently fail if label sets don't align
expr: metric_a > 10 and metric_b > 5
# Use `and on(job)` or `and ignoring(instance)` to control matching
expr: metric_a > 10 and on(job) metric_b > 5
Burn Rate Alerting: Alerting on SLO Consumption
This is where alerting gets genuinely powerful. Everything above is still symptom-based — you’re watching metrics and setting thresholds. Burn rate is different. It asks: at this rate of failure, when will we exhaust our error budget?
This comes from the Google SRE Workbook and it’s the right mental model for production systems.
The Core Concept
Suppose your SLO is 99.9% availability over 30 days. Your error budget is 0.1% of requests, which works out to roughly 43 minutes of downtime equivalent per month.
A burn rate of 1 means you’re consuming your error budget at exactly the rate it replenishes. You’ll use it all up in 30 days. Perfectly acceptable.
A burn rate of 10 means you’d exhaust the entire 30-day budget in 3 days. Page someone now.
A burn rate of 60 means the budget is gone in 12 hours. Wake someone up immediately.
The formula:
burn_rate = current_error_rate / (1 - SLO)
For a 99.9% SLO, (1 - SLO) = 0.001. If your current error rate is 0.02 (2%), your burn rate is 0.02 / 0.001 = 20. Your 30-day budget evaporates in 1.5 days.
Multi-Window Burn Rate Alerts
Single-window burn rate has the same flap problem as regular threshold alerts. The fix is multi-window: require both a short window (fast detection) and a long window (confirmation) to exceed the burn rate threshold simultaneously.
This is the production-grade pattern:
groups:
- name: slo.http_availability
rules:
# -------------------------------------------------------
# Recording rules - compute error ratios at multiple windows
# -------------------------------------------------------
- record: job:http_error_ratio:rate1h
expr: |
sum(rate(http_requests_errors_total[1h])) by (job)
/
sum(rate(http_requests_total[1h])) by (job)
- record: job:http_error_ratio:rate5m
expr: |
sum(rate(http_requests_errors_total[5m])) by (job)
/
sum(rate(http_requests_total[5m])) by (job)
- record: job:http_error_ratio:rate30m
expr: |
sum(rate(http_requests_errors_total[30m])) by (job)
/
sum(rate(http_requests_total[5m])) by (job)
- record: job:http_error_ratio:rate6h
expr: |
sum(rate(http_requests_errors_total[6h])) by (job)
/
sum(rate(http_requests_total[6h])) by (job)
# -------------------------------------------------------
# Burn rate alerts - page on SLO budget consumption rate
# SLO: 99.9% → error budget = 0.1% = 0.001
# -------------------------------------------------------
# CRITICAL: burn rate > 14.4 exhausts 1h budget in ~1h (5m window detects, 1h confirms)
# At 14.4x burn, 30-day budget gone in ~2 days
- alert: SLOBurnRateCritical
expr: |
(
job:http_error_ratio:rate5m > (14.4 * 0.001)
)
and
(
job:http_error_ratio:rate1h > (14.4 * 0.001)
)
for: 2m
labels:
severity: critical
slo: http_availability
annotations:
summary: "CRITICAL: SLO burn rate exhausts budget in < 2 days"
description: >
{{ $labels.job }} is burning error budget at {{ $value | humanizePercentage }} error rate.
At this rate the 30-day budget is consumed in ~{{ div 720 14.4 | printf "%.0f" }} hours.
Investigate immediately.
# WARNING: burn rate > 6 exhausts budget in ~5 days (30m window detects, 6h confirms)
- alert: SLOBurnRateWarning
expr: |
(
job:http_error_ratio:rate30m > (6 * 0.001)
)
and
(
job:http_error_ratio:rate6h > (6 * 0.001)
)
for: 15m
labels:
severity: warning
slo: http_availability
annotations:
summary: "WARNING: elevated SLO burn rate"
description: >
{{ $labels.job }} error rate {{ $value | humanizePercentage }} sustained.
Burn rate ~6x: 30-day budget consumed in ~5 days if unchanged.
The two thresholds — 14.4x and 6x — come from the Google SRE Workbook’s recommended alert thresholds for a 30-day window SLO. They’re not magic numbers; they’re derived from "what burn rate detected over what time window gives you enough time to react while minimizing false positives."
Gotcha: Burn Rate Alerts Miss Slow Degradation
A burn rate of 1.5x is bad. It means you’re consuming your budget 50% faster than it refills. But a 1.5x burn rate over 30 days is hard to detect without looking at a month of data. The multi-window approach above won’t catch it because neither threshold is breached.
For slow burns, add a third tier that looks at very long windows and fires with very low urgency — maybe just a Slack message, not a page:
- alert: SLOBudgetDepleting
# Budget will be exhausted in ~20 days at this rate
expr: job:http_error_ratio:rate6h > (1.5 * 0.001)
for: 1h
labels:
severity: info
route: slack-only # Don't page, just notify
annotations:
summary: "Slow SLO degradation detected — check within business hours"
Putting It Together: A Complete Alerting Stack
A real alerting configuration for a backend HTTP service:
groups:
# -------------------------------------------------------
# Recording rules — always evaluate before alerts
# -------------------------------------------------------
- name: http.recording
interval: 30s
rules:
- record: job:http_error_ratio:rate5m
expr: |
sum by (job) (rate(http_requests_errors_total[5m]))
/
sum by (job) (rate(http_requests_total[5m]))
- record: job:http_error_ratio:rate30m
expr: |
sum by (job) (rate(http_requests_errors_total[30m]))
/
sum by (job) (rate(http_requests_total[30m]))
- record: job:http_error_ratio:rate1h
expr: |
sum by (job) (rate(http_requests_errors_total[1h]))
/
sum by (job) (rate(http_requests_total[1h]))
- record: job:http_error_ratio:rate6h
expr: |
sum by (job) (rate(http_requests_errors_total[6h]))
/
sum by (job) (rate(http_requests_total[6h]))
- record: job:http_latency_p99:rate5m
expr: |
histogram_quantile(0.99,
sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))
)
# -------------------------------------------------------
# Availability alerts (SLO: 99.9%)
# -------------------------------------------------------
- name: http.slo.availability
rules:
- alert: SLOBurnRateCritical
expr: |
job:http_error_ratio:rate5m > (14.4 * 0.001)
and on(job)
job:http_error_ratio:rate1h > (14.4 * 0.001)
for: 2m
labels:
severity: critical
annotations:
summary: "{{ $labels.job }}: critical SLO burn — page now"
- alert: SLOBurnRateWarning
expr: |
job:http_error_ratio:rate30m > (6 * 0.001)
and on(job)
job:http_error_ratio:rate6h > (6 * 0.001)
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: elevated burn rate, review within the hour"
# -------------------------------------------------------
# Latency alerts (with hysteresis via dual-window)
# -------------------------------------------------------
- name: http.latency
rules:
- alert: HighLatencyP99
# Both windows must show elevated latency — suppresses transient spikes
expr: |
job:http_latency_p99:rate5m > 1.0
and on(job)
# Approximate longer window inline when no recording rule exists
histogram_quantile(0.99,
sum by (job, le) (rate(http_request_duration_seconds_bucket[30m]))
) > 0.5
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: p99 latency {{ $value | humanizeDuration }}"
# -------------------------------------------------------
# Availability — catch complete outages fast
# -------------------------------------------------------
- name: http.availability
rules:
- alert: ServiceDown
# absent() fires if the metric disappears entirely
expr: absent(up{job="my-service"} == 1)
for: 1m
labels:
severity: critical
annotations:
summary: "my-service is completely down or not scraped"
- alert: InstancesDown
# Some instances down but service still running
expr: |
(
count by (job) (up{job="my-service"} == 0)
/
count by (job) (up{job="my-service"})
) > 0.5
for: 3m
labels:
severity: warning
annotations:
summary: "{{ $labels.job }}: >50% of instances are down"
Gotchas and Production Lessons
Recording rules must be in a separate group evaluated before alert groups. Prometheus evaluates rule groups in the order they’re defined, but groups evaluate in parallel by default in newer versions. To guarantee recording rules exist before alerts reference them, put them in a separate group and rely on the evaluation interval keeping them fresh. Or use depends_on if your Prometheus version supports it.
absent() lies if your scrape fails. If Prometheus can’t reach your target, up goes to 0, but absent(your_metric) also fires because the metric literally isn’t there. Use up == 0 for scrape failures and absent() only for truly missing time series (like a service that was deleted from service discovery).
Burn rate math breaks at low traffic. If you’re handling 10 requests per hour, one error gives you a 10% error rate and a 100x burn rate. Add a minimum traffic guard:
expr: |
(
job:http_error_ratio:rate5m > (14.4 * 0.001)
and on(job)
job:http_error_ratio:rate1h > (14.4 * 0.001)
)
and on(job)
# Only alert if we have meaningful traffic
sum by (job) (rate(http_requests_total[5m])) > 1
Test your alerting rules before they’re needed. Use promtool check rules rules.yml to validate syntax. For logic validation, use Prometheus rule unit testing — it lets you mock time series and assert which alerts fire. This is non-optional for critical alerts.
Don’t alert on everything. Every alert that fires and gets ignored trains your team to ignore alerts. If you’re not going to act on it at 3 AM, it shouldn’t page. Separate your routing in Alertmanager: critical fires PagerDuty, warning fires Slack, info is just a dashboard annotation.
The Practical Checklist
Before you deploy any new alerting rule, ask:
- Is this alerting on a ratio or an absolute count? (It should be a ratio)
- Does the
forduration cover your longest acceptable transient spike? - Would a load spike or deployment naturally trigger this? If yes, add a second confirming window
- Is this a symptom that directly impacts users, or an internal metric that might lead to user impact? (Prefer the former)
- Is there a burn rate equivalent that would catch this more reliably?
- What’s the on-call action when this fires? If the answer is "check if it’s still happening and maybe restart something," the alert is probably premature
Alert fatigue kills your on-call rotation and, eventually, your ability to respond to real incidents. Every false positive you eliminate is an incident you can actually react to. Start with recording rules, build the two-window patterns, add burn rate for your critical services, and trim ruthlessly. Sleep is a feature.