Every on-call engineer knows the slow slide into alert fatigue. It starts with a noisy alert nobody wants to tune, so people learn to glance and dismiss. Then the dismissing becomes a habit, and one day a real outage scrolls past in a sea of cried wolf. Alert fatigue isn't a personal failing - it's a design failure. Alerts that get ignored were built wrong, and the fix is to treat alerting like a product with users, not like exhaust from your monitoring system.

Why alert fatigue is dangerous

The danger isn't annoyance; it's missed incidents. When most alerts are non-actionable noise, the brain rationally down-weights all of them, including the one that mattered. A system that pages you fifty times a week trains you to ignore the fifty-first page - which is the one where the database is actually on fire. Fatigue also burns people out and drives good engineers off on-call rotations, which makes everything worse.

What a healthy alert is

One rule cuts through most of the problem: every alert should be actionable and urgent. If an alert fires and the right response is "nothing right now," it shouldn't have paged - it should have been a dashboard metric or a ticket. A good alert answers three questions immediately: what's wrong, how bad is it, and what should I do. If it can't, it's noise wearing an alert's clothes.

Principles for alerts people don't ignore

• Alert on symptoms, not causes. Page on "users can't check out," not on "CPU is at 80%." High CPU may be fine; a broken checkout never is. Symptom-based alerts map to user impact and fire far less often.
• Tie severity to response. Page only for things that need a human now. Everything else is a warning that goes to a channel or a ticket, not to someone's phone at 3 AM.
• Make every alert actionable. Include a runbook link, the affected service, and recent context. An alert with no next step is a riddle, not a signal.
• Use sensible thresholds and durations. Require a condition to persist before firing, so a momentary blip doesn't page anyone. Alert on rates and trends, not single noisy data points.

A Prometheus alert that doesn't suck

The difference shows up in the rule. Instead of "fire if error count > 0," a healthy alert expresses a meaningful rate, sustained over time, tied to impact:

- alert: HighCheckoutErrorRate
  expr: |
    sum(rate(http_requests_total{route="/checkout",status=~"5.."}[5m]))
      / sum(rate(http_requests_total{route="/checkout"}[5m])) > 0.05
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "Checkout 5xx error rate above 5% for 10m"
    runbook: "https://runbooks.internal/checkout-errors"

It fires on a sustained 5% error rate on a user-facing route, carries a severity that maps to paging, and links a runbook - actionable, urgent, and self-explaining.

Common pitfalls

• Alerting on everything "just in case" - the fastest route to fatigue. Delete alerts that have never driven an action.
• No severity tiers - when everything pages, nothing is urgent.
• Static thresholds on dynamic systems - what's normal at peak isn't normal at 3 AM; use rates and, where useful, multi-window conditions.
• No ownership or review - alerts rot. Review them like code, and retire the ones that only ever cry wolf.

Design alerts like a product

The teams with healthy on-call treat their alert set as something with users and a quality bar: each alert earns its place by being actionable, every page is reviewed when it turns out to be noise, and the default answer to "should we add an alert?" is "only if someone must act on it now." Do that, and the signal comes back - when the phone buzzes, people believe it, because experience has taught them it's real.