Alerting

An alert is a request to interrupt a person

Alerting is usually taught as a technical question — which metric, which threshold, which window. It is better understood as an ethical and organisational one, because what an alert actually does is take a person away from their evening, their sleep or their concentration and point them at a problem. That framing produces a test that no threshold discussion can: if this fires at three in the morning, is there something a human must do immediately, and do they know what it is? If either half is no, it is not a page.

The bottom-left quadrant is the one worth staring at, because it is where most alerting systems actually go wrong. A condition looks urgent, someone is woken, and there is nothing they can do — the disk is filling on a node the platform will replace anyway, a dependency is degraded and its owners are already on it, the alert fired on a spike that resolved itself before anyone opened a laptop. The instinct is to tell the engineer to be more tolerant. The correct response is to change the alert, because every page that could not be acted on is a small deposit into the belief that pages can be ignored, and that belief is withdrawn during the real one.

Alert on symptoms, not on causes

The natural way to build alerting is to enumerate the things that can go wrong and put an alert on each: CPU above 90%, memory above 85%, disk above 80%, replica count below three, queue longer than a thousand. Each is defensible on its own and the set is a trap, for two reasons that pull in opposite directions. It fires constantly on conditions that are not hurting anyone — a batch job legitimately pinning CPU for ten minutes — and it misses entirely the failure nobody enumerated, which is the one that takes you down.

Symptom-based alerting inverts it. Instead of asking what could break, ask what the user experiences when something has broken, and alert on that: requests are failing, requests are slow, the queue is not draining, orders are not being placed. There are only a handful of such conditions per service, they cover causes nobody predicted, and each one is by construction something a user would complain about — which makes "is this worth waking someone" answer itself.

SLOs and burn rate: alerting on a number that means something

Symptom alerting needs a target, and picking one by feel produces either an alert that never fires or one that fires every Tuesday. Three terms make the target explicit, and they are worth defining precisely because they are used loosely everywhere.

• An SLI is the measurement: the proportion of requests that succeeded within 300 ms, as a number between 0 and 1. It is a ratio of good events to valid events, not an average.
• An SLO is the target for that measurement over a window: 99.9% of requests over the last 28 days. It is a promise you choose, not one the system makes.
• The error budget is what the SLO permits: 0.1% of 28 days is about 40 minutes of failure you are allowed to spend, and spending it is normal rather than a failure of discipline.

The budget is what makes alerting sane, because it converts an unanswerable question ("is 0.3% errors bad?") into an arithmetic one ("at this rate, how long until the month’s budget is gone?"). That ratio is the burn rate: burning at 1× spends the budget exactly over the window; at 14.4× a whole month’s budget is gone in two days. Alert on the burn rate rather than on the raw error rate, and the same alert covers both the sudden outage and the slow bleed that a fixed threshold always misses.

Alert fatigue is a design failure

When an on-call engineer stops reacting quickly, the usual diagnosis is that the person is not taking it seriously. The honest diagnosis is that the system taught them not to. A human who is interrupted twenty times a week by conditions that turned out to be nothing learns, correctly and unavoidably, that the next interruption is probably nothing too. You cannot train that away; you can only remove the interruptions that caused it.

The mechanics that keep the volume survivable are unglamorous and they work:

• Group related alerts into one notification. A failing dependency should produce one page, not one per service that depends on it.
• Suppress the downstream while the upstream is firing, so the cause is what reaches the person rather than forty consequences.
• Require a duration, not an instant. A condition that must hold for five minutes filters out the spikes that resolve themselves.
• Give every page a runbook link with the first three things to check. An alert without one is a puzzle handed to someone half asleep.
• Review the alerts that fired every week, and delete or downgrade the ones nobody acted on. This meeting is the whole discipline.
• Track pages per on-call shift as a number somebody owns. If it is not measured, it only ever goes up.

What has to exist before the page is sent

An alert is only the first second of a process, and the parts after it decide whether the interruption was worth anything. Four of them have to exist before the alert is switched on, not after the first incident proves they were missing.

• A named owner

  A rotation, not a mailing list. An alert routed to a group where everyone assumes someone else has it is an alert nobody has.

• A runbook

  Linked from the alert itself: what this means, the first three checks, how to mitigate, and who to escalate to. Written before the alert, in daylight.

• Escalation with a timer

  If nobody acknowledges within a few minutes it goes to the next person automatically. Phones are silenced and people miss things; the system should assume it.

• A review afterwards

  Blameless, focused on what made the failure possible and what made it hard to see. Its output is changes, including changes to the alert that found it.

One organisational point is worth stating plainly, because it decides whether any of this survives contact with a busy quarter. On-call has to be compensated and bounded — a rotation big enough that nobody carries it constantly, hours that are acknowledged as work, and permission to fix the causes of pages during normal time. Where that is absent, the alerting system degrades in a predictable direction: thresholds get loosened, alerts get muted, and the monitoring quietly stops describing the system. That outcome is not a failure of the engineers on the rotation, and it will not be fixed by adding another alert.

Anti-patterns worth naming

• An alert per resource metric on every host, producing a wall of pages during any normal load spike.
• Thresholds picked once and never revisited, so they describe a system that has since tripled in traffic.
• Alerts routed to a chat channel that everyone has muted, which is deletion with extra steps.
• A page with no runbook, handed to whoever is newest on the rotation at two in the morning.
• Monitoring that runs inside the thing it monitors, so the outage takes the alerting with it.
• A dashboard treated as monitoring, which only works while somebody happens to be looking at it.

When to use it

Related pages

• InfrastructureLoggingLogging is the record a running system leaves behind so a question about it can be answered later — and it is the part of operations most often done in a way that cannot be used. The test is not how much you record — it is whether a question asked three weeks later has an answer, and that is decided when the code is written rather than when the incident starts. This page covers why a log line should be an event rather than a sentence, what logs are for compared with metrics and traces, the fields that make an incident answerable, and the arithmetic that turns a logging habit into a bill nobody predicted.
• InfrastructureModel ServingModel serving is putting a trained model behind an interface that answers requests. Once a model is trained, serving it is a latency and cost problem before it is a machine learning problem. The inference itself is usually the smallest part of the response time, the hardware is throughput hardware being asked to do a latency job, and the thing most teams monitor — accuracy — is the one thing they cannot see live. This page covers the request path, the three shapes of serving and why the cheapest usually wins, batching and the right scaling signal, how to ship a new version without betting on it, and what to watch when the truth arrives weeks late.
• InfrastructureKubernetesKubernetes is a control loop with an API in front of it. You do not tell it to start a container; you record a desired state, and controllers keep comparing that record to what is actually running and act on the difference — forever, not once. Every object it offers and every surprising behaviour it has comes from that. This page covers the loop, the handful of objects that matter, why resource limits and health probes cause most production incidents, and when you genuinely need any of it.