Multi-region uptime checks: how 2-of-3 consensus kills false positives

A false positive is not a wrong email. It is the sequence that follows it: someone wakes up, someone opens a dashboard, someone checks logs, someone decides the alert was noise — and the next real alert gets a slower response because of it. Alert fatigue is not a personality flaw of on-call engineers; it is a rational response to a system that cries wolf. Every false positive you remove makes the alerts you keep more credible.

The second cost is the one nobody logs: the time spent investigating an incident that never happened. When a monitor says "down", the first question is always "is this real?" — and that question gets asked by a human at 3am. Multi-region consensus exists to make the answer boring: the alert fired because two independent regions agreed, so the investigation starts from "this is real", not from "is this real?".

There is a third cost that compounds the first two: the public status page. When a false positive becomes a public incident, the people watching your status page — your customers — learn that the page can be wrong. Regaining that trust is harder than fixing any alert.

PingRoot runs every check from three regions — us-east, eu-west and asia-pacific — executed by Cloudflare Workers. Each region runs as its own Worker, so the three checks are independent executions, not three views of one process. Each region checks the target independently and casts a vote. A monitor is only marked down when at least two of the three regions agree it is down. One region alone can never mark a monitor down.

Below that threshold, nothing happens. A single regional failure is treated as a regional observation, not as an incident. If one region sees a failure while the other two see success, the monitor stays up: the observation is recorded, but it does not change state. The consensus is the product, not the individual check.

Here is the problem a simple majority vote does not solve: the three regions observe the same target at the same instant. If that target is slow — not down, just slow — all three regions can time out together. Each check has a 10-second timeout, and a response that arrives one second late looks identical to no response at all, from all three regions at once.

A majority vote would mark that monitor down. The next check succeeds, the monitor comes back up. The check after that times out again. The monitor oscillates between up and down indefinitely, and every oscillation pages someone.

So PingRoot adds a second rule, per region: a region must observe two consecutive failures before it is allowed to vote "down". One slow response is not enough for a region to change its vote. Two in a row is.

With the two-strike rule, the same target behaves differently. A check times out — strike one. The next check succeeds — the strike resets, and the region votes up. A target that is slow for one check and fast for the next never accumulates two consecutive failures in any region, so no region ever votes down, and the monitor never flips. The hysteresis is what breaks the oscillation.

The asymmetry is deliberate: confirming a down requires corroboration, confirming a recovery does not. The moment a region's last check succeeds, it votes "up" immediately. Recovery is not held hostage to a second opinion — a service that is back should be reported back as fast as possible.

This is the reasoning written in the code: MINIMUM_REGIONS_FOR_CONSENSUS = 2 and CONSECUTIVE_DOWN_CHECKS_REQUIRED = 2. The constants matter less than the logic behind them — the logic is that a slow target fools all regions at once, so the system needs a rule that survives a shared slow period.

There is a state the system can reach where it chooses to stay silent: degraded. When too many regions are excluded from the vote — an unstable region is removed from the vote entirely, neither "up" nor "down" — the remaining votes cannot reach consensus. The monitor is marked degraded, and degraded triggers no alert and no state change.

This is a deliberate choice. An alert that says "we are not sure" is worse than no alert, because it trains the reader to ignore alerts. PingRoot would rather say nothing than say something it cannot stand behind.

Let's be honest about the trade-off. A real outage that is brief enough to stay under the two-check threshold — a failure that happens and recovers before two consecutive checks observe it — triggers nothing. No alert, and no change on the status page.

That is a trade-off, not an oversight. The same mechanism that filters out slow-response noise also filters out very short real outages. PingRoot chose to miss the rare short blip rather than page people on the common slow response. If you need to catch every single blip, no consensus-based system will do that — this one doesn't pretend to.