How does the alarm know it is not the broken one?

What happened and why it matters

I put a watchdog on a phone. It sits in my pocket, checks that my machines are answering, and tells me when they are not. A watcher that lives somewhere other than the thing it watches is a genuinely good idea, and it immediately taught me two things I did not want to know.

The first problem was time

The watchdog has a dead man's switch. It is supposed to check in on a fixed rhythm, and a missed check in is itself the alarm. For days the rhythm was wrong. Not broken, wrong. Checks that should have been minutes apart were arriving tens of minutes late, in bursts, as though the phone had been asleep and then suddenly remembered.

It had. A modern phone aggressively pauses background work to save battery. It does not tell your timer that it has been postponed. It simply postpones it. So the dead man's switch, whose entire job was to be reliable about time, was running on a device that treats time as negotiable.

The fix was to have each check hold the phone awake for the few seconds it needs, so the system keeps the processor running until the work is genuinely finished.

The second problem is not fixed, and this guide exists mostly to say so

My phone cannot reach the house. What does that mean? It means one of two things, and from where the phone is standing they look identical. Either the house is genuinely down, or my phone has lost its own connection and everything at home is fine without me.

One watcher cannot tell these apart. This is not a harder version of the same problem. It is a different problem, and no amount of cleverness inside the phone resolves it, because the phone is the thing in doubt.

What I did instead was stop pretending. The watchdog used to report that situation as "the network is down", which was a claim it could not support. Now it has a third answer that says, in effect, "I cannot reach the house from here and I have no second opinion". That is less satisfying and considerably more true, and the difference matters, because an alert that overstates its confidence trains you to ignore alerts.

How to do it yourself

The dead man's switch pattern, the sleep problem that makes it lie on a phone, how to describe severity honestly when you do not know, and why a second watcher is not optional. The last part is a design rather than a finished thing, and it is labelled as such.

1. Build the dead man's switch the right way round

# the watcher writes a timestamp every N seconds
# heartbeat: { at: <timestamp>, by: <which watcher> }

# something ELSE checks how old that timestamp is
# older than 3N -> the watcher itself is the problem

2. Expect the phone to postpone you, and hold it awake

# the shape: take it, do the work, ALWAYS let go
hold_awake()
try:
    do_the_check()
finally:
    let_go()

3. Measure the real gaps before trusting the fix

# log every beat with its real time, then read the gaps
# healthy : N, N, N, N
# asleep  : N, N, 14N, N, 9N   <- bursts after long gaps

4. Give "I do not know" its own answer, and do not colour it like a failure

# three answers, with honest meanings
# OK      : I reached it
# DOWN    : I could not reach it AND a second watcher agrees
# UNKNOWN : I could not reach it and I have no second opinion

# and make the code refuse to say DOWN without the witness

5. Check your own connection first, and say which leg failed

# the order matters
# 1. can I reach the wider internet at all?  no  -> I am offline, say so
# 2. can I reach the target?                 yes -> OK
# 3. internet yes, target no, witness says yes  -> target is DOWN
# 4. internet yes, target no, no witness        -> UNKNOWN

6. The second watcher, which I have not finished

# requirements
# outside your building and off your connection
# checks the SAME target on its own schedule
# publishes a result your first watcher can read
# fails independently, or it is not a second opinion

# candidates, and the tradeoff of each
# a small rented machine : most control, a monthly cost, one more thing to look after
# a friend's spare       : free, depends on their power and goodwill
# a paid checking service: least work, least control, another account to manage

7. Keep the known blind spot written next to the code

# watchdog.md
# KNOWN LIMIT: one watcher only. UNKNOWN means cannot-see-it-from-here,
# not down. Do not act on UNKNOWN as though it were DOWN.
# OPEN: second watcher not built yet. Candidates above.

Related reading: the most dangerous alert is the one that says everything is fine is what happens when a watcher fails quietly, and prove the gauge is not the broken part is the same question one layer in.