Prove the monitor isn't the thing that's broken
The Thing That Reports Is The Thing That Lies. The component you trust to tell you the truth, the assistant, the monitor, the log, the health check, is exactly the component that will lie to you. Five logs, five ways it happens, five fixes.
The alerts said the service was down. They said it dozens of times, over days, at all hours. The service was never down. Not once. Every alert was real in the sense that something genuinely failed, and every alert was pointing at the wrong thing entirely.
What actually failed was the monitor's own name lookup. Before it could check anything, it had to turn a name into an address, and the resolver it used kept coming back empty for that particular name. The monitor asked once, got nothing, and concluded the only thing it had been taught to conclude: down.
It had two states. Up and down. There was no state for "I asked and could not find out," so that condition had to land somewhere, and it landed in the alarming one. Not because anyone decided it should. Because nobody decided anything, and down was the default.
A broken gauge does not read honest, it reads empty. Your fuel gauge does not display a shrug when the sender fails. It swings to zero, and zero is a number you believe, so you pull over and start investigating an empty tank you do not have.
The near miss is the interesting part. Armed with days of outage alerts, the obvious move is to go work on the service. Restart it, add retries, tune its timeouts, maybe move it somewhere more reliable. All of that effort would have gone into a system that had never once misbehaved, and the alerts would have kept coming, because you would not have touched the thing that was actually broken.
There was a second bug underneath, and it is the kind that hides for years. The check ran in a way that captured only its final result and threw away everything it had learned on the way, including the address it had looked up. So every alert recorded that field as empty. Reading the alert history later, it looked like the service had no address at all, over and over. That was not evidence. It was the same missing value, printed dozens of times, masquerading as a pattern.
This deserves its own warning, because it is the trap that turns one bug into a false conclusion. When your diagnostics all agree, check whether they are actually independent observations or one broken observation repeated. Dozens of identical readings from one instrument is one reading.
The fix has two halves and neither is clever. First, give the monitor a third state. Not up, not down, but cannot determine. That state gets logged and does not page anyone, because waking someone for "my instrument is confused" trains them to ignore the instrument. Second, before it is allowed to conclude anything, make it try more than one path. Several resolvers, not one. If any of them answers, you have an address and you can do a real check.
The result reads differently, and the difference is the whole point. Not "the service is down," which was false. Instead: "I could not resolve the name from any resolver, so I am blind right now, and the service may well be fine." That sentence is true, it is actionable, and it points at the correct machine.
Before you debug the service, prove your instrument works. It costs a few minutes and it saves you from confidently fixing something that was never wrong.
A three-state health model with a fallback lookup path, plus the shell gotcha that silently throws away everything your probe learned. Generic, works for any uptime check you run at home.
1. Stop using two states
Binary up and down forces every unknown into one of them, and it will always be the alarming one. Add a third state for "could not determine" and treat it as its own outcome with its own handling.
# three states, not two
# CANT_RESOLVE = the monitor is blind, log it, do NOT page
# UP = resolved AND the request succeeded
# DOWN = resolved BUT the request failed <- only this pages2. Try several lookup paths before concluding anything
One resolver failing is not an outage. Ask a few in turn and take the first that answers. Only when every path comes back empty are you allowed to say you cannot determine.
RESOLVERS="1.1.1.1 8.8.8.8 9.9.9.9"
lookup() {
for r in $RESOLVERS; do
ip=$(dig +short "$1" @"$r" | grep -E '^[0-9.]+$' | head -1)
[ -n "$ip" ] && { echo "$ip"; return 0; }
done
return 1 # every path failed: blind, not down
}3. Only page on a confirmed DOWN
Resolution succeeded, so you have a real address, and the actual request to that address failed. That is an outage. Everything else is a log line.
if ! ip=$(lookup "$HOST"); then
log "CANT_RESOLVE: monitoring is blind, service may be fine"
exit 0 # no page
fi
code=$(curl -s -o /dev/null -w '%{http_code}' "https://$HOST/")
[ "$code" = "200" ] && log "UP $ip" || page "DOWN $ip (http $code)"4. THE GOTCHA: a subshell throws away everything it learned
This one is silent and it will ruin your alert history. Capturing a function's output with backticks or the dollar-parenthesis form runs it in a subshell. Any variable it sets inside is gone the moment it returns, so your alerts record empty fields and you draw conclusions from them.
# BROKEN: probe sets PUB internally, but $( ) forks a subshell,
# so PUB is empty out here and every alert logs "address: none"
code=$(probe)
echo "down, address was $PUB" # always empty. always.5. Return everything you learned in one string
Do not rely on variables set inside a captured call. Pack the result into one line, return it, and split it in the caller. Now the address is real in your logs and you can trust what you read later.
# FIXED: one line out, parsed by the caller
probe() { echo "${code:-000}|${ip:-none}|${resolver:-none}"; }
result=$(probe)
code="${result%%|*}"; rest="${result#*|}"
ip="${rest%%|*}"; resolver="${rest##*|}"6. Say the true sentence in the alert
Write the message to match the state. Blind is not down, and recovery from blind is not recovery of the service. Precise wording here is what stops the next person chasing the wrong machine.
# blind: "cannot resolve from any resolver, may be fine"
# down: "resolved to 203.0.113.10 but returned http 502"
# recovered: "lookup recovered AND service verified up"7. Test your instrument before you trust its history
Point the monitor at a name that does not exist and confirm you get the blind state and no page. Point it at something real and confirm it comes back up. If you have never seen your monitor report each of its states on purpose, you do not know which one it defaults to.
Next in this series: the health check that became the outage it was watching for.