Before you fix the machine, prove the gauge is not the broken part

What happened and why it matters

The warnings said the service was down. They said it dozens of times, over several days, at all hours. The service was never down. Not once. Every warning was real, in the sense that something genuinely failed, and every single one was pointing at the wrong thing.

What actually failed was the monitoring tool's own lookup. Before it could check anything, it had to turn a name into an address, and the phone book it was using kept coming back empty for that one name. The monitor asked once, got nothing, and concluded the only thing it had ever been taught to conclude: down.

It only had two answers

Up and down. There was no answer for "I asked and could not find out", so that situation had to land somewhere, and it landed in the alarming one.

Not because anyone decided it should. Because nobody decided anything, and down was what happened by default.

The near miss

Armed with days of outage warnings, the obvious next move is to go and work on the service. Restart it. Make it retry. Adjust its timings. Perhaps move it somewhere more reliable.

All of that effort would have gone into a system that had never once misbehaved, and the warnings would have kept arriving, because you would not have touched the thing that was actually broken.

The second bug, which hides for years

The check ran in a way that kept only its final answer and threw away everything it had learned along the way, including the address it had looked up. So every warning recorded that field as empty.

Reading the history back later, it looked as though the service had no address at all, over and over again. That was not evidence. It was one missing value, printed dozens of times, wearing the costume of a pattern.

The fix, in two unclever halves

First, give the monitor a third answer. Not up, not down, but "cannot determine". That one gets written down and wakes nobody, because waking someone for "my instrument is confused" simply trains them to ignore the instrument.

Second, make it try more than one route before concluding anything. Several phone books rather than one. If any of them answers, you have an address and you can do a real check.

The result reads completely differently, and the difference is the entire point. Not "the service is down", which was false. Instead: "I could not look up the name from any source, so I am blind right now, and the service may well be perfectly fine."

That sentence is true, it is useful, and it points at the correct machine.

How to do it yourself

A three-answer health check with a fallback lookup, plus one genuinely nasty trap that silently throws away everything your check learned.

1. Stop using only two answers

# three answers, not two
#   CANT_RESOLVE = the monitor is blind, log it, do NOT wake anyone
#   UP           = looked up fine AND the request worked
#   DOWN         = looked up fine BUT the request failed   <- only this pages

2. Try several lookup routes before concluding anything

RESOLVERS="1.1.1.1 8.8.8.8 9.9.9.9"
lookup() {
  for r in $RESOLVERS; do
    ip=$(dig +short "$1" @"$r" | grep -E '^[0-9.]+$' | head -1)
    [ -n "$ip" ] && { echo "$ip"; return 0; }
  done
  return 1        # every route failed: blind, not down
}

3. Only raise the alarm on a confirmed failure

if ! ip=$(lookup "$HOST"); then
  log "CANT_RESOLVE: monitoring is blind, service may be fine"
  exit 0                       # no alarm
fi
code=$(curl -s -o /dev/null -w '%{http_code}' "https://$HOST/")
[ "$code" = "200" ] && log "UP $ip" || page "DOWN $ip (http $code)"

4. The trap: a captured command forgets everything it learned

# BROKEN: probe sets the address internally, but $( ) makes a copy,
# so it is empty out here and every alert logs "address: none"
code=$(probe)
echo "down, address was $PUB"        # always empty. always.

5. Return everything you learned in one line

# FIXED: one line out, taken apart by the caller
probe() { echo "${code:-000}|${ip:-none}|${resolver:-none}"; }
result=$(probe)
code="${result%%|*}"; rest="${result#*|}"
ip="${rest%%|*}"; resolver="${rest##*|}"

6. Make the message say the true sentence

Blind is not down, and recovering from blind is not the service recovering. Precise wording here is the thing that stops the next person chasing the wrong machine.

# blind:     "cannot look up the name from any source, may be fine"
# down:      "looked up fine but returned an error"
# recovered: "lookup recovered AND service verified working"

7. Test your instrument before you trust its history

Point the monitor at a name that does not exist and confirm you get the blind answer and no alarm. Then point it at something real and confirm it reports working.

If you have never watched your monitor report each of its answers on purpose, you do not know which one it falls back to.

Next in this series: the health check that became the outage it was watching for.