The health check that caused the outage it was watching for
Part 4 of 5 of the series When the computer lies to you. Start at part one.
What you will learn
- How the thing checking whether a machine is healthy can be what makes it unhealthy
- Why a machine that looks overloaded often has capacity to spare
- The one change that fixes it, and why a slightly old answer is better than a perfect one
There was a small page whose only job was to answer one question: is this machine healthy? It answered by going and finding out, right then, while you waited. Checking the processor. Asking about temperatures. Querying the graphics card. Every single time anybody asked.
On a quiet day, that is fine. Slow, but fine.
The trouble with being asked a lot
The entire point of such a page is that things check it automatically. And they check it more eagerly at exactly the moment you are worried.
So the dashboard asks. The alerting asks. You open the page yourself, because something feels off. Now three requests arrive at once, and the page can only handle one at a time, so they form a queue.
Each one takes as long as all that real work takes. Stacked up, they take longer than whoever asked was willing to wait. The dashboard gives up and shows the machine as offline. Which makes you look harder, which means more checking, which makes it slower still.
The health check had become the outage.
The thing doing the watching has to be cheap. If looking at something costs about as much as doing it, then looking becomes the work, and eventually you spend most of your capacity answering the question "are you busy?"
A manager who checks on progress by walking over and asking for a full spoken update, in detail, every ten minutes.
Nobody is lying. The updates are accurate. And no work is getting done, because the reporting has consumed the thing it was reporting on. At some point the honest answer to "how is it going" becomes "worse, because you keep asking".
The wrong diagnosis I nearly reached
The machine was not underpowered. That is exactly what I would have concluded if I had trusted the dashboard, because a machine that reads as offline under load looks precisely like a machine that has run out of capacity.
It had capacity to spare. It was spending it on doing the same expensive measurement over and over, on demand.
The fix is one idea
Separate collecting the numbers from handing them out.
A small background worker gathers the numbers on its own schedule and puts them somewhere. The page simply reads what is there and returns it. Collecting got no faster. It just stopped happening while somebody was waiting.
The response goes from something you can feel as a wait to something instant. Simultaneous requests stop queueing, because none of them do any work.
The numbers you get are now a few seconds old. For a health check that is not a compromise at all.
You were never asking about this exact instant. You were asking whether the machine is broadly all right, and a recent answer delivered reliably beats a perfect answer that arrives too late to read.
How this closes the series
The assistant lied because sounding right was easier than being right. The nightly job lied because its single rule could not see the case in front of it. The monitor lied because it had no way to say it was blind. And the health check lied because measuring cost more than the thing being measured.
In every case, the part you trusted to tell you the truth was the part that misled you. And in every case the fix was structural, rather than a matter of trying harder or asking more nicely.
Assume the reporter is wrong. Build so that it cannot be.
The pattern is the same in any programming language: a background refresher, a shared value, and a page that only reads.
1. Recognise the shape in your own code
What to look for: go through every status or health page you have and ask what it does before it replies.
The warning signs: measurements that pause, starting up another program, querying hardware, asking a database. If any of that happens while the caller waits, you have this bug waiting for a busy afternoon.
# BROKEN: real work inside the request, every single time
def handle_status():
cpu = sample_cpu(interval=1) # pauses for a whole second
temp = run("sensors") # starts another program
gpu = run("gpu-query") # starts another one
return json({"cpu": cpu, "temp": temp, "gpu": gpu})2. Add a background refresher
What this does: one small worker that loops forever, does the expensive gathering on its own schedule, and stores the result.
The key property: nothing waits on it. It is not connected to anybody's request.
import threading, time
_snapshot = {"status": "starting"}
def _refresh(every=5):
global _snapshot
while True:
_snapshot = {"cpu": sample_cpu(interval=1),
"temp": run("sensors"),
"gpu": run("gpu-query")}
time.sleep(every)
threading.Thread(target=_refresh, daemon=True).start()3. Make the page read-only
What this does: the page now returns a value. That is all it does.
Resist the temptation: no checking whether the value is fresh, no fetching if it is stale. Any of that puts work back on the request path and reintroduces the bug.
# FIXED: takes the same tiny time no matter how many callers
def handle_status():
return json(_snapshot)4. Handle several callers at once
What this does: switches to a server that can answer several people simultaneously.
Why it is still needed: a cheap page on a one-at-a-time server still forms a queue. Cheap and simultaneous is what actually holds up.
# one line, and simultaneous callers stop queueing
from http.server import ThreadingHTTPServer # not HTTPServer
ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever()5. Say how old the answer is
What this does: includes the time the numbers were gathered.
Why it matters: it costs nothing and it lets the caller spot that the background worker has died, which is the one new way this design can fail.
_snapshot = {..., "collected_at": time.time()}
# the caller can now spot an old snapshot = the refresher is stuck6. Protect the asking side too
What this does: if a machine has been failing recently, stop asking it for a while.
Why keep it afterwards: fix the page properly, and leave this in permanently as a safety net so one sick machine can never flood your network with retries while you work on it.
COOLDOWN = 60
if failed_recently(host, COOLDOWN):
return cached_or_unknown(host) # stop hammering it7. Test it with many callers at once, not one at a time
What this does: fires a dozen requests simultaneously and prints how long each took.
Why one at a time proves nothing: a single request often looks perfectly acceptable even when the design is wrong. Simultaneous requests are the condition that took the machine down, so that is the condition to test.
seq 12 | xargs -P12 -I{} curl -s -o /dev/null -w '%{time_total}\n' \
http://localhost:8080/status8. Apply the rule anywhere something reports
Anything whose job is to describe the state of something should read that state, never go and gather it. Logging, statistics, status pages, readiness checks.
If it has to do work to answer, move that work behind it and hand back the last known answer.
That closes the series. Start again at the assistant that said it had done something it had not, or read the standalone guide on never letting a robot spend your money while you sleep.
This page in the original Kyber Cypher voice: If your health check does real work, it will take down what it watches