If your health check does real work, it will take down what it watches
The Thing That Reports Is The Thing That Lies. The component you trust to tell you the truth, the assistant, the monitor, the log, the health check, is exactly the component that will lie to you. Five logs, five ways it happens, five fixes.
A status endpoint existed to answer one question: is this machine healthy? It answered by going and finding out, right there, while you waited. Sampling the processor. Shelling out to ask about temperatures. Querying the graphics card. Every single time anyone asked.
On a quiet day, fine. Slow, but fine. The trouble is that the whole point of a status endpoint is that things poll it, and things poll it more eagerly exactly when you are worried. So the dashboard asks. The alerting asks. You open the page yourself because something feels off. Now three requests arrive at once, and the endpoint is single threaded, so they queue.
Each one takes as long as it takes to do all that real work. Stacked up, they blow past the timeout the caller was willing to wait. The dashboard gives up and shows the machine as offline. Which makes you look harder, which means more polling, which makes it slower. The health check has become the outage.
The observer has to be cheap. If looking at something costs about as much as doing the work, then looking becomes the work, and you will eventually be spending most of your capacity answering the question "are you busy?"
There is a lovely everyday version of this. A manager who checks on progress by walking over and asking for a full verbal status update, in detail, every ten minutes. Nobody is lying, the updates are accurate, and no work is getting done. The reporting consumed the thing it was reporting on. At some point the honest answer to "how's it going" becomes "worse, because you keep asking."
The machine was not underpowered. That was the diagnosis I would have reached if I had trusted the dashboard, because a machine that reads as offline under load looks exactly like a machine that has run out of capacity. It had capacity to spare. It was spending it on a blocking sample inside a request handler, over and over, on demand.
The fix is one idea: separate collecting from serving. A small background worker gathers the numbers on its own schedule and writes them into a variable. The endpoint reads that variable and returns it. Collection got no faster. It simply stopped happening on the caller's clock.
The response goes from something you can perceive as a wait to something instant. Concurrent requests stop queueing, because none of them do any work. And the numbers you get are a few seconds old, which for a health check is not a compromise at all. You were never asking about this exact instant. You were asking whether the machine is broadly all right, and a recent answer delivered reliably beats a perfect answer that arrives too late to be read.
That is the theme of this whole series, arriving from a fourth direction. The assistant lied because sounding right was easier than being right. The job lied because its one rule could not see the case in front of it. The monitor lied because it had no way to say it was blind. And the health check lied because measuring cost more than the thing being measured. In every case the component you trusted to tell you the truth was the component that misled you, and in every case the fix was structural rather than a matter of trying harder.
Assume the reporter is wrong. Build so that it cannot be.
Move collection off the request path and serve a cached snapshot. The pattern is the same in any language: a background refresher, a shared value, an endpoint that only reads.
1. Recognise the shape in your own code
Look at every status or health handler you have and ask what it does before it replies. Blocking samples, shelling out to another program, querying hardware, hitting a database. If any of that happens inside the handler, you have this bug waiting for a busy afternoon.
# BROKEN: real work inside the request, every time
def handle_status():
cpu = sample_cpu(interval=1) # blocks a full second
temp = run("sensors") # spawns a process
gpu = run("gpu-query") # spawns another
return json({"cpu": cpu, "temp": temp, "gpu": gpu})2. Add a background refresher
One thread, one loop, one sleep. It does the expensive collection on its own schedule and stores the result. Nothing waits on it.
import threading, time
_snapshot = {"status": "starting"}
def _refresh(every=5):
global _snapshot
while True:
_snapshot = {"cpu": sample_cpu(interval=1),
"temp": run("sensors"),
"gpu": run("gpu-query")}
time.sleep(every)
threading.Thread(target=_refresh, daemon=True).start()3. Make the handler read-only
The endpoint now returns a value. That is all it does. No branching on freshness, no fetching if stale, because any of that reintroduces work on the request path.
# FIXED: constant time, no matter how many callers
def handle_status():
return json(_snapshot)4. Serve concurrently as well
A cheap handler on a single threaded server still serialises. Use a threading server so simultaneous callers do not queue behind each other. Cheap plus concurrent is what actually holds up.
# one line, and concurrent callers stop queueing
from http.server import ThreadingHTTPServer # not HTTPServer
ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever()5. Say how old the answer is
Include the age of the snapshot in the response. It costs nothing and it tells the caller whether the background worker has died, which is the one new failure mode this design introduces.
_snapshot = {..., "collected_at": time.time()}
# caller can now spot a stale snapshot = refresher is stuck6. Protect the caller too
On the polling side, give a slow endpoint a short timeout and a cooldown so one sick machine cannot flood your network with retries while you fix it. Fix the endpoint properly, and leave the backoff in as a permanent safety net.
COOLDOWN = 60
if failed_recently(host, COOLDOWN):
return cached_or_unknown(host) # stop hammering it7. Measure it under concurrency, not one request at a time
A single request often looks acceptable even when the design is wrong. Fire a dozen at once and compare. That is the test that shows you the difference, because that is the condition that took you down.
# before: requests queue, worst case blows the caller's timeout
# after: all return immediately, no queue, no timeouts
seq 12 | xargs -P12 -I{} curl -s -o /dev/null -w '%{time_total}\n' \
http://localhost:8080/status8. Apply the rule everywhere something reports
Any endpoint whose job is to describe state should read state, never gather it. Logging, metrics, status pages, readiness probes. If it has to do work to answer, move that work behind it and hand back the last known answer.
That closes the series. Start again at the assistant that said it did something it hadn't, or read the standalone on never letting an automated job spend money unattended.