The worst alert is a green one
The Thing That Reports Is The Thing That Lies. The component you trust to tell you the truth, the assistant, the monitor, the log, the health check, is exactly the component that will lie to you. Five logs, five ways it happens, five fixes.
A maintenance job ran every night for days and reported success every single time. It was telling the truth. It was also doing absolutely nothing, and the thing it was supposed to be keeping under control had quietly grown past the point where anything else could read it.
Here is the shape of it, and it is worth following closely because your setup almost certainly has one of these in it. The job's rule was simple: take anything older than a few days and move it to long term storage. That rule is correct. It exists so that a conversation still in progress does not get filed away underneath you.
But then everything arrived at once, in a single long working day. Every item was hours old. Not one of them was older than a few days, so not one of them qualified. The job looked at the pile, correctly determined that nothing was eligible, correctly did nothing, and correctly logged that it had finished without error.
Now run that forward. Tomorrow the pile is one day old. Still too new. The day after, two days. Still too new. Every night, a clean success message about work that is not happening, and every night the problem is bigger. Nothing in the system was broken. That is the entire horror of it.
A red alert is a gift. It interrupts you, it points at something, it is impossible to ignore. A green report that is wrong gives you nothing at all, and it does it while actively reassuring you. You do not go looking, because you were just told everything is fine.
Think about the smoke alarm you never test. It chirps happily on its monthly self check. The self check confirms the battery has voltage. It does not confirm the sensor can still see smoke, because the only way to confirm that is to put smoke in front of it. For years you get a reassuring chirp that is technically honest and practically worthless.
The root cause is not the rule. The rule is good. The root cause is that the job had exactly one way to decide what to do, and therefore exactly one way to be wrong, and no way at all to notice. Every check carries an assumption. This one assumed that a problem builds up slowly enough that age is a reasonable proxy for it. Most days that assumption holds. On the day it does not, the check is blind and cheerful.
So the fix is not to loosen the rule. Loosen it and you start filing away live conversations, which is the exact harm the rule was written to prevent. The fix is to add a second trigger that watches a completely different signal. Age was the first. Size is the obvious second, and size does not care what day anything arrived.
One more thing, and it is the part most people skip. The second trigger must be able to fail out loud. If it tries to bring things back under the limit and cannot, because everything left is protected by the first rule, it has to say so. If it quietly gives up and exits successfully, you have just built a second thing that lies to you, and you will trust it more than the first one because it is newer.
A check that can only ever say ok is not a check. It is a decoration that happens to run on a schedule.
How to find the silent-green failures already in your setup, and how to add a second trigger that is capable of failing. Applies to any scheduled job, in any language.
1. Name the assumption in every healthy report
Go through your scheduled jobs and for each one write down the sentence "this reports success when ____". If that sentence is only about the job not crashing, it is not measuring the outcome you care about. That gap is where a silent-green failure lives.
# job says "ok" when: it ran without raising an error
# you actually care about: the pile got smaller
# the gap between those two lines is the bug2. Measure the outcome, not the run
Record the thing you actually care about before and after. If a run reports success without moving that number, you now have evidence rather than a feeling.
before=$(measure_the_thing)
run_the_job
after=$(measure_the_thing)
echo "job ok: $before -> $after" # suddenly obvious3. Add a trigger on an orthogonal signal
Keep the original rule exactly as it is. Add a second one that watches something unrelated. If the first is time-based, make the second volume-based, and give it a high water mark and a target.
HIGH = 80 # above this, the size trigger fires
TARGET = 40 # bring it back down to roughly this
KEEP = 6 # never touch the newest N, whatever happens4. Let the original rule run first, then top up
Order matters. Apply the safe age-based rule, and only then let the size trigger take more if the result is still over the limit. The second trigger is a backstop, not a replacement.
selected = pick_by_age(items, days=3)
if size_after(selected) > HIGH:
selected += pick_oldest_until(items, TARGET,
protect_newest=KEEP)5. Make it report a shortfall instead of exiting green
This is the step that matters. If the trigger cannot reach its target because everything left is protected, it must say so in its output. Silence here recreates the original bug in a new place.
final = size_after(selected)
if final > TARGET:
print(f"SIZE TRIGGER fired but could NOT reach target: "
f"{final} still over {TARGET}. Everything remaining "
f"is protected. NOT silently ignored.")6. Protect the recent and the unfinished
The size trigger must never be able to eat live work. Keep a fixed number of the newest items, and skip anything still marked open or in progress. A backstop that causes the harm the first rule prevented is not a backstop.
if index in newest_n or item.is_open:
continue # never size-trim these, ever7. Test the failing path deliberately
Force the condition where the trigger cannot reach its target and confirm you see the shortfall message. Then run it against normal input and confirm it is a clean no-op that writes nothing. You have not verified a check until you have watched it fail.
# make everything protected, then run it
# expect: the shortfall line, NOT a silent success8. Prefer a loud wrong answer to a quiet right one
Given the choice between a job that occasionally complains unnecessarily and a job that occasionally succeeds at nothing, take the complainer. You can tune a noisy check. You cannot tune a check that never speaks.
Next in this series: the monitor that fired dozens of outage alerts at a service that was completely fine.