The most dangerous alert is the one that says everything is fine

What happened and why it matters

A tidying up job ran every night for days and reported success every single time. It was telling the truth. It was also doing absolutely nothing, while the thing it was supposed to be keeping under control quietly grew past the point where anything else could read it.

Here is the shape of it, and it is worth following closely, because your own setup almost certainly contains one of these.

The job had one simple rule: take anything older than a few days and move it into long term storage. That rule is correct. It exists so that something still in progress does not get filed away underneath you.

Then everything arrived on the same day

Every item was a few hours old. Not one of them was older than a few days, so not one of them qualified. The job looked at the pile, correctly worked out that nothing was eligible, correctly did nothing, and correctly recorded that it had finished without error.

Now run that forward. Tomorrow the pile is one day old. Still too new. The day after, two days. Still too new. Every night a clean success message about work that is not happening, and every night the problem is bigger.

Nothing in the system was broken. That is the entire horror of it.

The real cause was not the rule

The rule was good. The cause is that the job had exactly one way of deciding what to do, and therefore exactly one way to be wrong, and no way whatsoever of noticing.

Every check carries an assumption. This one assumed that a problem builds up slowly enough that age is a reasonable stand-in for size. Most days that holds. On the day it does not, the check is blind and cheerful.

So the fix is not to loosen the rule. Loosen it and you start filing away live work, which is the exact harm the rule was written to prevent. The fix is to add a second trigger that watches a completely different signal. Age was the first. Size is the obvious second, and size does not care what day anything arrived.

How to do it yourself

This applies to any job that runs on a schedule, in any programming language. The first two steps only look at things and change nothing.

1. Write down the assumption hiding in every healthy report

# the job says "ok" when: it ran without raising an error
# you actually care about: the pile got smaller
# the gap between those two lines is the bug

2. Measure the outcome, not the run

before=$(measure_the_thing)
run_the_job
after=$(measure_the_thing)
echo "job ok: $before -> $after"   # suddenly obvious

3. Add a trigger that watches something unrelated

HIGH   = 80    # above this, the size trigger fires
TARGET = 40    # bring it back down to roughly this
KEEP   = 6     # never touch the newest few, whatever happens

4. Let the safe rule run first, then top up

selected = pick_by_age(items, days=3)
if size_after(selected) > HIGH:
    selected += pick_oldest_until(items, TARGET,
                                  protect_newest=KEEP)

5. Make it announce a shortfall instead of exiting green

final = size_after(selected)
if final > TARGET:
    print(f"SIZE TRIGGER fired but could NOT reach target: "
          f"{final} still over {TARGET}. Everything remaining "
          f"is protected. NOT silently ignored.")

6. Protect the newest and the unfinished

if index in newest_n or item.is_open:
    continue          # never size-trim these, ever

7. Test the failing path on purpose

Force the situation where the trigger cannot reach its target, and confirm you actually see the shortfall message. Then run it against normal input and confirm it does nothing and says nothing.

You have not verified a check until you have watched it fail.

8. Prefer a loud wrong answer to a quiet right one

Given a choice between a job that occasionally complains unnecessarily and one that occasionally succeeds at nothing, take the complainer. You can calm a noisy check. You cannot fix a check that never speaks.

Next in this series: the monitor that reported dozens of outages at a service that was completely fine.