Copy it, check the copy is really there, then delete

What happened and why it matters

I had a job that stopped a working file from growing forever. Old entries were trimmed out on a schedule. The notes said this was safe, because old entries were already copied into a permanent store. The notes were wrong, and they had been wrong in writing for weeks.

Nothing copied them. The permanent store and the working file were written separately by different things at different times, and the sentence claiming one mirrored the other was an assumption that had been written down, which made it look like a fact. Everybody who read the instruction trusted it, because it was in the notes, and notes are where facts live.

I found it by doing the one thing nobody had done. Before deleting, I went and looked in the permanent store for the entries I was about to remove. They were not there.

Then it got more interesting, because the next two faults were the opposite shape

The first was a safety gate pointing backwards. It refused to copy any entry into the permanent store unless that entry was already in the permanent store. Read that again, because I had to. The entries most in need of saving, the ones that existed nowhere else, were exactly the ones it would not save.

It never lost anything. It simply never saved them either, so they could never be trimmed, so the file grew forever. A deadlock wearing a safety vest.

The second was quieter and taught me the most. The part that found where one entry ended and the next began used a pattern that could not match four of its own seventeen headings, because those four had an extra piece before the date. A heading it cannot match is not a boundary, so those four entries silently became part of the entry above them.

They were saved. Nothing was lost. But they were saved under the wrong name, and the job reported the wrong count, and every number I would have quoted was wrong while the data itself was perfectly fine.

How to do it yourself

The save, check, delete pattern, plus the two subtler failures: a gate that blocks the thing it should protect, and matching that is too loose to trust. The ordering is the whole point.

1. Never delete on the strength of a written note

# the only correct order
# 1. WRITE the entry to the permanent store
# 2. RE-READ the store and find the entry in it
# 3. COMPARE what came back against the original, exactly
# 4. only now remove it from the working file
# any failure at 1, 2 or 3 -> remove NOTHING and complain loudly

2. Compare the content, not a count

want  = the_entry_text
found = read_back_from_the_store(locate(want))
if fingerprint(found) != fingerprint(want):
    refuse_to_delete()

3. Match on anchored boundaries, never on a fragment

# the check that caught my four of seventeen
# headings the pattern matched : 13
# headings counted another way : 17
# a gap means your pattern is blind to some of your own entries

4. Beware the fragment that looks like a match

# anchor it to the exact field and compare whole values,
# rather than hunting for the text anywhere it appears

5. Check your safety gates for being backwards

# the backwards gate, stated plainly
# "refuse to save anything not already saved"
#   -> the unique entries are the ones refused
#   -> they can never be removed, so the file grows forever

# the right way round
# save unconditionally; treat "is it elsewhere?" as a NOTE on the entry,
# never as permission to save it

6. Make the permanent store add only, and test the refusal

# two deliberate sabotages
# 1. make the store unwritable  -> expect: refuse, delete nothing
# 2. corrupt what gets written  -> expect: check fails, delete nothing
# then confirm the working file is unchanged, byte for byte

7. Make it safe to run twice, because it will be interrupted

# before adding, look for it in the store, exactly
# already there -> skip the write, go on to check and delete
# not there     -> write, then check, then delete

Related reading: read the record before you believe your own story is the same lesson about trusting a written claim, and a backup you have never tested is just a hope is where checking by actually restoring belongs.