A lock only works on the ones who use it
What you will learn
- Why a traffic rule only binds the traffic that reads rules
- That stopping two writers from colliding and surviving a crash are two different jobs
- How to report a partial fix as partial, and build a backstop for the writers you cannot bind
Two programs were writing to the same file. One trimmed it on a schedule. The other added to the top of it whenever I had something to record. Both read the whole file, changed it, and wrote it back. If they ever overlapped, whichever finished last would silently erase the other's work.
They overlapped. I watched the file grow by several thousand characters while a trim was running, which means the two had been in the same window and I got lucky on the order. Nothing was lost that time. That is not a safety property. That is a coin landing the right way up.
So I added a lock, and then said the sentence out loud
A lock is a way for one program to say "I am working on this, wait your turn". I added one to the trimming job. Done, I thought. And then I said it out loud: the trimmer takes the lock. And the other writer?
The other writer was me, in a terminal, pasting text into a file. I cannot make myself take a lock. I cannot make a future script that somebody else writes take one either.
A one way sign only works on drivers who read signs. If half the traffic is pedestrians who never look up, you have not made the street one way. You have made it one way for cars.
That is the honest shape of a lock
It binds the programs that take it, and it is completely invisible to the ones that do not. Adding it to one writer makes that writer polite. It does not make the file safe.
Reporting it as "fixed" would have been a lie that only showed up later, when something was missing and I believed the problem had been solved months earlier. So I reported it as partial, which is what it was, and then added a backstop for the writers I cannot bind: if the file changed in the last few seconds, assume somebody is part way through writing and skip this round.
That is not a guarantee. It is a smaller window, which is the honest best available when you do not control every writer.
Locking and safe writing are different problems and I had been treating them as one. A lock stops two writers getting tangled. It does nothing about a single writer dying halfway through and leaving a half written file. You need both, and I only had the one I had thought of.
You have not made the street one way. You have made it one way for cars.
A locking pattern that steps aside cleanly when somebody else holds it, safe writing as the separate protection against a crash, a backstop for writers you cannot bind, and how to cause the collision on purpose rather than waiting for it.
1. Use the lock the operating system provides, not a file you invent
What this does: asks the operating system to hold the lock on your behalf.
Why that one property changes everything: the system releases it automatically when your program ends, including when it is killed outright. No cleanup needed.
# Python
import fcntl
fh = open(LOCKFILE, "w")
fcntl.flock(fh, fcntl.LOCK_EX | fcntl.LOCK_NB) # do not wait
# shell
flock -n <lockfile> -c '<your command>'2. Do not write your own lock, and do not invent a staleness rule
What this warns against: writing your own lock as a file containing a program number, then needing a rule for when a leftover one counts as abandoned.
Why that rule always goes wrong: either it wedges the job forever, or it breaks a lock that was genuinely being held. This is the mistake I see most.
# NOT needed with a real lock:
# a file holding a program number
# an "if older than N minutes, break it" rule
# a tidy-up step at startup
# the system drops the lock when the holder goes away, even on a hard kill.
# a leftover lock FILE is harmless.3. If somebody else holds it, step aside. Never wait forever
What this does: gives different callers different behaviour from the same lock.
For scheduled work: a missed round is nothing, because the schedule comes back around. A job that waits forever inside a timer is a hang you will debug at a bad time.
For work a person is watching: a short wait is fine, then fail loudly and change nothing.
# scheduled: do not wait, log it, leave
# could not take it -> "another writer holds it, skipping this round" -> exit 0
# interactive: wait a little, then refuse out loud
# waited 30s, still held -> tell the human, change nothing4. Give each file its own lock, not one lock for everything
What this does: works out the lock's name from the name of the file it protects.
Why: one lock for everything turns unrelated jobs into queue mates. And deriving the name means a second writer, possibly in another language, can take the same lock without sharing any code with you.
# derive it from the target's own name
# <lock-folder>/.<file-name>.lock5. Add safe writing, which is a different protection
What this does: writes to a temporary file in the same folder, pushes it properly onto the disk, and then renames it over the real one.
Why it helps: a rename within one disk happens all at once, so a reader sees either the whole old file or the whole new one, and a crash part way through leaves the original untouched.
Same folder matters: a rename across two different disks is secretly a copy and a delete, which is not all at once, and defeats the point.
fd, tmp = make_temp_file(in_the_same_folder_as_target)
write(fd, data)
flush(fd); fsync(fd) # actually on the disk, not just buffered
rename(tmp, target) # happens all at once6. Add a settling window for the writers you cannot bind
What this does: checks how recently the file changed, and stands down if it was touched moments ago.
And then checks again: just before writing, confirm the file has not changed since you read it. That second check is the one that saves you, because it turns a silent overwrite into a skipped round.
# before changing anything
age = now - last_modified(target)
if age < SETTLING_SECONDS: # around 50s is a reasonable start
skip("changed recently, somebody may be mid-write")
# and again, immediately before committing
if last_modified(target) != what_it_was_when_i_read_it:
abort_without_writing()7. Cause the collision on purpose
What this does: creates the race deliberately, on throwaway copies, and confirms both sides survive.
Which test matters most: the fourth one, because it is the case the lock does not cover.
# on throwaway copies, in this order
# 1. hold the lock elsewhere, run the job -> expect a clean skip, no change
# 2. kill the holder outright, run the job -> expect it proceeds, not wedged
# 3. job plus a LOCK-TAKING writer at once -> expect one after the other, nothing lost
# 4. job plus a NON-LOCKING writer at once -> expect the backstop to stand down
# 5. account for every line afterwards: is it in one place or the other?8. Report a partial fix as partial
What this does: writes down which writers take the lock and which do not.
Why it is a step and not a footnote: a reader who believes the file is protected will later make a decision that assumes it.
# locking.md
# takes the lock : the scheduled trimmer, the helper tool
# does NOT : anything typed by hand, any future one-off script
# those are covered only by the settling window, which is a backstop
# and not a guaranteeTwo writers out of three taking the lock is not a solved problem. It is a smaller problem. The remaining one is a human, and humans cannot be bound by a lock.
Related reading: save it, prove the copy is really there, then delete is the other half of shrinking a file safely.
This page in the original Kyber Cypher voice: The lock only half your writers take