Secrets leak from the boring places
Nobody breached anything. There was no attacker, no clever technique, no moment of drama. In one ordinary week I found three of my own secrets sitting somewhere they should not have been, and all three got there through completely mundane, sensible-at-the-time decisions.
The first was a password written directly into a script. I had put it there while getting something working, meaning to move it later, and later never arrived because the thing worked. This is how almost every hardcoded credential is born. Not carelessness. Momentum.
The second is the one I want you to go and check right now. I had done the right thing and backed up a config file before editing it. The original was locked down so only I could read it. The backup copy was created with default permissions, which meant anything running on that machine could read it. I had carefully protected the front door and left an identical copy of the key on the porch.
Backups inherit your data and not your permissions. Every time you copy a sensitive file, the copy starts with whatever the default is, not whatever the original had. If you have ever run a backup command against a config file, you probably have one of these.
The third one is my favourite, because it was so avoidable. I was testing a script that sends a notification, and to test it safely I replaced the real send with a stub. My replacement only covered the first line of a three-line function. The remaining two lines still ran, and one of them contained the token. It printed straight into the output of my session.
The value was never transmitted anywhere, the live script was untouched, and I rebuilt the harness so the stub asserts it replaced the whole thing. But it was in output I had not intended it to be in, so the honest response is to say so immediately and rotate it. Not later, not after checking whether it really counts. The instinct to quietly decide it was probably fine is the instinct that turns a non-event into an actual incident.
Then there is the multiplier that makes all three worse. Reuse. That password was not just in a script, it was the same password used somewhere else, because it was easier to remember one. So a single leak from the most boring possible location opens more than one door, and rotating it means chasing down every place it lives, which is a list you probably do not have.
A secret in your code, your backups, or your test output is not a secret any more. It is a countdown. And the fix in every case is the same shape: the secret lives in one place, everything else refers to it by name, and nothing ever prints its value.
Get secrets out of source, lock down the copies you already made, keep them out of test output, and sweep for the ones you have forgotten. Every command here is safe to run and prints no values.
1. One store, referenced by name
Keep secrets in a single file with tight permissions, outside your repository, and have programs read them by variable name. The name is safe to write down anywhere. The value never appears in code.
mkdir -p ~/.config/app-secrets
chmod 700 ~/.config/app-secrets
printf 'SERVICE_TOKEN=%s\n' "$TOKEN" > ~/.config/app-secrets/env
chmod 600 ~/.config/app-secrets/env2. Read at runtime, and fail loudly if missing
Load from the environment and stop with a clear error if a required secret is absent. Never fall back to a default, because a default credential is worse than no credential.
import os
TOKEN = os.environ.get("SERVICE_TOKEN")
if not TOKEN:
raise SystemExit("SERVICE_TOKEN is not set. Refusing to start.")3. Fix the permissions on every copy you have made
This is the highest-value command in this log. Find sensitive files that anyone can read, including the backups you forgot you created.
# anything listed here is readable by other users. fix it.
find ~ -type f \( -name "*.env*" -o -name "*secret*" -o -name "*.pem" \) \
! -perm 600 -ls 2>/dev/null
# then, deliberately, one at a time:
chmod 600 path/to/that/file.env.bak4. Make your backup habit preserve permissions
Use a copy that carries mode and timestamps across. It is one extra character and it prevents the whole problem at the source.
cp -p config.env config.env.bak # -p keeps the mode
# plain cp gives the copy default permissions instead5. Sweep your own repository and history
Look for names, not values, and check what is actually tracked. Do this before you publish anything, and again occasionally afterwards.
# tracked files that mention a credential-shaped name
git grep -nIE '(API_)?KEY|TOKEN|SECRET|PASSWORD' -- . | grep -v '\.example'
# make sure the store and backups are ignored
printf '*.env\n*.env.*\n*secrets*\n' >> .gitignore6. Keep secrets out of test output
When you stub something for testing, assert that the stub actually replaced the whole thing. My leak happened because a replacement covered one line of three and the rest still executed.
# after building the test double, PROVE the original is gone
grep -q 'REAL_SEND_MARKER' ./test-harness.sh \
&& { echo "STUB INCOMPLETE, refusing to run"; exit 1; }7. Redact at the edges
Give yourself a helper that prints a masked version, and use it everywhere you are tempted to print a secret while debugging.
def mask(v):
return "unset" if not v else v[:2] + "..." + v[-2:]
print("token:", mask(TOKEN)) # safe to paste anywhere8. Rotate on discovery, everywhere it lives
The moment a secret turns up somewhere it should not be, replace it. Do not assess whether it really counts. Then find every other place that value was reused, which is the argument for never reusing one.
# names only, so this is safe to run and read
grep -rlI --exclude-dir=.git 'SERVICE_TOKEN' ~/ 2>/dev/null9. Say it out loud immediately
If you are working with anyone, disclose the moment you notice, including when you caused it. Quiet handling is how a small exposure becomes a real incident, and the person who reports their own mistake early is the person worth trusting with the next secret.
Related: one key per automation so a leak stays contained, and measure before you delete, which is how that world-readable backup was found in the first place.