One wrong character, two months
What you will learn
- Why nothing going wrong is not the same as everything being right
- How to find out whether your home has accidentally become two networks
- The one check that catches a setting which saved happily and never took effect
My network worked. Pages loaded, calls connected, nobody complained. It had also been set up wrong since July, by exactly one character, and I had no idea.
The setting was a handover: one box is supposed to pass its public identity straight through to the box behind it, instead of doing its own translation. I had set it up months earlier, read it back, nodded, and moved on. One digit in the identifier was wrong. The box accepted it happily, because the value looked like a valid value. It simply never matched anything, so the handover never happened, and my network quietly ran in a mode I thought I had turned off.
Then I found the second thing, which was worse
An old router, still plugged in, still powered, still handing out addresses and still answering name lookups on a little network of its own. Some devices in the house had been taking their instructions from it for weeks.
They worked. They just worked according to a different authority, with an out of date list of names, and when one of them could not find something the symptom looked like a broken app.
Two clocks in a house that disagree by four minutes do not look broken. Every room is consistent with itself. You only discover it when something has to be true in both rooms at once.
The uncomfortable lesson
It contradicts how most of us actually operate: the absence of complaints is not verification. A network can be two networks. A setting can be accepted and completely inert. Something can be working and wrong at the same time, and the wrongness is invisible precisely because the working part absorbs it.
Both of my faults had the same shape. I had confirmed a belief by remembering that I once arranged for it to be true, rather than by reading what is true now and comparing it to what I intended. One digit and one forgotten box, two months, no symptoms. Each fix took under a minute. Finding them was the work.
The absence of complaints is not verification.
How to check your own network for a second authority, confirm a handover is genuinely happening, and make sure every device is on the network you believe it is on. Every step only looks at things, except where it says otherwise.
1. Ask a device what it was told, not what you configured
What this does: asks one ordinary device what instructions it is actually running on.
Why start at a device and not the router: the device's own view is the only evidence that your setting took effect anywhere.
Two details matter and both get skipped: which box handed out the instructions, and which box is answering name lookups. If either came from something you did not expect, you have found a second authority.
# Linux
ip addr show
ip route show default
resolvectl status
# macOS
ipconfig getpacket en0
# Windows
ipconfig /all2. Find out how many things are handing out instructions
What this does: watches the network while a device asks to join, and counts how many boxes answer.
What the answer should be: exactly one. More than one is the fault.
Where the extra one usually comes from: an old router somebody left plugged in to use as a socket splitter, a travel router, or a device with sharing turned on by accident.
# watch for answers while something asks to join
sudo tcpdump -ni <interface> port 67 or port 68
# then, on a device, drop and renew its instructions so it really asks,
# and count how many distinct boxes reply3. Compare the answers from every name service you can reach
What this does: asks each candidate the same question about a name inside your house, and compares what comes back.
Why this is harder to spot than a wrong address: an out of date answer still looks like an answer. Nothing errors.
What a disagreement means: different answers for the same internal name is the signature of a stale second authority. Note which devices use which, because that tells you which symptoms belonged to which cause.
# ask each candidate the same question
dig +short <a-name-inside-your-house> @<first-candidate>
dig +short <a-name-inside-your-house> @<second-candidate>
# and what your device uses when left to itself
dig +short <a-name-inside-your-house>4. Verify a handover by reading both boxes, never the setting
What this does: compares the identity the inner box is holding with the identity the outside world sees.
Why this is the check that would have caught my wrong digit: the settings page tells you what you asked for. The two boxes tell you what happened.
What the answer means: if the handover is working, these agree and the inner box holds a public identity. If the inner box holds a private one instead, the handover is not happening, whatever the setting claims.
# on the inner box: what is its outward facing side actually holding?
ip addr show <outward-interface>
# and what does the public internet think you are?
curl -s https://api.ipify.org; echo5. Re-enter identifiers by copying, never by typing
What this does: reads the identifier from the device that owns it, so you can paste rather than retype.
Why my fault was almost inevitable: the value is long, meaningless to a human, and accepted as long as it merely looks valid. Those three properties together are a trap.
Then read it back and compare character by character, or compare two files properly, rather than looking at it and feeling satisfied. Treat "it saved without an error" as meaning nothing at all.
# read the identifier from the device that owns it
ip link show <interface>6. List every device and which network it sits on
What this does: walks the whole list once. The odd ones out are the story.
What an outlier means: anything on an unexpected range is being served by something other than the box you intended. That is the forgotten router, found from the other direction.
# what is actually near you on the network
ip neigh
# or a sweep of your own range, which is yours to look at
nmap -sn <your-own-range>7. Write down what you expect, then compare against it every few months
What this does: records the intended state in one short file.
Why this is the only step that prevents a repeat: the fault was never technical. It was that nothing ever compared belief to reality.
# network.md
# one box hands out instructions: ....... range .......
# one box answers names: .......
# handover: expected ON, confirmed by the inner box holding a public identity
# known exceptions: ....... and why
# last verified: dateThis finds disagreements between things that are switched on right now. The old router I found was only discoverable because it was still running. A box that misbehaves now and then will pass a quarterly check and still ruin a Tuesday.
Related reading: checking what your own programs are offering the house is the same habit pointed at software instead of wiring.
This page in the original Kyber Cypher voice: One digit, two months