One digit, two months
My network worked. Things loaded, calls connected, nobody complained. It had also been configured wrong since July, by exactly one character, and I had no idea.
The setting was a passthrough: one box is supposed to hand its public address straight to the box behind it rather than doing its own translation. I had set it up months earlier, read it back, nodded, and moved on. One digit in the identifier was wrong. The box accepted the configuration happily, because the value was well formed. It just never matched anything, so the passthrough never engaged, and the network quietly ran in a mode I thought I had turned off.
While I was in there I found the second thing, which was worse. An old router, still powered, still plugged in, handing out addresses and answering name lookups on its own little subnet. Some devices had been getting their configuration from it for weeks. They worked. They just worked according to a different authority, with a stale list of names, and when one of them could not find something the symptom looked like an application bug.
Two clocks in a house that disagree by four minutes do not look broken. Every room is internally consistent. You only discover it when something has to be true in both rooms at once.
Here is the lesson, and it is uncomfortable because it contradicts how most of us actually operate: the absence of complaints is not verification. A network can be two networks. A setting can be accepted and inert. Something can be working and wrong at the same time, and the wrongness is invisible precisely because the working part absorbs it.
Both faults had the same shape. I had confirmed a belief by remembering that I had once arranged for it to be true, rather than by reading the current state and comparing it to what I intended. One digit and one forgotten box, two months, no symptoms. The fix for each was under a minute. Finding them was the work.
How to audit your own network for a second address or name authority, confirm a passthrough is genuinely passing through, and verify every device is on the subnet you believe. Every step is read only except where it says otherwise.
1. Ask a device what it was actually told, not what you configured
Start from the leaf, not the router. The device's own view is the only evidence that the configuration took effect.
# Linux
ip addr show
ip route show default
resolvectl status # which resolver, per interface
# macOS
ipconfig getpacket en0 | grep -E 'yiaddr|server_identifier|domain_name_server'
# Windows
ipconfig /allTwo fields matter most and both get skipped. The address that handed out the lease, and the resolver being used. If the lease came from something you did not expect, you have found a second authority.
2. Find out how many things are offering addresses
There should be exactly one. Ask the network and count the answers.
# watch for offers on the wire while something requests a lease
sudo tcpdump -ni <interface> port 67 or port 68
# then, on a device, release and renew so a request actually happens
# and read how many distinct servers replyMore than one responder is the fault. The second one is usually an old router someone left plugged in as a switch, a travel router, or a device with sharing enabled by accident.
3. Compare answers from every resolver you can reach
A stale resolver is harder to spot than a stale address, because the wrong answer still looks like an answer.
# ask each candidate the same question and compare
dig +short <a-name-on-your-network> @<resolver-A>
dig +short <a-name-on-your-network> @<resolver-B>
# and check what your device uses by default
dig +short <a-name-on-your-network>Different answers from different resolvers for an internal name is the stale-authority signature. Note which devices use which, because that tells you which symptoms belonged to which cause.
4. Verify a passthrough by reading both sides, not the setting
The configuration page tells you what you asked for. The two boxes tell you what happened. Compare the address the inner box holds with the address the world sees.
# on the inner box: what address does its outward interface actually have?
ip addr show <wan-interface>
# what does the public internet think you are?
curl -s https://api.ipify.org; echoIf passthrough is working these agree, and the inner box holds a public address. If the inner box holds a private address instead, the passthrough is not engaged no matter what the setting claims. That is the single check that would have caught my wrong digit in July.
5. Re-enter identifiers by copy, never by typing
My fault was a transcription error in a value that is long, meaningless to a human, and accepted if it is merely well formed. Those three properties together are a trap.
# read the identifier from the device that owns it
ip link show <interface> | grep ether
# paste it into the setting, then read the setting back and compare
# character by character, or diff two files, rather than eyeballingValidation that accepts any well formed value will never tell you the value is wrong. Treat "it saved without an error" as meaning nothing at all.
6. Inventory every device and the subnet it sits on
Walk the whole list once. The outliers are the story.
# what is actually on the segment you are on
ip neigh
# or a sweep of your own range, which you own and may scan
nmap -sn <your-range>/24Anything on an unexpected range is being served by something other than your intended authority. That is the second router, found from the other direction.
7. Write the expected state down, then diff against it quarterly
This is the only step that prevents a repeat, because the fault was never technical. It was that nothing ever compared belief to reality.
# network.md
# one address authority: <which box>, range ............
# one resolver: <which box>
# passthrough: expected ON, verified by inner box holding a public address
# known exceptions: ............ and why
# last verified: dateThe honest catch: this finds disagreements between things that are currently powered on. The old router I found was discoverable only because it was still running. A box that misbehaves intermittently will pass a quarterly audit and still ruin a Tuesday.
Related: auditing what your own services expose is the same habit pointed at software instead of wiring.