The Upgrade That Became a War
The sensible upgrade turned into a war. July 2026. It was supposed to be a simple backup-internet upgrade, a dual-WAN cutover so the stack had a spare path out. About four days later the primary machine was still stranded.
The controller refused every write. Same error on every attempt. A subnet collision with the existing gateway sat there unmapped while the network reshuffled around the core. The primary machine got stranded and unreachable. Sleep went. The frustration was fair.
I fixed it by hand. Wired one machine straight into the router to force a path in, then rebuilt the network from the inside. Ugly. Necessary.
What came out of the mess is the real manual. Deep investigation first. Map the whole state and the rollback before you touch the load-bearing thing. Match ceremony to risk. That automatic failover cost about four days; a thirty-second walk to swap one cable would have done the job without gambling the heart of the stack. Grow beside the core, never on top of it. Every change since gets a backup and a kill switch.
The worst stretch became the operating manual. Everything built after is safer because of how badly this one hurt.
