Your computer is not dying. The thing watching it is cooking it.
What you will learn
- How to tell an overheating machine from an overworked one in five minutes
- Why the software watching for problems can be the problem
- The number that proves it, and the one that only looks like it does
One of my computers had been running far too hot for weeks. The processor was deliberately slowing itself down, which is what a chip does when it is trying not to cook itself. Every explanation I reached for was a hardware explanation. Dust. Dried out paste. A fan on its way out. An old machine finally showing its age.
It was none of those. The thing making the machine hot was the software I had installed to tell me whether the machine was hot.
A processor core is one of several separate workers inside the chip. Most computers have several, and they can be busy independently of each other.
Throttling is the chip choosing to run slower on purpose because it is too hot. It is a safety measure, not a fault, but it makes everything feel sluggish.
The culprit was a monitoring program, the kind that quietly collects statistics so you can look at nice graphs of how your system is doing. It was sitting on one core and holding it, constantly, day and night. Not an occasional spike. Not a nightly job. A permanent load that I had introduced myself and then stopped noticing, because monitoring is the sort of thing you set up once and then mentally file under helping.
The thing doing the watching was the load. The graph showing me a hot machine was being drawn by the very process making the machine hot.
I was watching a fire on a camera that was itself on fire, and every reading made perfect sense as long as I never asked what was producing it.
The clue I walked past for weeks
The evidence was there the whole time. The slowing down was not spread across the processor. It was almost entirely on one core.
That matters enormously. A machine with failing cooling gets hot everywhere, because heat does not respect the boundaries between cores. One core buried while its neighbours sit idle is not a cooling story at all. It is a workload story, and workload stories have a program at the end of them.
So I moved the statistics collection onto a different, low powered machine that exists to be switched on all the time and do very little. That was the entire fix. No parts, no paste, no screwdriver.
The result was not subtle. The counter tracking how often the chip had slowed itself down, which had been climbing for weeks, stopped dead and has not moved since. The temperature fell by more than twenty degrees Celsius. The machine that was supposedly dying of old age was perfectly fine. It had simply been asked to carry a passenger it never needed.
Two things I got wrong on the way
A debugging story where the person debugging is always right is a story that will not help you. So here are both mistakes.
First, I wrote a confident plan that turned out to be impossible. I was going to read the fan speeds, load the right software for the sensors, and get hard numbers. Then I actually looked at what the machine physically was. It is a type of computer where those sensors do not exist. Not difficult to read. Not present. The component my plan depended on was never fitted, and forcing that sensor software onto hardware that lacks it is a well known way to make the machine freeze completely. My plan was not risky, it was impossible, and I only discovered that because I checked the hardware before running a plan I had already written down.
Second, I had been calling it a dust problem in writing. Confidently. Then the case came off and it was clean. Not fairly clean. Clean. Somewhere between thinking it and typing it, I had promoted a plausible guess into a stated fact. That happens quietly and constantly, and it is worth watching for in your own notes.
I still cannot read the fan speed on that machine, and I am not going to pretend that a reasonable guess is the same as a measurement. What I can tell you is that the slowdown counter froze and the temperature dropped more than twenty degrees the moment the watching software moved. Those are measurements.
The lesson
Before you blame the hardware, check whether the thing monitoring the hardware is the thing straining it. Your monitoring is not free. It is a program, it uses the processor, and processor work turns into heat.
This is the same shape as the health check that caused the outage it was watching for. The act of observing changed the thing being observed.
I was watching a fire on a camera that was itself on fire.
This takes about five minutes and tells you whether you have a workload problem or a cooling problem. Everything here only reads information, apart from step five, which is the one that actually changes something.
1. Find out what is holding a core
What this does: lists the programs using the most processor, busiest first.
What to look for: something with a high number that never comes back down. You want the program that is always there, not one that briefly spikes while you happen to be looking.
Why two rounds: the first reading is a lifetime average and is misleading. The second one is the real picture.
top -bn2 | head -20
# if the program runs in a container, this is the quicker question
docker stats --no-stream2. Ask whether it is one core or all of them
What this does: shows how busy each core is separately.
Why this single question splits the whole diagnosis: heat on every core points at cooling. One core buried while the others idle points at a program. Run the first command, then press the 1 key to break the display out per core.
top # then press 1 for a line per core
# or this, if you have it installed
mpstat -P ALL 1 33. Read the slowdown counter, not just the temperature
What this does: the first command shows how many times each core has been forced to slow down. The second shows current temperatures.
Why the counter is the better number: temperature moves around constantly and on its own tells you very little. The counter only goes up when the processor has genuinely had to protect itself, so it is the number that says whether you have a real problem. It is also counted per core, which is exactly the split you need.
Write these down now. They are your before. Without them you will never be able to prove the fix worked.
grep . /sys/devices/system/cpu/cpu*/thermal_throttle/core_throttle_count
sensors4. Check the two line up, before touching anything
Confirm that the busy core and the slowing down core are the same core. If core 3 is pinned and core 3 is throttling, you have your answer. If they are different cores, keep looking, because you are about to fix the wrong thing.
5. Move the watcher, or make it cost less
What this does: the command below switches the monitoring program off on this machine and stops it starting again at boot. The data it has already collected is not touched.
The cheaper option first: most monitoring programs check far more often than anyone actually needs. Changing it to look every fifteen seconds instead of every second is fifteen times less work for exactly the same graphs. Try that before moving anything.
How to undo it: the same command with enable in place of
disable.
# option A: check less often (in your monitoring program's own settings)
# option B: move it. switch it off here, run it on another machine.
sudo systemctl disable --now metrics-collectorA low powered machine that is always on is ideal for this, because collecting statistics is exactly the sort of small steady job it is good at. You can leave a tiny reporting piece on the machine being watched if you still want its numbers. The expensive part is the collecting and storing, not the reporting.
6. Measure again, and prove it rather than assuming it
What this does: exactly the same two commands from step three.
What should happen: come back a few hours later. The slowdown counter should have stopped increasing, which is the proof. The temperature should have fallen, which is the reward.
If the counter is still climbing: the monitoring was not your problem. That is a real result and worth knowing, rather than telling yourself a comforting story.
grep . /sys/devices/system/cpu/cpu*/thermal_throttle/core_throttle_count
sensorsMine froze and stayed frozen, and the temperature came down by more than twenty degrees. That is what cause and effect looks like when you actually check for it.
Related reading: the health check that caused the outage it was watching for is the same illness in a different costume, and measure before you delete is the habit that makes all of this possible.
This page in the original Kyber Cypher voice: Your machine isn't dying. The thing watching it is cooking it.