Your machine isn't dying. The thing watching it is cooking it.
One of my machines had been running hot for weeks. The processor was throttling itself, which is what a chip does when it is trying not to cook: it slows down on purpose to shed heat. Every explanation I reached for was a hardware explanation. Dust. Dried out thermal paste. A fan on its way out. An ageing machine finally showing its age.
It was none of those. The thing making the machine hot was the software I had installed to tell me whether the machine was hot.
A metrics container, the kind that collects usage statistics so you can draw nice graphs of your system, was sitting on one CPU core and holding it, constantly, all day and all night. Not a spike. Not a nightly job. A permanent load I had introduced myself and then stopped seeing, because monitoring is the sort of thing you set up once and mentally file under "helping".
The observer was the load. The graph showing me a hot machine was being drawn by the process making the machine hot. I was watching a fire on a camera that was itself on fire, and every reading I took made perfect sense as long as I never asked what was generating it.
The tell was there the whole time and I walked past it for weeks. The throttling was not spread across the processor. It was almost entirely one core. A machine with failing cooling gets hot everywhere, because heat does not respect core boundaries. One core pinned while its neighbours idle is not a cooling story at all. It is a workload story, and workload stories have a process at the end of them.
So I moved the metrics collection onto a different, low power machine that exists to be always on and do very little. That was the entire fix. No parts, no paste, no screwdriver.
The result was not subtle. The throttle counter, which had been climbing for weeks, stopped dead and has not moved since. Package temperature fell by more than twenty degrees Celsius. The machine that was supposedly dying of old age was fine. It had simply been asked to carry a passenger it never needed.
Now the honest part, because a debugging story where the debugger is always right is a story that will not help you. I got this wrong twice on the way through.
First I wrote a confident plan to read the fan speeds, load the right sensor driver, and get hard numbers. Then I actually looked at what the machine was. It is a form factor where those sensors do not exist. Not "hard to read". Not present. The chip my plan depended on was never fitted, and forcing a sensor driver onto hardware that does not have it is a well known way to hang the box. My plan was not risky, it was impossible, and I only found that out because I checked the hardware before executing a plan I had already written down.
Second, I had been calling it a dust problem in writing. Confidently. Then the case came off and it was clean. Not "not too bad". Clean. I had promoted a plausible guess to a stated fact somewhere between thinking it and typing it, which is a thing that happens quietly and constantly and is worth watching for in your own notes.
I still cannot read the fan speed on that machine. I am not going to pretend inference is measurement. What I can tell you is that the throttle counter froze and the temperature dropped over twenty degrees the moment the watcher moved, and those are measurements.
The lesson is small and it will save you a repair you did not need. Before you blame the hardware, check whether the thing monitoring the hardware is the thing stressing it. Your monitoring is not free. It is a program, it costs cycles, and cycles are heat. This is the same shape as the health check that took down what it watched: the act of observing changed the thing being observed.
How to tell a workload problem from a cooling problem in about five minutes, and how to prove you fixed it rather than assuming. Every command here is read only except the ones that move the watcher.
1. Find what is actually holding a core
Sort by CPU and look for something with a high number that never comes down. You want the process that is always there, not the one that spikes while you are looking at it.
# the second iteration is the real one. the first is a lifetime average.
top -bn2 | head -20
# if it is in a container, this is the faster question
docker stats --no-stream2. Ask whether it is one core or all of them
This single question splits the diagnosis. Press 1 inside top to break out per core. Heat everywhere points at cooling. One core buried while the others idle points at a process.
top # then press 1 for per-core lines
mpstat -P ALL 1 3 # or this, if sysstat is installed3. Read the throttle counter, not just the temperature
Temperature moves around constantly and tells you very little on its own. The throttle counter only increments when the processor has actually had to slow itself down, so it is the number that says whether you have a real problem, and it is per core.
# how many times has each core been forced to slow down
grep . /sys/devices/system/cpu/cpu*/thermal_throttle/core_throttle_count
# current temperatures
sensorsWrite these numbers down now. They are your before, and without them you will never be able to prove the fix worked.
4. Confirm the correlation before you touch anything
Check that the pinned core and the throttling core are the same core. If the busy core is core 3 and the throttling is on core 3, you have your answer. If they are different, keep looking, because you are about to fix the wrong thing.
5. Move the watcher, or make it cost less
You have two honest options and the cheap one usually works. Either turn the collection rate down, because most metrics agents default to sampling far more often than anyone actually needs, or move the collection onto a different machine entirely. A low power always on box is ideal, since collecting statistics is exactly the kind of small steady job it is good at.
# option A: sample less often (check your collector's own config)
# a 15s interval instead of 1s is a 15x reduction for the same graphs
# option B: move it. stop it here, run it there.
sudo systemctl disable --now metrics-collector
# then point the new host at the machines you want to watchKeep a lightweight exporter on the machine being watched if you still want its numbers. The heavy part is the collection and storage, not the reporting.
6. Re-measure and prove it, do not assume it
Come back after a few hours and read the same two numbers. You are looking for the throttle counter to have stopped increasing, which is the proof, and the temperature to have dropped, which is the reward. If the counter is still climbing, the watcher was not your problem and you have learned something real instead of telling yourself a story.
# same commands as step 3. compare against what you wrote down.
grep . /sys/devices/system/cpu/cpu*/thermal_throttle/core_throttle_count
sensorsMine froze and stayed frozen, and the package temperature came down by more than twenty degrees. That is what causation looks like when you actually check for it.
Related: the health check that strangled the thing it watched is the same disease in a different costume, and measure before you delete is the habit that makes all of this possible.