I stopped renting my own thinking
What you will learn
- Why a paid AI assistant runs on a computer no more magical than yours
- The three reasons to run one at home, only one of which is money
- Exactly what a local one is not as good at, said plainly
Here is the thing nobody selling you a monthly subscription wants you to sit with. The assistant you pay for every month runs on a computer. Not a magic cloud. A computer. And you probably already own one that can do a good chunk of the same work, sitting in a cupboard or under a desk, doing almost nothing.
So I stopped renting my own thinking back to people who charge me for access to it. I put one of these models on my own machine. This is what that actually looked like, without the sales pitch.
A model is the file that does the thinking. It is large, it is just a file, and once you have it on your computer it works whether or not you are connected to anything.
Why bother
Three reasons, and none of them is really about saving a few pounds a month.
Privacy. When it runs on your machine, your questions never leave your machine. Nobody records them, nobody learns from them, nobody hands them to anyone who asks.
Control. It does not change underneath you. It does not quietly get worse to protect somebody's profit margin. It does not disappear because a policy shifted.
And the real one: it is yours. You stop being a tenant.
What it actually takes
Less than the advertising implies, and more than the enthusiasts admit.
You need a machine with a decent amount of memory and, ideally, a graphics card that was already gathering dust. You need a small program that loads the model and answers questions. And you need the model file itself. That is the entire shopping list, and you probably already have most of it.
The parts used here were neither new nor expensive. That is rather the point. It is closer to a junk drawer that happens to run a language model.
A model running at home is not the biggest one money can rent, and pretending otherwise is exactly how people end up disappointed.
It is smaller. It takes a moment longer to wake up. It will not know something that happened last week unless you tell it. What it is, is enough. Enough to draft, to summarise, to answer, to handle the small relentless jobs that fill a day, on a machine that owes nothing to anybody.
What broke
Plenty. Drivers that half installed. A model that loaded and then said nothing, because one setting was wrong. Fans that would not spin. A network address that moved and took everything with it.
Every one of those became a later guide, because that is what this place is for. Not the clean demonstration. The actual mess, and the actual fix.
You do not need the most powerful model on earth. You need one that answers to you, and stays answering to you.
A local assistant is three things: something to run it, a model, and something to talk to it with. This works on a desktop with a decent graphics card, and a smaller model runs on a very small computer with no graphics card at all, just more slowly.
Replace anything in angle brackets with your own value.
1. Install the program that runs the model
What this does: checks that the runner installed correctly and lists any models you already have.
What should happen: the first command prints a version number. The second prints an empty list, which is exactly right at this stage.
Which one: Ollama is the easiest, and runs on Windows, Mac and Linux. Install it from its own website first.
ollama --version
ollama list # empty for now, that is fine2. Download a model that fits your memory
What this does: downloads a model onto your machine.
The one number that matters: the model has to fit in your memory. A rough rule is to keep the download size below the memory you have free. On a machine with 16 gigabytes, a mid-sized model is a good first choice.
What to check: run the second command and look at the size. A download that stopped early is the most common cause of the next problem.
ollama pull <model>
ollama list # confirm it downloaded fully, check the size3. Talk to it
What this does: starts a conversation in your terminal window.
What should happen: you type something and it answers. Hold
Ctrl and press D to leave.
If it loads but says nothing: the download was almost certainly incomplete. Download it again and check the size.
ollama run <model>4. Point your other tools at it
What this is: two settings, not commands. Most programs that can talk to a paid AI service can be pointed at your own machine instead, because the runner speaks the same language.
About the key: a paid service needs a secret key. Yours does not, because it is on your own machine. Put any text at all in that box.
# the address of the runner on your own machine
http://127.0.0.1:<port>/v1
# the key: any non-empty text. it is local, so it is ignored.
<anything>5. Keep it on your machine
By default the runner only answers requests from the same computer, which is exactly what you want. Leave it that way.
If you genuinely need to reach it from another room, do not open a door in your home network. Use the same approach described in the guide about hosting a website from a spare room, so nothing is ever exposed from the outside.
This page in the original Kyber Cypher voice: I stopped renting my brain