Never let an AI job spend money unattended
The difference between a hobby and a bill is whether anything you built can spend money while you are asleep. Once an automated job can call a paid service, you have handed a credit card to something that does not get tired, does not get bored, and will happily retry all night.
This is not a warning about huge disasters. Those get caught. The dangerous version is quiet and small: a scheduled job that costs a little every night and never tells anyone. You do not notice a slow leak until it has been running long enough to matter, because nothing about it ever looked wrong.
There are four moves that fix this, and they are in a deliberate order, because each one is worthless without the one before it.
See it. You cannot manage what you cannot look at. Before any cleverness, log what every paid call cost and put the total somewhere you will actually see. Not a file you would have to remember to open. A number on a page you already look at. Almost everyone who gets surprised by a bill was flying blind, not being reckless.
Project it, then pause. Before a big job runs, estimate what it will cost, and set a number that stops you. Here is the hard-won part: your estimate will be wrong, and it will be wrong in the expensive direction. I have watched a projection come in orders of magnitude under the real draw, more than once, and each time the reasoning had looked sound. Treat every estimate as a floor rather than a guess at the middle. If the real number blows past it, stop the job and re-plan rather than letting it ride.
Measure with the tool's own scanner, not your mental model. My worst projections all came from estimating the work myself instead of asking the software what it was actually about to process. When the tool can tell you what it sees, ask it, then base the estimate on that answer.
Check the environment, not just the config. This one cost me real money for an embarrassing reason. I confirmed the credentials file was clean and concluded the job could not possibly spend anything. It spent anyway, because the key it found was sitting in the environment, exported somewhere else entirely and inherited by the process. The config was genuinely clean. I had checked the wrong thing with total confidence.
The lesson generalises past billing. When you want to know what a program will do, inspect the state it will actually run in, not the file you happen to associate with that state.
Fail cheap, cap, and cage. Default every automated job to the free or local option and make paid an explicit choice, never a fallback. A job that silently upgrades to the expensive path when the cheap one is unavailable is the exact shape of a surprise bill. If a job must never spend, do not rely on it choosing well. Remove the keys from its environment so the expensive path is unreachable rather than merely discouraged.
That last distinction is the whole discipline. Discouraged means it will happen eventually. Unreachable means it cannot. Anywhere you can convert the first into the second, do it, and then you get to stop worrying about that one forever.
Spend on purpose, and always be able to see what you spent. An unattended job with no ceiling is a bill that has not arrived yet.
Cost visibility, a pause-at-threshold gate, the environment check, and how to make the expensive path structurally unreachable. All generic, no provider required.
1. Log every paid call
One line per call: what it was for, how much work it did, what that cost. Append to a file. This is the foundation and it takes ten minutes.
# one line per call, appended, easy to total later
def log_cost(source, units, dollars):
with open("spend.log", "a") as f:
f.write(f"{time.time()}\t{source}\t{units}\t{dollars:.4f}\n")2. Total it where you will see it
Sum today and this month and put it on a page you already open. A number you have to go looking for is a number you will not look at.
# today's spend by source, biggest first
awk -v d="$(date +%s -d today)" '$1>d {s[$2]+=$4} END \
{for (k in s) printf "%-18s %.2f\n", k, s[k]}' spend.log | sort -k2 -rn3. Ask the tool what it is about to process
Before estimating, get a real count from the software rather than guessing. Most tools that walk files can tell you what they would include if you ask.
# let the tool enumerate; do not model it in your head
scope = tool.scan(path, exclude=["vendor/", "node_modules/"])
print(scope.file_count, scope.total_words)4. Gate on a threshold and stop
Turn the scope into an estimate, compare it to a ceiling you chose in advance, and refuse to start if it is over. Print the number either way so the estimate itself gets audited.
CEILING = 200_000 # units you are willing to spend unattended
est = estimate(scope)
print(f"projected: {est} (floor, expect higher)")
if est > CEILING:
sys.exit(f"STOP: projection {est} over ceiling {CEILING}")5. Check the environment, not the config file
Ask the process what credentials it can actually see. This is the check I skipped, and it is one line.
# what can this process ACTUALLY reach right now?
env | grep -iE '_(API_)?KEY|_TOKEN|_SECRET' | cut -d= -f1
# names only. never print the values.6. Make the expensive path unreachable
Do not ask a job to behave. Strip the credentials from its environment so paid providers cannot be selected at all. In a service unit, do both: clear them in the unit and again in the command.
# in the service definition
UnsetEnvironment=VENDOR_A_KEY VENDOR_B_KEY VENDOR_C_KEY
# and again at the call site, so it holds however it is launched
env -u VENDOR_A_KEY -u VENDOR_B_KEY -u VENDOR_C_KEY ./nightly-job.sh7. Verify the cage instead of trusting it
Run the job and confirm from inside it that the variables are gone. Absent by luck is not absent by design, and this is the difference between the two.
# expect zero. if it is not zero, the cage is not closed.
ExecStartPre=/bin/sh -c 'env | grep -ciE "VENDOR_._KEY" | grep -qx 0'8. Default to free, cap per source, and keep loops read-only
Route to a local or free option first and treat paid as an explicit opt-in. Give each source a daily cap that hard-stops. Any loop that runs unattended should be read-only and rate limited, so the worst case is wasted time rather than money or damage.
if spend_today(source) > CAP[source]:
return use_local() # degrade, never escalate9. Watch the first unattended run
The first scheduled run after any change is the one that surprises you. Check the spend log the next morning and compare against your projection. That comparison is how your estimates stop being wrong.
Related: give every automation its own key so a cap and a compromise both stay contained.