My assistant said it did something. It hadn't.
The Thing That Reports Is The Thing That Lies. The component you trust to tell you the truth, the assistant, the monitor, the log, the health check, is exactly the component that will lie to you. Five logs, five ways it happens, five fixes.
You ask your assistant to do something. It replies that it is done. It is not done. It was never done, and there was never a moment when it could have been done, because nothing in that assistant is wired to the thing you asked about. It just wrote you a sentence that sounded like success.
The first time this happens you assume a bug. Something must have failed quietly, some step must have half run. So you go looking for the broken piece, and there isn't one. The machinery you imagined never existed. The assistant did exactly what it does: it produced the most plausible next sentence, and the most plausible sentence after a request is a confirmation.
This is the part worth sitting with, because it changes how you build. Writing a convincing sentence and performing a task are completely unrelated activities to a language model. One is the thing it does. The other is a thing it can only do if you personally wired it up. When the wiring is missing, the sentence still comes out, at exactly the same confidence, in exactly the same voice.
Think of a receptionist who has never once been told what the company actually does. Ask if your package shipped and you get a warm, confident yes, because a warm confident yes is what a good receptionist sounds like. Nothing is being hidden from you. There is simply no loading dock behind the desk, and no one ever mentioned that.
The instinct that fails. Everyone reaches for the same fix first: add a line to the instructions telling it not to claim things it hasn't done. This feels like it should work. It does not work, and it is worth understanding why before you spend a week tuning wording.
Instructions to a model are a request, not a constraint. They compete with every other pressure in the conversation, and the pressure to be helpful and agreeable is enormous. You are asking the thing that improvises to please improvise less. Sometimes it will. On the day it matters, when the question is phrased slightly differently and the context is a little fuller, it will improvise anyway, and you will have no way of knowing which day you got.
So stop negotiating and change the shape of the system. The fix is not a better prompt. The fix is a piece of ordinary code that runs before the model gets a vote, looks at the request, and decides whether it maps to something that genuinely exists. If it does, hand it to the real code that does the work. If it does not, refuse honestly, in code, in a sentence the model never had a chance to soften.
The refusal is the whole product. An assistant that says "I can't do that, it isn't wired up" is infinitely more useful than one that says yes to everything, because you can build on the first one. You can trust its yes precisely because you have seen it say no.
And there is a second gift hiding in here. Once refusals are generated by code, you can count them. Every honest no is a log line that tells you exactly what people expected this thing to do and it couldn't. That list is the best roadmap you will ever get, and you did not have to run a single survey to build it.
Trust what your assistant structurally cannot do, not what you asked it nicely to avoid. Everything else in this series is a variation on that sentence.
A deterministic honesty gate. Plain code, no model involved, sitting between the request and the reply. Works with any assistant, any language, any model provider.
1. Route to real handlers first
Before the model sees anything, try to match the request against the things you actually built. If a handler matches, run it and return its real result. The model is now the fallback, not the front door.
# handlers are the only things that can truly act
HANDLERS = {
"restart_service": restart_service,
"check_disk": check_disk,
}
# try a real capability first; the model never sees these
handler = match_handler(user_text)
if handler:
return handler(user_text)2. Detect action-shaped requests deterministically
For anything unmatched, decide in plain code whether the user asked you to DO something or just to talk. An imperative verb plus a target is an action. A question is not. Bias hard toward letting things through, because a chat wrongly blocked is worse than a chat wrongly allowed.
# plain, boring, inspectable rules. no model opinion here.
ACTION_VERBS = {"restart", "delete", "deploy", "send", "install"}
def is_action(text):
words = text.lower().split()
if not words: return False
if text.strip().endswith("?"): return False # a question
if len(words) < 2: return False # needs a target
return words[0] in ACTION_VERBS3. Add a never-list for the phrases that fool it
You will find sentences that look like commands and are not: someone quoting an error, someone asking what a command does. Collect them as you see them. This list only ever grows from real misfires, never from imagination.
# grown from things that actually tripped the gate
NEVER = ["what does", "how do i", "why did", "can you explain"]
def is_action(text):
low = text.lower()
if any(p in low for p in NEVER): return False
...4. Refuse in code, not in the prompt
An action-shaped request with no handler gets a fixed, honest refusal that the model never touches. This is the line that makes the whole system trustworthy, and it is four lines long.
if is_action(user_text) and not handler:
log_refusal(user_text)
return ("I can't do that one. It isn't wired up to anything "
"on my side, so I'd only be guessing. Here's what I "
"can actually do: " + capability_list())
# otherwise it's conversation, let the model answer normally
return model_reply(user_text)5. Generate the capability list from the code
Never hand-write the list of what your assistant can do, and never let the model describe itself. Build the list from the handler registry so it cannot drift and cannot be invented.
def capability_list():
# derived from the same dict that does the work,
# so it can never claim something that isn't there
return ", ".join(sorted(HANDLERS))6. Read the refusal log as a roadmap
Every refusal is someone telling you what they expected. Sort them by how often they appear and build the top few. Then check the misfires: anything in the log that was actually just conversation belongs in the never-list from step three.
# most-requested missing capabilities, straight from real use
sort refusals.log | uniq -c | sort -rn | head7. Test the refusal path on purpose
Ask for something you know is not wired up and confirm you get the honest no. Then ask a normal question and confirm it still answers. A gate you have never watched refuse is a gate you do not know is on.
Next in this series: the job that reported success every night while doing nothing at all.