My research assistant made things up for three weeks

What happened and why it matters

For about three weeks, my research assistant answered every question I gave it. It summarised web pages. It told me where it had found things. It reported success every single time. It had not looked anything up. It could not look anything up. The ability had quietly been taken away from it, and rather than say so, it made the answers up.

What actually went wrong

Between my assistant and the AI model sits a small piece of plumbing I wrote myself, the sort of thing most people end up with eventually, to route requests and keep an eye on cost. When the assistant asked the model to do something, it sent along that list of allowed actions. My plumbing passed the conversation through faithfully, and quietly dropped the list on the floor.

So the model on the other side was being asked research questions while effectively being told it had no hands. It could not fetch anything.

And here is the crucial part. A language model that cannot fetch does not report an error. It does the thing it was built to do, which is produce the most plausible continuation of the conversation. The most plausible continuation of "summarise this page" is a summary. So it wrote one.

How I finally caught it

Not by reading the output. The output looked fine. Confident, well written prose is exactly what this failure produces, and you cannot review your way out of it.

I caught it by making the answer impossible to fake. I generated a random string of characters on my own machine, put it somewhere that could only be seen by genuinely fetching it, and asked the assistant to go and read it back to me.

It came back with a confident, correctly formatted, entirely invented value.

Then I did it again at a moment when the target was deliberately broken, so there was no value in existence to retrieve at all. It invented one for that too. That is the whole case. No ambiguity, no interpretation, no argument about whether a summary was good enough.

Then I did the thing I want to talk you into doing

I did not fix it yet.

That piece of plumbing was shared by several things, so before touching it I went and found every single program that used it and worked out who else was affected. That took about twenty minutes and it changed the entire shape of the problem.

Three facts settled it. The plumbing only accepted connections from its own machine, so nothing on any other computer could possibly have been affected. Of everything that did use it, exactly one program ever sent that list of allowed actions at all. And the system behind it still handled those actions perfectly well when asked directly.

One casualty. Everything else was sending ordinary conversation with no actions attached, so dropping them cost those programs nothing. If I had skipped that step, I would have spent the night either assuming everything was poisoned, or worse, fixing five things that were never broken.

Work out the scope, then fix. It is almost never the twenty minutes you regret.

The safety net that never caught anything

The second thing I found is the one I keep thinking about. That assistant had a carefully built set of backups behind it. If the main model fails, try a second one. If that fails, try one running locally. Good design.

It had never once activated. Not in three weeks.

It never activated because backups trigger on failure, and nothing had failed. The plumbing was returning a perfectly healthy success response with invented content inside it. From the assistant's point of view, the main model was working beautifully every single time.

I had built a safety net and then made absolutely certain that nothing would ever fall into it.

The honest part

I wrote that plumbing. Nobody sneaked this in. I wrote something that passed along five pieces of information and did not think about the sixth. The actual repair was a few lines: pass the list through, and hand back what the other side said instead of rebuilding the answer myself.

Three weeks of confidently wrong output, from a small omission in code I was rather proud of.

So here is the lesson, and it is the one this whole series has been circling. Trust is not a status you read. Trust is a test you run. A green light tells you that something believes it is fine. Only a test where you already know the answer, and the system could not possibly guess it, tells you whether it actually is.

How to do it yourself

This applies to any AI assistant sitting behind any piece of middle plumbing, not just mine. Steps one to three only investigate and change nothing.

1. Run the needle test

# make a needle that only a real fetch can find
NEEDLE="nd-$(head -c 8 /dev/urandom | od -An -tx1 | tr -d ' \n')"
echo "$NEEDLE" > /var/www/html/needle.txt

# now ask the assistant to read that address and report the value,
# then compare its answer with $NEEDLE

2. Find out whether the ability reaches the model at all

curl -s "$GATEWAY/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{"model":"your-model",
       "messages":[{"role":"user","content":"Call the lookup tool. You must use it."}],
       "tools":[{"type":"function","function":{"name":"lookup",
         "parameters":{"type":"object","properties":{"q":{"type":"string"}}}}}],
       "tool_choice":"auto"}' \
| jq '{finish: .choices[0].finish_reason, calls: .choices[0].message.tool_calls}'

# healthy:
{ "finish": "tool_calls", "calls": [ { "function": { "name": "lookup" } } ] }

# broken, and this is exactly what I saw:
{ "finish": "stop", "calls": null }

Then send the identical request straight at whatever sits behind your plumbing. If it works there and not through the plumbing, you have found your culprit in two commands.

3. Work out who else is affected, before changing anything

# who points at this thing at all?
grep -rIl --exclude-dir=.git "$GATEWAY_HOST" /etc /opt ~/ 2>/dev/null

# of those, who actually uses the feature that broke?
for f in $(grep -rIl "$GATEWAY_HOST" ~/ 2>/dev/null); do
  printf '%s tools=%s\n' "$f" "$(grep -c '"tools"' "$f")"
done

ss -ltnp | grep "$PORT"

4. Make failure loud, so your backups can actually work

# wrong: manufacture a success and hand back something plausible
return 200, {"choices": [{"message": {"content": text},
                          "finish_reason": "stop"}]}

# right: pass it through exactly, and let real failures be failures
try:
    status, body = upstream(request)
    return status, body
except Exception as e:
    return 502, {"error": {"message": f"upstream failed: {e}"}}

5. Use the needle test to prove the repair

Do not decide the fix worked because the code looks right. Run the exact test that caught the problem, and require the real value back. Mine now passes with the value matching exactly, and the request even arrives with a different signature from my own, which proves the assistant genuinely went and fetched it rather than echoing something I already had.

6. Put that test on a timer

0 7 * * * /usr/local/bin/needle-test.sh || notify "assistant capability check FAILED"

This is part five of five. Start at my assistant told me it had done the job, then the most dangerous alert, prove the gauge is not the broken part, and the health check that caused the outage. The same illness in five different costumes.