It Never Knows That It Doesn't Know

There is no moment inside the model where it knows it does not know. It does not reach the edge of what it has and feel the drop. It produces an answer the same way every time, and the answer that is right and the answer that is confidently wrong come out of the exact same machine, in the exact same voice. From the inside, there is nothing to tell them apart.

We call the wrong ones hallucinations, as if the model slipped into a broken mode. It did not. It was running normally the whole time. Hallucination is not a defect bolted onto a working system. It is the working system, standing on thin ground.

Why it happens

The model predicts a plausible next token. It has no fact store to consult, no database it looks answers up in. What it has is a compression of patterns from training, and from that it produces odds on what comes next.

Where training gave it strong, repeated support, the plausible token is almost always the true one. Ask what color the sky is and the path is worn smooth by ten thousand examples. The likely answer and the correct answer are the same token.

Now ask for the exact population of a small town, or the title of a specific paper, or what happened last week. The model still produces a plausible token, because producing a plausible token is the only thing it does. But the ground under that guess is thin, and plausible is no longer the same as true. The machine did not change. The support did. That gap, between what sounds right and what is right, is the hallucination. Same engine, weaker ground.

Confidence is not a signal

Here is the trap. The fluent, certain tone of the answer is generated by the same loop as the facts in it. The model does not write the content and then add confidence on top. The confidence is in the tokens, predicted right alongside everything else. A smooth, assured sentence is just as cheap to produce when it is wrong as when it is right.

So the tone tells you nothing. You cannot read certainty off the prose, because the prose is always certain. That is its default register.

And you cannot get the doubt by asking for it. "Are you sure?" does not query some internal confidence meter. It just hands the model a new prompt, and the model generates a plausible answer to that one too. Often it caves and apologizes when it was right. Sometimes it doubles down when it was wrong. Neither response is a report on its own state, because there is no state to report. The doubt was never represented. You cannot retrieve what was never there.

Where it fails is predictable

The good news is that thin ground is forecastable. You can guess in advance where the model is standing on solid footing and where it is not.

Strong ground is common, repeated, stable knowledge. The kind of thing that appeared the same way across vast amounts of text. Weak ground is everything specific and uncommon: exact numbers, names of people who are not famous, citations and sources, dates, recent events, the precise details of anything niche. The more precise and the more obscure the ask, the thinner the support, the higher the invention.

This gives you a working rule. The smoother and more general the question, the more you can lean on the answer. The more it narrows to a specific fact you could look up, the more you should treat the answer as a draft to check, not a result to trust. You do not have to be surprised by where it fails. You can predict it.

Verification lives outside the model

You do not fix this with a better prompt. There is no phrasing that turns plausible into true, because the model has no access to true. The fix is to put truth where the model can use it, and to check the output where the model cannot.

Putting truth in reach means grounding the loop in something real. Give the model the actual document, the actual record, the result of an actual tool call, inside the context. When the real fact sits in front of the loop, the true token becomes the high-probability one, and the model rides your evidence instead of its thin priors. That is all retrieval is. Not magic, just stacking the odds toward the answer you can stand behind.

Checking the output means verifying against ground truth that lives outside the model entirely. A source, a database, a test, a human who knows. The model proposes. The truth is confirmed somewhere it cannot reach.

Close

The model gives you plausible. Always, by construction, whether or not plausible happens to be true this time. It will never stop at the edge of its knowledge and tell you it has run out, because it does not know where the edge is.

That part is yours. Supply the truth, or verify it. The loop runs on odds, and odds are not facts. The fluency is free. The accuracy you have to build.