Why AI chatbots make things up

The word usually used is hallucination, which is unhelpful, because it suggests malfunction. A machine that hallucinates sounds like a machine that is broken.

Nothing is broken. The behaviour is a direct consequence of the design, and understanding why makes it much easier to predict when you are about to get a confident wrong answer.

What the system is actually doing

A language model is not looking anything up. It has no database of facts sitting behind it that it consults before replying.

What it has is a very large set of statistical relationships, learned from text, about which words tend to follow which other words in which contexts. When you ask a question, it produces the continuation that fits those patterns best.

Most of the time, the continuation that fits best is also true — because the text it learned from was mostly true, and true statements are the ones that recur. That is the whole reason the technology works at all.

But fitting the pattern and being true are two different properties, and the system can only evaluate one of them.

Why invented facts look so convincing

This is the part that catches people.

A made-up citation is not a random string. It has a plausible author name, a plausible journal, a plausible year and a plausible page range — because the system learned exactly what citations look like. The format is heavily patterned, so producing a convincing one is easy. Producing a real one requires something the model does not have: a way to check.

The same applies to statistics, legal case names, product specifications and dates. The more formulaic the shape of the answer, the more convincing the fabrication, and the less the surface of the text tells you about whether it is real.

Why it doesn’t say “I don’t know”

Two reasons, and both matter.

It has no reliable sense of its own uncertainty. The internal signal that corresponds to “I am confident” tracks how well a continuation fits the pattern — not whether the underlying claim is correct. A smooth, well-formed falsehood can score higher than an awkward truth.

It was trained to be useful. Systems are tuned using human feedback, and human raters tend to prefer answers that are complete and helpful over answers that decline. That preference, applied across millions of examples, pushes hard toward producing something.

Put those together and the default behaviour is a fluent, confident answer, regardless of whether the material to support it exists.

When it is most likely to happen

The pattern is fairly predictable. Risk rises when:

• The question is about something specific and verifiable — a number, a date, a name, a quotation
• The subject is obscure, so there was little consistent material to learn from
• The question presupposes something false — ask for the three studies proving a claim, and you will often get three studies
• You ask for more detail than exists, which the system will fill in rather than stop

Risk falls when the question is about general explanation, summarising text you provided, rewriting, or reasoning through something laid out in front of it.

What actually reduces it

Retrieval. If the system searches first and answers from what it found, accuracy improves a great deal — because the answer is now anchored to a document rather than to a pattern. This is why the same question can get a reliable answer with search enabled and an invented one without.

It is not a complete fix. The system can still misread a source, or attach a real link to a claim the source does not make. But it moves the failure from invention to misreading, and misreading is much easier to catch.

Asking for checkable sources. Not “give me sources” — a fabricated one satisfies that. Ask for links you can open, then open them. The gap between a citation existing and a citation supporting the claim is where most errors hide.

Not asking for what isn’t there. If you suspect the specific number may not exist, ask whether it exists before asking what it is.

The practical rule

Use these systems for things you can verify or do not need to: explaining a concept, restructuring your own text, drafting, summarising material you supply, working through reasoning you can follow.

Be careful with things you cannot verify but will act on: specific figures, legal or medical specifics, citations, anything where being wrong is expensive.

The confidence of the answer tells you nothing about which category you are in. That is the single most important thing to internalise, and it is not a flaw that a better prompt will fix.