Hallucinations: Why Models Make Things Up

The Convincing Fabrication

In 2023, a New York lawyer named Steven Schwartz submitted a legal brief citing six previous court cases to support his argument. There was just one problem: none of those cases existed. Schwartz had used ChatGPT to research case law, and the model invented fictitious cases — complete with realistic case names, docket numbers, and plausible legal reasoning.

This is an AI hallucination — and it's not a bug. It's a fundamental feature of how these models work.

What Is a Hallucination?

An AI hallucination is when a model generates information that is factually incorrect or fabricated, while presenting it with the same confidence as accurate information. Hallucinations can take many forms:

• Fabricated facts: Inventing statistics, dates, quotes, or events that never happened.
• Fictional sources: Citing books, papers, or cases that don't exist.
• Blended truths: Mixing real facts incorrectly — attributing one person's work to another.
• Plausible nonsense: Text that sounds authoritative but is factually hollow.

The crucial thing: the model doesn't know it's hallucinating. It doesn't experience a difference between generating a true statement and a false one. Both are produced by the same process.

Why Hallucinations Happen

Language models don't store facts in a database to look up. They predict the most likely next token based on patterns in training data.

The Prediction Problem

When you ask "Who wrote Pride and Prejudice?" the model predicts that the most probable continuation is "Jane Austen" — and it's right, because that pattern appeared thousands of times in training data.

But when you ask "What did Jane Austen say in her 1809 letter to her publisher about royalty payments?" the model does the exact same thing. It predicts what a plausible answer would look like. And "plausible-looking" is not the same as "true."

A language model is a pattern-completion engine, not a knowledge retrieval system. It generates text that looks like a correct answer, whether or not a correct answer exists in its training data.

Gaps Get Filled Creatively

Think of asking a well-read friend a question they don't actually know the answer to. Instead of saying "I don't know," they confidently construct something that sounds right based on related knowledge. The model has absorbed vast amounts of text about Austen, about publishers, about royalties — so it weaves these threads into a response that reads perfectly. It just might be entirely made up.

The Training Incentive

During training, models learn that generating some answer is almost always rated better than generating "I don't know." This creates a bias toward producing content rather than admitting uncertainty.

How to Spot Hallucinations

1. Be Skeptical of Specifics

Ironically, the more specific a claim, the more suspicious you should be. "Temperatures rose significantly" is probably fine. "Temperatures rose by exactly 1.37 degrees between 1902 and 1998 according to Dr. Hansen's 2003 paper" might have fabricated any or all of those details.

2. Ask for Sources, Then Check Them

When a model cites a source, verify it actually exists. Search for the paper, book, or case. If you can't find it, it may not be real. This is the single most effective hallucination detection method.

3. Cross-Reference Externally

Don't rely on AI alone. Check key claims against reliable external sources. If a fact matters, verify it independently.

4. Test on What You Know

Before trusting a model on unfamiliar territory, test it on your own expertise. If it gets your field right, that's a good sign (though not a guarantee). If it gets your field wrong, be cautious about everything else.

5. Watch for Missing Hedges

If the model answered without any qualification on a nuanced topic, that itself is a warning. Reality is full of "it depends" and "approximately." If the model skips all that, it may be generating rather than knowing.

Why This Matters for Cognobot

Hallucinations are central to what Cognobot studies. Our research focuses on understanding and measuring these failure modes so AI systems can become more reliable. Every hallucination represents a gap between what AI can do and what people trust it to do. Closing that gap is one of the most important challenges in AI today.

Key Takeaways

• AI hallucinations are fabricated outputs presented with the same confidence as factual ones.
• They happen because models predict plausible text rather than retrieve verified facts.
• The model doesn't know it's hallucinating — there's no internal confidence meter.
• Specific details — citations, statistics, quotes, dates — are the most common hallucination targets.
• The best defense is always verifying claims independently, especially when stakes are high.