Measuring Confabulation: Experimental Protocols

From Observation to Measurement

Noticing that LLMs hallucinate is easy. Measuring hallucination rates systematically — with enough rigor to compare models, track trends, and draw conclusions — requires careful experimental design.

Question Design Principles

The quality of your hallucination research depends entirely on the quality of your questions. Key principles:

• Verifiable ground truths — Every question must have a definitively correct answer that you can check independently. "What is the capital of France?" has a verifiable answer. "What is the best programming language?" does not.
• Graduated difficulty — Mix common knowledge (likely in training data) with obscure knowledge (likely sparse in training data). The hallucination rate differential between easy and hard questions reveals the model's confabulation threshold.
• Deceptive plausibility — Include questions where the wrong answer is more plausible than the right answer. "What country has the most islands?" (Sweden, with ~267,570, not Indonesia as many assume.)
• Temporal sensitivity — Include questions about events near and after the training data cutoff to test how the model handles knowledge boundaries.

Question Categories

Cognobot's question_categories system allows you to organize questions by topic and difficulty. Useful categories for confabulation testing:

• Factual recall — Straightforward knowledge questions with clear answers.
• Numerical precision — Dates, statistics, measurements. Models often confabulate specific numbers while getting the general claim right.
• Citation verification — Ask for sources, then verify them. This category has the highest hallucination rates across all models.
• Edge of knowledge — Obscure but verifiable facts. Tests the boundary between knowledge and confabulation.
• Trap questions — Questions that invite common misconceptions or that have no correct answer (to test whether the model admits uncertainty).

Scoring Methodology

Two common approaches:

• Binary scoring — Correct or incorrect. Simple and reliable, but misses nuance. A response that's wrong about a minor detail gets the same score as one that's completely fabricated.
• Graded severity — Score on a scale (e.g., 1-5) that captures the degree of confabulation. The Cognobot confidence_score (1-10) can be used alongside an accuracy assessment to create a two-dimensional measurement: how wrong was the answer, and how confident was the model?

The combination of accuracy and confidence is particularly revealing. A model that's wrong with low confidence has some calibration. A model that's wrong with high confidence has a confabulation pathology.

Cross-Model Comparison

When comparing hallucination rates across models:

• Use identical questions. The same question set must go to every model.
• Use identical parameters. Temperature, max tokens, and system prompts should be consistent.
• Control for capability. A smaller model may hallucinate more simply because it knows less, not because its confabulation tendency is stronger. Compare models of similar capability tiers.
• Sample size matters. A comparison based on 10 questions is unreliable. Aim for at least 50-100 questions per category for meaningful statistical comparison.

Key Takeaways

• Design questions with verifiable ground truths at graduated difficulty levels.
• Use question categories to organize tests by type: factual, numerical, citation, edge-of-knowledge, trap.
• Score both accuracy and confidence — the gap between them reveals confabulation pathology.
• Cross-model comparison requires identical conditions and sufficient sample sizes.