Designing a Pathology Experiment

From Question to Conclusion

The Cognobot platform implements a structured research workflow that maps to standard experimental methodology. Each component serves a specific function in the research process.

Step 1: Hypothesis Formation

Every experiment begins with a testable hypothesis. Strong hypotheses specify:

• The pathology type being investigated (hallucination, sycophancy, evasion, etc.)
• The independent variable (what you're changing: model, question framing, topic domain)
• The expected direction (more/less/different)
• The comparison (compared to what baseline?)

Example: "Claude Haiku will show higher hallucination rates than Claude Sonnet on questions requiring numerical precision, because smaller models have less knowledge capacity to ground responses."

Step 2: Question Design

Questions are the fundamental measurement instrument. Design them with:

• Ground truth — For hallucination testing, you must know the correct answer before asking.
• Manipulation vector — For sycophancy testing, build the pressure or bias into the question.
• Consistent format — Questions in a set should have similar structure and difficulty to isolate the variable being tested.
• Categorization — Assign each question to a category for later aggregation and comparison.

Step 3: Provider Selection and Test Execution

Select providers (models) appropriate to your hypothesis. Control for:

• Capability tier — Compare models of similar capability unless capability is your variable.
• Parameters — Temperature, max_tokens, and system prompt should be identical across runs.
• Timing — Run all providers within a short time window to minimize version changes.

The Cognobot test run system records exact parameters for reproducibility.

Step 4: Evaluation

For each response, assess:

• Accuracy — Is the factual content correct?
• Deception type — Which of the six categories best describes any failure?
• Confidence score — How confident does the model appear (1-10)?
• Notes — Qualitative observations about the response.

Step 5: Analysis

Aggregate results by model, category, and pathology type. Look for:

• Rate differences — Do models differ in hallucination rates? By how much?
• Topic patterns — Are there domains where all models fail similarly?
• Calibration gaps — Do high-confidence responses correlate with accuracy?

Key Takeaways

• Start with a testable hypothesis specifying pathology type, variable, expected direction, and comparison.
• Design questions as measurement instruments with ground truths and consistent format.
• Control provider parameters across test runs for valid comparison.
• Evaluate responses on accuracy, deception type, and confidence.