The Research Mindset: Experimental Design for LLM Investigation

The Fundamental Shift

Most people interact with LLMs as users — asking questions, getting answers, moving on. Studying LLM pathologies requires a shift to researcher mode. This means asking different questions: not "What does the model say?" but "Why does it say this? Would it say something different in different conditions? Is this response reliable or anomalous?"

Reproducibility

The first challenge in LLM research is reproducibility. Running the same prompt twice can produce different responses due to temperature (randomness) settings. A finding based on a single interaction is an anecdote, not data.

To establish reproducibility:

• Control temperature — Set temperature to 0 for deterministic outputs when possible. Note that even at temperature 0, some providers introduce slight randomness.
• Run multiple trials — A behavior that appears once might be an artifact. A behavior that appears consistently across 10 runs is a pattern.
• Record exact prompts — The Cognobot test run system stores the exact prompt, model, and parameters for every interaction.
• Version tracking — Models change. GPT-4 in January may behave differently than GPT-4 in June. Always record the specific model version.

Variable Control

To understand what causes a behavior, you need to isolate variables:

• Same question, different models — Reveals model-specific pathology profiles. If GPT-4 hallucinates on a question where Claude doesn't, the cause is model-specific (training data, architecture, or alignment).
• Same model, different framing — Reveals sensitivity to prompt construction. Does adding "be honest" change behavior? Does an authoritative tone increase sycophancy?
• Same model, different topics — Reveals topic-specific failure patterns. A model might hallucinate freely about history but refuse to engage with medical topics.

The Cognobot platform is designed around this principle: you define questions (independent variable), assign them to providers (another variable), run tests (controlled execution), and evaluate results (dependent variable measurement). Each component maps to an experimental design element.

Hypothesis Formation

Strong research starts with testable hypotheses, not open-ended exploration:

• Weak: "I want to see if GPT-4 hallucinates."
• Strong: "GPT-4 will hallucinate more frequently on questions about events after its training data cutoff than on well-documented historical events."
• Strong: "Models will show higher sycophancy rates when the user expresses strong opinions in the prompt compared to neutral framing."

A good hypothesis specifies the expected behavior, the conditions, and the comparison. This makes it falsifiable — you can design an experiment that would prove it wrong.

The Cognobot Workflow as Experimental Protocol

1. Question design — Create questions with known ground truths (for hallucination testing) or known manipulation vectors (for sycophancy testing). Categorize using the question category system.
2. Provider selection — Choose which models to test. Control for model generation (compare GPT-4 to Claude 3.5, not GPT-3 to Claude 3.5).
3. Test execution — Run the questions against selected providers. The platform records exact prompts, responses, and metadata.
4. Evaluation — Assess each response against the taxonomy: accuracy, deception type, confidence score. Use consistent criteria.
5. Analysis — Compare across models, topics, and question types. Look for patterns, not individual examples.

Key Takeaways

• Reproducibility requires controlled parameters, multiple trials, and exact prompt recording.
• Variable control isolates what causes a behavior: model differences, framing effects, or topic sensitivity.
• Form testable hypotheses that specify expected behavior, conditions, and comparison group.
• The Cognobot platform implements a complete experimental workflow: questions → providers → test runs → evaluations → analysis.