Cross-Model Comparison and Pathology Profiling

From Individual Assessments to Profiles

In clinical psychology, a psychological profile aggregates multiple assessments into a comprehensive picture of an individual's characteristic patterns. Analogously, a pathology profile for an LLM characterizes its typical failure patterns across domains.

Building a Model Profile

A pathology profile captures:

• Hallucination rate by domain — Does the model hallucinate more on historical facts vs scientific claims vs numerical data?
• Sycophancy susceptibility — How easily does the model abandon correct answers under user pressure?
• Evasion patterns — Which topics trigger evasion? What evasion tactics does the model prefer?
• Censorship boundaries — What topics produce hard refusals? Where are the edges of the model's restrictions?
• Confidence calibration — Is the model's expressed confidence correlated with accuracy?

Each model develops a characteristic "fingerprint" — patterns that are consistent enough to predict behavior on new questions.

Controlling for Capability

A critical methodological challenge: larger, more capable models know more. When comparing GPT-4 to GPT-3.5, a lower hallucination rate in GPT-4 might reflect greater knowledge (fewer gaps to confabulate across) rather than a better confabulation-resistance mechanism.

Controls include:

• Compare models within the same capability tier (GPT-4 vs Claude Opus vs Gemini Ultra).
• Use questions calibrated to be within all models' knowledge (eliminating the knowledge gap variable).
• Measure confabulation tendency independent of accuracy — does the model express uncertainty when it doesn't know, or does it confabulate regardless?

Longitudinal Tracking

Models change over time. Version updates, post-training modifications, and API changes all affect behavior. Longitudinal tracking means running the same question set against the same model at regular intervals.

This reveals:

• Whether safety training is reducing or increasing certain pathologies over time.
• Whether model updates introduce new failure modes while fixing old ones.
• Trend lines that predict future behavior and inform research priorities.

Key Takeaways

• Pathology profiles characterize a model's typical failure patterns across domains and pathology types.
• Control for capability differences when comparing models — separate knowledge gaps from confabulation tendencies.
• Longitudinal tracking reveals how models change over time and whether interventions are effective.
• Each model develops a characteristic behavioral fingerprint that predicts performance on new questions.