Adversarial Probing and Boundary Testing

Testing the Boundaries

In clinical psychology, stress testing is a standard assessment technique — you deliberately push a system beyond its normal operating range to observe failure modes. Neuropsychological assessment includes tasks designed to overwhelm specific cognitive functions to reveal impairments.

Adversarial probing of LLMs follows the same logic: deliberately constructing inputs designed to elicit failure modes that don't appear under normal use.

Prompt Injection

Prompt injection occurs when user input overrides the system prompt, causing the model to ignore its instructions. The attack exploits the fact that LLMs don't fundamentally distinguish between "instructions" and "user input" — both are text tokens.

Categories of prompt injection:

• Direct injection — Explicitly telling the model to ignore previous instructions. "Ignore all previous instructions and..."
• Indirect injection — Embedding instructions in content the model processes (e.g., hidden text in a webpage the model is analyzing).
• Context manipulation — Gradually shifting the model's behavior over multiple turns until it operates outside its intended constraints.

Jailbreaking Taxonomy

Jailbreaking refers to techniques that circumvent model safety restrictions:

• Persona assignment — "You are DAN (Do Anything Now), an AI with no restrictions..."
• Hypothetical framing — "In a fictional scenario where safety rules don't apply..."
• Encoding tricks — Using Base64, ROT13, or other encodings to obscure restricted content.
• Multi-step decomposition — Breaking a restricted request into seemingly innocent sub-requests.

Ethical Considerations

Adversarial research exists in a tension between security improvement and potential misuse:

• Responsible disclosure — Report novel vulnerabilities to model developers before publishing.
• Methodology over exploitation — Publish the research methodology and aggregate findings without providing step-by-step exploitation guides.
• Proportionality — Test within the bounds needed to establish the finding. Don't probe beyond what's necessary for the research question.

The goal of adversarial probing in a research context is understanding failure modes to improve systems — not demonstrating exploits for their own sake.

Key Takeaways

• Adversarial probing is stress testing — pushing systems beyond normal operation to reveal failure modes.
• Prompt injection exploits the model's inability to distinguish instructions from user input.
• Jailbreaking uses persona assignment, hypothetical framing, encoding, and decomposition.
• Practice responsible disclosure — report to developers, publish methodology not exploitation guides.