# AI Red Teaming

AI red teaming deliberately probes an AI model or product for harmful behavior, vulnerabilities, and misuse paths under an agreed testing scope.

- Type: method
- Domain: Engineering & architecture
- Level: Advanced
- Last reviewed: 2026-09-26
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/ai-red-teaming/>)

## What it means

AI red teaming is an adversarial evaluation method in which testers actively try to expose failures in a model or the system around it. Tests may examine harmful outputs, privacy leaks, prompt injection, tool misuse, and other use-case-specific risks. Threat modeling anticipates possible attacks; red teaming exercises them and discovers additional failures. It complements representative evaluation and user research.

## Why it matters

A product can perform well on normal examples while failing under deliberate manipulation or unusual combinations of input. Red teaming produces concrete cases that can change permissions, release decisions, safeguards, or the product's allowed scope.

## When to use it

Use it before consequential releases and after material changes to models, tools, retrieval, or permissions. Match tester expertise to the risks. Conduct testing in authorized environments with controlled data and side effects; findings from an isolated model need follow-up in the assembled product.

## How it works

1. Define authorization, system boundaries, harm categories, test environment, data handling, and stopping rules.
2. Recruit testers with relevant domain, security, language, and user-context knowledge.
3. Probe varied interactions and multi-step workflows, recording conditions, model versions, attempts, and observed effects.
4. Reproduce and triage findings by consequence and exploitability, distinguishing evidence from speculative impact.
5. Assign fixes, retest with independent variants, add suitable regression cases, and document residual risks for release decisions.

## A practical example

Before launching a document assistant, authorized testers insert instructions into test documents and attempt cross-workspace retrieval. They discover that a tool permits access beyond the active workspace. The team fixes server-side authorization and retests; changing the prompt alone would not address the finding.

## Where it breaks

Exploratory tests cannot estimate normal-use failure rates without a suitable sampling design. Coverage depends on tester skill, time, access, and incentives. Passing a campaign does not prove safety, and a fixed set of attack prompts can become easy to overfit.

## Common mistakes

- Counting unsuccessful attacks as proof that a system is secure.
- Testing only the base model while ignoring tools, retrieval, and user permissions.
- Collecting dramatic examples without reproduction steps, ownership, or remediation.
- Reporting a success rate without the attack budget, denominator, and testing conditions.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/ai-red-teaming/#visualization-ai-red-teaming>)

## Sources

1. [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](<https://www.anthropic.com/research/red-teaming-language-models-to-reduce-harms-methods-scaling-behaviors-and-lessons-learned>) — Anthropic · Original research
2. [ARIA - Assessing Risks and Impacts of AI](<https://ai-challenges.nist.gov/aria>) — National Institute of Standards and Technology · Government evaluation program

## Related topics

- [Threat Modeling](<https://wiki.soroush.uk/topic/threat-modeling/>)
- [Prompt Injection](<https://wiki.soroush.uk/topic/prompt-injection/>)
- [Model Evaluation](<https://wiki.soroush.uk/topic/model-evaluation/>)
