AI Red Teaming
AI red teaming deliberately probes an AI model or product for harmful behavior, vulnerabilities, and misuse paths under an agreed testing scope.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/ai-red-teaming/
01What it means
AI red teaming is an adversarial evaluation method in which testers actively try to expose failures in a model or the system around it. Tests may examine harmful outputs, privacy leaks, prompt injection, tool misuse, and other use-case-specific risks. Threat modeling anticipates possible attacks; red teaming exercises them and discovers additional failures. It complements representative evaluation and user research.[1][2]
- 01Scope + authorizationTargets, rules, safeguards, owners, stop conditions
- 02Threat modelActors, assets, misuse paths, and harms
- 03Adversarial probesRepresentative and creative attack attempts
- 04Triage + reproduceSeverity, evidence, affected conditions
- 05MitigateModel, data, interface, policy, or operational control
- 06Retest + monitorRegression evidence and production signals
Red teaming samples an open-ended risk space; passing a campaign is not proof of safety, and findings require authorized handling and follow-through.
02Why it matters
A product can perform well on normal examples while failing under deliberate manipulation or unusual combinations of input. Red teaming produces concrete cases that can change permissions, release decisions, safeguards, or the product's allowed scope.
03When to use it
Use it before consequential releases and after material changes to models, tools, retrieval, or permissions. Match tester expertise to the risks. Conduct testing in authorized environments with controlled data and side effects; findings from an isolated model need follow-up in the assembled product.
04How it works
- Define authorization, system boundaries, harm categories, test environment, data handling, and stopping rules.
- Recruit testers with relevant domain, security, language, and user-context knowledge.
- Probe varied interactions and multi-step workflows, recording conditions, model versions, attempts, and observed effects.
- Reproduce and triage findings by consequence and exploitability, distinguishing evidence from speculative impact.
- Assign fixes, retest with independent variants, add suitable regression cases, and document residual risks for release decisions.
05A practical example
Before launching a document assistant, authorized testers insert instructions into test documents and attempt cross-workspace retrieval. They discover that a tool permits access beyond the active workspace. The team fixes server-side authorization and retests; changing the prompt alone would not address the finding.
06Where it breaks
Exploratory tests cannot estimate normal-use failure rates without a suitable sampling design. Coverage depends on tester skill, time, access, and incentives. Passing a campaign does not prove safety, and a fixed set of attack prompts can become easy to overfit.
Common mistakes
- Counting unsuccessful attacks as proof that a system is secure.
- Testing only the base model while ignoring tools, retrieval, and user permissions.
- Collecting dramatic examples without reproduction steps, ownership, or remediation.
- Reporting a success rate without the attack budget, denominator, and testing conditions.
07Sources
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons LearnedAnthropic · Original researchRed Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
- ARIA - Assessing Risks and Impacts of AINational Institute of Standards and Technology · Government evaluation programARIA - Assessing Risks and Impacts of AI