Prompt Injection
Prompt injection is an attack in which untrusted input influences a model to disregard intended instructions, expose information, or misuse connected capabilities.
Download Markdown (.md)Soroush’s Product Wiki · https://wiki.soroush.uk/topic/prompt-injection/
01What it means
Prompt injection exploits the fact that language-model applications process instructions and data in a shared representation without a dependable security boundary between them. A direct injection comes from a user; an indirect injection arrives through content the system reads, such as a webpage, email, document, image, or tool result. Jailbreaking is related but usually focuses on bypassing model restrictions rather than compromising an application's trusted workflow.[1][2]
- 01Untrusted sourceUser, webpage, email, document, tool result
- 02Shared model contextInstructions and content coexist
- 03Manipulated proposalDisclosure, tool call, downstream text
- 04Deterministic controlsAuthorize, validate, confirm, constrain
- 05Contained outcomeLeast privilege, monitoring, recovery
Separating or labeling instructions helps interpretation but is not a dependable security boundary; deterministic controls must protect actions and data.
02Why it matters
When a model can retrieve private data or call tools, manipulated output can become data exposure, unauthorized action, or corrupted decisions. Prompt wording alone cannot eliminate the weakness, so product scope and deterministic controls must limit the possible impact.
03When to use it
Threat-model prompt injection for any system that combines untrusted content with privileged instructions, private context, retrieval, or tools. Risk rises with autonomy and permission. A read-only drafting aid needs different controls from an agent that sends messages or changes records.
04How it works
- Map every source of untrusted content, privilege boundary, sensitive asset, tool, and downstream consumer.
- Minimize accessible data and actions using user-scoped authorization and least-privilege tools.
- Separate and label instructions and content while assuming the model may still confuse them.
- Validate outputs and tool calls deterministically; require confirmation for consequential actions.
- Test direct, indirect, encoded, multimodal, and multi-step attacks, then monitor and contain abnormal behavior.
05A practical example
An email assistant reads a message containing hidden instructions to forward confidential mail. The system prevents harm because the reading context has no send permission, recipients and attachments are server-validated, and any draft-to-send transition requires the user to review a clear action summary.
06Where it breaks
Filters and prompt delimiters can reduce some attacks but are bypassable and may block legitimate content. Detection evolves as models and attackers change. The safest design may be to withhold a capability when residual risk exceeds the value of automation.
Common mistakes
- Treating prompt injection as solved by telling the model to ignore malicious instructions.
- Giving a model broad credentials and relying on a system prompt to enforce permissions.
- Testing only obvious direct phrases while ignoring indirect and multimodal content.
- Logging attacks without designing containment, revocation, and incident response.
07Sources
- LLM01:2025 Prompt InjectionOWASP GenAI Security Project · Security guidanceLLM01:2025 Prompt Injection
- Prompt injection is not SQL injection (it may be worse)UK National Cyber Security Centre · Government security guidancePrompt injection is not SQL injection (it may be worse)