termEngineering & architecture

Prompt Injection

Prompt injection is an attack in which untrusted input influences a model to disregard intended instructions, expose information, or misuse connected capabilities.

Download Markdown (.md)

01What it means

Prompt injection exploits the fact that language-model applications process instructions and data in a shared representation without a dependable security boundary between them. A direct injection comes from a user; an indirect injection arrives through content the system reads, such as a webpage, email, document, image, or tool result. Jailbreaking is related but usually focuses on bypassing model restrictions rather than compromising an application's trusted workflow.[1][2]

Untrusted content crossing an instruction boundary Data can influence model decisions unless the application limits authority around it
  1. 01Untrusted sourceUser, webpage, email, document, tool result
  2. 02Shared model contextInstructions and content coexist
  3. 03Manipulated proposalDisclosure, tool call, downstream text
  4. 04Deterministic controlsAuthorize, validate, confirm, constrain
  5. 05Contained outcomeLeast privilege, monitoring, recovery

Separating or labeling instructions helps interpretation but is not a dependable security boundary; deterministic controls must protect actions and data.

02Why it matters

When a model can retrieve private data or call tools, manipulated output can become data exposure, unauthorized action, or corrupted decisions. Prompt wording alone cannot eliminate the weakness, so product scope and deterministic controls must limit the possible impact.

03When to use it

Threat-model prompt injection for any system that combines untrusted content with privileged instructions, private context, retrieval, or tools. Risk rises with autonomy and permission. A read-only drafting aid needs different controls from an agent that sends messages or changes records.

04How it works

  1. Map every source of untrusted content, privilege boundary, sensitive asset, tool, and downstream consumer.
  2. Minimize accessible data and actions using user-scoped authorization and least-privilege tools.
  3. Separate and label instructions and content while assuming the model may still confuse them.
  4. Validate outputs and tool calls deterministically; require confirmation for consequential actions.
  5. Test direct, indirect, encoded, multimodal, and multi-step attacks, then monitor and contain abnormal behavior.

05A practical example

Example

An email assistant reads a message containing hidden instructions to forward confidential mail. The system prevents harm because the reading context has no send permission, recipients and attachments are server-validated, and any draft-to-send transition requires the user to review a clear action summary.

06Where it breaks

Filters and prompt delimiters can reduce some attacks but are bypassable and may block legitimate content. Detection evolves as models and attackers change. The safest design may be to withhold a capability when residual risk exceeds the value of automation.

Common mistakes

  • Treating prompt injection as solved by telling the model to ignore malicious instructions.
  • Giving a model broad credentials and relying on a system prompt to enforce permissions.
  • Testing only obvious direct phrases while ignoring indirect and multimodal content.
  • Logging attacks without designing containment, revocation, and incident response.

07Sources

  1. LLM01:2025 Prompt InjectionOWASP GenAI Security Project · Security guidanceLLM01:2025 Prompt Injection
  2. Prompt injection is not SQL injection (it may be worse)UK National Cyber Security Centre · Government security guidancePrompt injection is not SQL injection (it may be worse)
View saved topics

Start typing to search titles, definitions, categories, and frameworks.