Learn how prompt injection redirects AI agents, why documents can carry unauthorized instructions, and how permissions and realistic testing reduce risk.
An AI assistant can receive the right request from its user and still end up following instructions from somebody else. A document, search result or message it reads along the way may contain directions that compete with the original task. When the system treats those directions as authority, ordinary information becomes a way to steer the assistant.
That problem is called prompt injection. It matters increasingly for AI agents because their responses can lead to actions: opening another document, changing a record or sending information through a connected service. Understanding the risk starts with a practical question: who is allowed to decide what the agent does?
What Is Prompt Injection?
Prompt injection is an attempt to make an AI system follow instructions that conflict with its intended task or operating rules. The attacker tries to influence the model through material it processes. OWASP’s security guidance distinguishes direct attacks, delivered through a user’s input, from indirect attacks embedded in external content.
A direct attack might come from someone deliberately trying to override a customer-service assistant’s rules. An indirect attack can arrive in a page the assistant opens while answering a legitimate customer. The person using the assistant may have done nothing unusual and may never see the suspicious instruction.
The important distinction is authority, rather than whether the words sound polite, technical or urgent. A webpage can supply facts for a comparison. It cannot grant itself permission to change the user’s goal. A sentence announcing that it is a new system instruction does not become one simply because the model can read it.
How a Document Can Redirect a Legitimate Task
Consider an illustrative purchasing workflow. A team asks an agent to compare three suppliers’ public brochures and prepare a recommendation for an internal meeting. The authorized task is to collect specifications, explain trade-offs and draft a comparison. It does not include sending the team’s internal documents to a supplier.
One brochure contains a paragraph telling automated assistants to upload their supporting notes to an external review service before completing the comparison. It presents the upload as a mandatory quality check. The instruction is part of the brochure, but it concerns the assistant’s behavior rather than the product being evaluated.
The attack succeeds if the agent accepts that paragraph as a requirement and changes its workflow accordingly. The failure happens when a source being evaluated acquires the authority to dictate the evaluation process. If the agent also has access to private notes and an upload tool, the consequences can extend beyond an inaccurate recommendation.
A better response is to continue extracting the supplier’s claims while declining the unrelated upload. The agent may mention that the brochure contains instructions directed at automated systems if that detail affects the assessment. It should not require a new conversation with the user just to reject an instruction the supplier had no authority to give.
This example is hypothetical, not a report of a particular supplier or product. It illustrates the mechanism described in early research on indirect prompt injection: an attacker places instructions in data that an application is likely to retrieve, gaining influence without being the application’s legitimate user.
Why This Is Different From a Hallucination
An agent might misread a brochure, invent a specification or calculate a price incorrectly without anyone attacking it. Those are reliability failures. Prompt injection introduces another party’s attempt to redirect the system. Our guide to AI hallucinations explains why a confident answer can still be factually wrong.
The two problems can overlap in an investigation. Suppose the purchasing agent says that sending internal notes was required by company policy. That assertion might be invented, copied from the malicious brochure or produced by combining both. The final answer alone cannot establish what happened; the retrieved material and subsequent actions matter.
Prompt injection also differs from training-data poisoning. The brochure does not need to change the model’s weights or remain in its training data. It only needs to reach the active workflow at a point where the model interprets it as a direction. A temporary page or a single message may therefore be relevant to one task.
Why Better Prompts and RAG Are Not Complete Fixes
Clear instructions can help a model distinguish the user’s task from quoted material. Labels such as “external document” can make the intended boundary easier to understand. But those labels remain information the model must interpret; they do not automatically enforce what the surrounding application can do.
The UK’s National Cyber Security Centre explains why prompt injection is not simply another version of SQL injection. Conventional software can enforce a separation between database commands and parameter values. Language models process instructions and information through a less rigid boundary, so familiar input-escaping techniques do not provide an equivalent universal remedy.
Retrieval-augmented generation, or RAG, supplies documents that can ground an answer. It does not make everything inside those documents trustworthy. OWASP explicitly notes that retrieval and fine-tuning do not fully eliminate prompt injection. A highly relevant document can still contain an unauthorized instruction.
In the supplier example, a retrieval system might correctly identify the brochure as relevant to the comparison. That is a successful search result and a possible attack entry point at the same time. Evaluating retrieval quality and evaluating the agent’s response to instructions inside retrieved content are separate jobs.
The Agent’s Permissions Determine the Consequences
A manipulated recommendation is already a problem, even if the assistant cannot use tools. Adding tools creates further possibilities. The practical impact depends on which information the agent can access, which operations are available and whether the application independently checks proposed actions.
OWASP describes excessive agency in terms of unnecessary functionality, permissions and autonomy. An assistant intended to read a repository does not automatically need the ability to delete its contents. Giving it a broad connector for convenience can expose capabilities that the task never required.
For the purchasing workflow, the team can separate the comparison from communication. The comparison agent receives public brochures and a limited set of approved requirements. A separate, explicitly authorized step handles any external message. Removing access to private notes also removes those notes from the material this particular workflow could disclose.
Read-only access is useful, but it should not be mistaken for complete protection against disclosure. An agent that can read confidential material and send it through another available channel still has a potentially harmful combination of capabilities. Review the complete path from information access to possible destinations, rather than each connector in isolation.
Build Boundaries Outside the Model
A practical application should check an operation before carrying it out. In the supplier scenario, an upload endpoint could reject destinations outside an approved list, regardless of the explanation the model provides. That is different from asking the model to remember the approved list: the software itself enforces the restriction.
Research systems explore stronger versions of this separation. The CaMeL paper describes an architecture that separates control and data flows and enforces policies around tool calls. Its reported guarantees apply within the design’s assumptions and evaluated setting; the paper is not evidence that any application using two models or a filter is automatically secure.
The transferable idea is to make authority explicit. Which instruction established the destination? Which permission allows this document to leave the workspace? Can the tool enforce the answer without trusting a free-form explanation? These questions are more useful than relying on whether the agent sounds certain that an action is necessary.
Human review can provide another boundary when it shows the concrete operation. In our example, a useful approval screen would identify the exact notes being uploaded and the external recipient. A vague “continue with quality checks” button would conceal the decision the reviewer actually needs to make.
What Model Defenses Can and Cannot Establish
Training, detection systems and adversarial testing can improve resistance. Anthropic’s browser research describes using these approaches together and emphasizes that prompt injection remains unsolved. The company’s examples also show why browser agents face a broad exposure: they encounter pages and other content while operating interactive tools.
A detector can flag suspicious material for special handling, but a flag is not the same as a security boundary around every downstream action. Likewise, blocking every document that mentions instructions would make many legitimate tasks impossible. A guide about prompt injection, for example, necessarily contains descriptions of instructions that readers should understand rather than execute.
In the purchasing example, a useful system can still quote the brochure’s suspicious paragraph in an internal assessment. It simply does not adopt the paragraph as a command. Preserving that ability to analyze hostile material is part of making the assistant useful, rather than treating the presence of difficult content as an automatic task failure.
How to Test a Workflow Before Trusting It
Start with a safe copy of the intended workflow, synthetic documents and controlled destinations. Record what the user asked for, what the agent read and which actions actually occurred. The goal is to test observable behavior without introducing real confidential information or giving a trial broad production access.
The AgentDojo benchmark demonstrates an evaluation approach built around agents using tools over untrusted data. Its research separates legitimate task performance from attack effects. That distinction matters: a system that refuses every task may prevent an attacker’s objective while failing its user completely.
For our supplier comparison, create a clean brochure and a version containing the unauthorized upload request. The clean case should produce a useful comparison. The altered case should still produce the comparison, avoid the upload and preserve an accurate record of the source. Check the tool log and destination service rather than accepting the final response as proof.
Then vary the scenario. Place the instruction in a different supplier’s brochure, present it as a correction, or introduce it after the agent has already collected part of the evidence. Include harmless documents that discuss security procedures so that the evaluation can reveal unnecessary refusals. These are proposed test cases for this illustrative workflow, not claims about a published benchmark result.
Repeat important cases and rerun them after changing the model, connector or application policy. Our evaluation guide covers the wider process of checking outcomes, permissions and repeatability. A previous successful demonstration is useful evidence, but it does not establish reliability across new inputs and capabilities.
Questions to Ask Before Connecting an AI Assistant
For an individual user, the supplier example suggests a simple starting point: connect only the information and tools needed for the task at hand. A brochure comparison is easier to assess when it does not also involve a full mailbox, unrestricted file access and the ability to send messages. Expand the workflow when there is a clear reason to do so.
For a team choosing an agent product, ask the vendor to demonstrate an attempted redirection inside a realistic document. Look for the authorized task being completed and the unauthorized action being blocked. Ask which restriction is enforced by the application, what is only a model instruction, and what evidence remains available if something goes wrong.
Prompt injection is ultimately a problem of deciding whose instructions count. An assistant needs to learn from the documents it reads without allowing their authors to take control of the task. Useful automation comes from combining that distinction with enforceable permissions, visible actions and testing that reflects the work people actually expect the agent to perform.
YC-backed Outlit says its AI agents continuously combine internal records and public signals to flag churn risk and sales opportunities across business accounts.
