Prompt Injection: Security Implications Beyond the Chat Interface

ARTICLE | AI Security | Secure AI & Emerging Technology | Cross-industry / Technology

Prompt injection becomes an enterprise security problem when a model reads content it cannot trust and is able to act on what it reads. The chat window is the least dangerous place to meet it.

Most early discussion treated prompt injection as a user trying to talk a chatbot out of its rules. That framing undersells the risk. The serious cases involve an attacker who never touches the model, writes some text in a place the model will later read, and waits.

Two kinds, with very different exposure

OWASP separates the two. In direct prompt injection, a user’s own input changes the model’s behavior in unintended ways. In indirect prompt injection, the model takes in outside material, such as web pages or files, and hidden instructions in that material steer it.

Direct injection mostly lets a person act on their own session, inside their own permissions. It matters for policy bypass and leaked system prompts, but the blast radius is bounded. Indirect injection removes that bound. The attacker needs no access to the model and no access to the victim. They need access to something the model will read: an email, a shared document, a web page, a ticket, a code comment.

Why this is not SQL injection

The UK National Cyber Security Centre has argued that comparing the two is misleading. SQL injection was largely solved by parameterized queries, which stop a database from ever treating input as an instruction. Language models have no equivalent boundary. They predict the next token from all the text in front of them, and nothing enforces which part is instruction and which part is data.

The NCSC’s conclusion is that prompt injection may never be mitigated the way SQL injection was. It recommends treating it as a residual risk to be managed, reducing likelihood and impact through design, and accepting that if a system cannot tolerate that risk, an LLM may be the wrong choice. That changes the CISO’s question. It stops being “which product blocks this” and becomes “what happens when it works.”

A documented case: EchoLeak

Researchers described a zero-click prompt injection against Microsoft 365 Copilot, tracked as a CVE and known as EchoLeak. A single crafted email, with no user interaction, let a remote and unauthenticated attacker pull data out through the assistant. The chain got past Microsoft’s cross-prompt-injection classifier, evaded link redaction by using reference-style Markdown, relied on auto-fetched images, and abused a Microsoft Teams proxy that the content security policy allowed.

Each of those defenses was reasonable alone. The chain worked because three conditions were present together. The assistant read untrusted content, it could reach private data, and an outbound channel existed. The paper presents this as a real vulnerability in a production system. It does not report attackers using it in the wild.

The test: three conditions in one system

Simon Willison named the combination the “lethal trifecta”: access to private data, exposure to untrusted content, and the ability to communicate externally. Remove any one of the three and the worst case shrinks sharply. For every AI feature, ask which of the three it has.

The third condition is wider than it looks. Outbound channels include rendered Markdown images, link previews, sending email, creating tickets, calling APIs, and writing to a shared document that someone else can read. EchoLeak used image and link channels. Many teams would not have listed those as exfiltration paths.

Where it appears beyond chat

What does not hold up on its own

• Instructions in the system prompt. An instruction is not enforcement. Later text can outweigh it.

• Deny-lists. The NCSC notes that blocking known attack phrases is ineffective, because attacks can be reworded without limit.

• A classifier as the only defense. Classifiers lower likelihood. The EchoLeak chain went around one. Treat detection as a probabilistic layer.

• Vendor assurances. The NCSC advises skepticism toward claims that a product stops prompt injection.

What does

The controls that hold are architectural. They assume the model will sometimes be fooled and limit what that costs.

1. Match privileges to the content. An agent processing untrusted content should not hold the tools or data scope of a more privileged party. The NCSC states the principle directly.

2. Break the trifecta by design. Separate the component that reads untrusted content from the one that holds sensitive data or can act externally.

3. Close outbound channels. Stop rendering remote images and links from model output, allow-list egress destinations, and strip active content.

4. Enforce policy outside the model. Validate output formats and check tool calls in ordinary code, as OWASP advises. Run privileged functions in code, not through the model.

5. Require approval for high-impact actions. Sending externally, paying, deleting, sharing and changing permissions should need a human. The approval screen should show the actual action, not the model’s description of it. That last point is our recommendation, not a published requirement.

6. Label and isolate untrusted content. This makes attacks harder. It does not make them impossible.

7. Log inputs, outputs and tool calls. Failed calls can mean an attacker is refining an attempt.

8. Test with indirect payloads in the real channels. Put hostile text in the mailbox, the wiki and the ticket queue, not only the chat box.

Research is moving the same way. A research design called CaMeL separates the control flow derived from the trusted user request from the data the agent retrieves, and enforces security policy at tool calls. On the AgentDojo benchmark it solved 77 percent of tasks with provable security, against 84 percent for an undefended system. The gap is the honest cost: constrained designs give up some capability. Expect that trade, and decide where it is worth making.

The decision to make

For each AI feature that reads outside content, ask what the worst action is if the model is fully hijacked. If that action is unacceptable, do one of three things: remove the capability, put an approval in front of it, or do not ship. Name an owner who signs the residual risk.

The same lens applies to procurement. Do not ask a vendor whether it prevents prompt injection. Ask how it separates trusted from untrusted content, what the model can do without approval, what is logged, and how it tests with indirect payloads.

Where to start

Build a one-page table of every AI feature that reads outside content, with its data scope, its tools and its outbound channels. Where all three conditions meet in one feature, fix that first.

Sources

1. LLM01: Prompt Injection — OWASP GenAI Security Project

2. Prompt injection is not SQL injection (it may be worse) — UK National Cyber Security Centre

3. EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System — AAAI Symposium paper, also on arXiv

4. The lethal trifecta for AI agents — Simon Willison. Practitioner framing, not a standard

5. Defeating Prompt Injections by Design (CaMeL) — Debenedetti et al., arXiv. Research result on one benchmark