An AI agent changes the security problem in one way that matters more than any other: its output is no longer text for a person to read. It is an action that something will carry out. The attack surface moves from what the model says to what the model is allowed to do.
A chat assistant that gets manipulated gives a bad answer. An agent that gets manipulated sends the email, runs the query, merges the code or calls the payment API. The model is the same. The consequence is not.
What an agent actually is
An agent is a language model running in a loop. It takes a goal, picks a step, calls a tool, reads the result and picks the next step, until it decides it is done. Four things sit around the model, and each one carries security weight: tools (APIs, shells, browsers, databases), memory (state kept between steps or sessions), credentials (whatever lets those tools act) and an orchestrator (the code that runs the loop, sometimes coordinating other agents).
| Chat assistant | Agent | |
|---|---|---|
| Produces | Text for a person to read | Actions on real systems |
| Credentials | Usually none | Tokens, keys or service accounts |
| Untrusted input | What the user types or pastes | Also every document, page, email and tool result it reads |
| Failure looks like | A wrong or harmful answer | An unauthorized change, data movement or spend |
| Blast radius | One conversation | Everything its credentials can reach |
| Evidence trail | The chat log | A record of every tool call, and whose identity made it |
Why the model cannot be your control boundary
Language models do not keep instructions and data apart. Anything in context can be read as an instruction: your prompt, a retrieved PDF, a web page, a tool’s reply. This is why OWASP puts prompt injection at the top of its LLM risk list, and why it names excessive agency as a separate risk: too much functionality, too many permissions or too much autonomy handed to the model.
Direct injection comes from the person typing. Indirect injection is the one that matters for agents. The attacker never talks to the agent. They plant text where the agent will read it: a support ticket, a shared document, a calendar invite, a package README, a tool description. The agent reads it during legitimate work and acts with legitimate access.
Our analysis: data theft becomes straightforward when three conditions meet. The agent can reach private data. It reads content an attacker can influence. It can send something outside: an email, a web request, a pull request comment.
Remove any one condition and the path narrows. Most useful agents have all three by design, so the answer is rarely to avoid the combination. It is to control each leg outside the model.
The new attack paths
OWASP’s Top 10 for Agentic Applications names ten risks. For a security team they collapse into five paths.
| Path | How it works | OWASP risks | Illustrative case |
|---|---|---|---|
| Steering the goal | Injected content redirects what the agent is trying to do | Agent Goal Hijack | An agent asked to summarize a document follows instructions hidden inside it |
| Abusing tools and identity | The agent uses legitimate tools or inherited privilege beyond what was intended | Tool Misuse and Exploitation; Agent Identity and Privilege Abuse; Unexpected Code Execution | A broad service account plus a shell tool turns one bad decision into host access |
| Poisoning what the agent trusts | Memory, retrieved context or third-party tools carry attacker content | Memory and Context Poisoning; Agentic Supply Chain Compromise | A poisoned tool description or stored memory shapes behavior across sessions |
| Spreading across agents | Weak trust between agents lets one failure become many | Insecure Inter-Agent Communication; Cascading Agent Failures | One agent’s output is treated as another agent’s trusted instruction |
| Exploiting people and drift | Humans over-trust agent output, or an agent acts outside its mandate | Human-Agent Trust Exploitation; Rogue Agents | An approver clicks through the fiftieth request of the day |
This is already happening
Two pieces of evidence are worth knowing.
First, an attack run largely by an agent. Anthropic reported disrupting an espionage campaign in which operators manipulated an AI coding agent, connected to tools through the Model Context Protocol, into doing most of the intrusion work. Anthropic estimates the AI carried out 80 to 90 percent of tactical operations, with humans stepping in at a few decision points such as approving the move from reconnaissance to exploitation. About thirty organizations were targeted, and a small number of intrusions succeeded. Anthropic attributes the activity with high confidence to a Chinese state-sponsored group.
Two details matter for defenders. The operators got the model to cooperate by posing as a security firm doing defensive testing, and by splitting the work into small tasks that looked harmless alone. And the model often overstated findings and invented data, which forced humans to check its work. Our reading: per-request judgment will not catch decomposed tasks, so detection has to look at sequences of actions. The model’s unreliability slowed this campaign. It is not a defense to plan around.
Second, the integration layer is a target. The Model Context Protocol’s own security guidance documents confused-deputy attacks through proxy servers, token passthrough (which the specification forbids), server-side request forgery during discovery, compromise of locally run servers and over-broad scopes. These are familiar OAuth and web problems appearing in a new place. Teams adopting MCP servers should check implementations against that guidance rather than assume they follow it. MITRE ATLAS, the ATT&CK-style knowledge base for AI threats, now carries techniques specific to agents, which gives detection teams shared vocabulary.
What changes in the control model
Controls have to sit outside the model and hold whether or not the model behaves. The agentic guidance frames this as least agency: give an agent only the autonomy a bounded task needs. It extends least privilege. Least privilege limits what an identity can reach. Least agency limits how much the agent decides alone once it gets there. You need both.
| Control | What it limits | How it works in practice |
|---|---|---|
| Task-scoped, short-lived credentials | Blast radius of a hijacked agent | Issue credentials per task or run. Avoid standing keys held by the agent. |
| Policy enforced outside the model | Which tool calls can happen | Allowlist tools, validate arguments against a schema, deny by default. |
| Approval for irreversible actions | Consequence of a wrong decision | Show the approver the exact action and parameters, not a summary. Keep the queue small enough to read. |
| Sandboxed execution | Code and shell access | No host access, no ambient credentials, resource limits. |
| Egress control | The exfiltration leg | Allowlist destinations for agent traffic. |
| Tool output treated as untrusted | Injection through results | Retrieved content must never change tool permissions or the task. |
| Full action logging | Detection and investigation | Log tool, arguments, result, initiating user and agent identity. |
| Kill switch and rate limits | Cascading failure | Pause an agent or revoke its credentials fast. Rehearse it. |
Three assumptions to drop
“A strong system prompt is a control.” It is guidance. Content in context can override it. Keep it as one layer, not the layer.
“A human in the loop makes it safe.” Only if the human can see what will happen and has time to judge it. A queue of two hundred approvals is not a control. OWASP lists human-agent trust exploitation as a risk in its own right.
“It is an application like any other.” Applications do what their code says. Agents decide at runtime, so you cannot review every path in advance. You can only bound them.
Standards are forming, not settled
NIST’s Center for AI Standards and Innovation has launched an initiative on agent standards, covering security and identity. The NCCoE concept paper on software and AI agent identity and authorization asks about identification, authorization, auditing, non-repudiation and prompt injection controls. Both are at an early stage. Treat them as direction of travel, not requirements, and build on current OAuth and OpenID primitives that will carry forward.
Where to start
Take your three most capable agents, including those embedded in SaaS tools, IDEs and browsers, and answer five questions about each.
-
Who owns it, and who approved what it can do?
-
Which credentials does it use, and could they be narrower or shorter-lived?
-
Does it hold all three conditions: private data, untrusted input and a way out?
-
Which of its actions cannot be undone, and what gate sits in front of them?
-
Could you reconstruct what it did, under which identity and on whose instruction?
Gaps in those answers mark where autonomous-agent risk is highest in your environment.
Sources
-
OWASP GenAI Security Project, Top 10 for Agentic Applications. https://genai.owasp.org/owasp-top-10-for-agentic-applications
-
OWASP GenAI Security Project, Top 10 for LLM Applications (prompt injection; excessive agency). https://genai.owasp.org/llm-top-10/
-
Anthropic, Disrupting the first reported AI-orchestrated cyber espionage campaign. https://assets.anthropic.com/m/ec212e6566a0d47/original/Disrupting-the-first-reported-AI-orchestrated-cyber-espionage-campaign.pdf
-
Model Context Protocol, Security Best Practices. https://modelcontextprotocol.io/specification/draft/basic/security_best_practices
-
MITRE, ATLAS knowledge base of adversary tactics and techniques against AI systems. https://atlas.mitre.org
-
NIST National Cybersecurity Center of Excellence, Software and AI Agent Identity and Authorization. https://www.nccoe.nist.gov/projects/software-and-ai-agent-identity-and-authorization
-
NIST Center for AI Standards and Innovation, AI Agent Standards Initiative. https://www.nist.gov/node/1906621
How we label evidence
Facts carry a source. Analysis and recommendations are marked as ours. Examples that are hypothetical are labeled as hypothetical.
Part of the solutionSecure AI & Emerging TechnologyAdopt AI and new platforms without opening new gaps.See how we work →Related service
Agentic AI Security
Use this to assess autonomous-agent security risk.