Most enterprise AI risk sits around the model, not inside it. Securing the model matters. It is not enough.
The phrase “AI security” has been tied closely to adversarial machine learning: evasion, data poisoning, model extraction and privacy attacks. NIST’s taxonomy of adversarial machine learning (AI 100-2) lays these out well. They are real, and they weigh most on organizations that train, fine-tune or host their own models. Most enterprises do something else. They consume a model through an API or a platform and build an application around it. Their exposure lives in that application.
A hypothetical
This is a hypothetical scenario, not a real incident.
A support assistant reads inbound customer email, searches internal documents for answers and can issue small refunds through an internal API. An attacker sends an email with hidden instructions in the body. The assistant treats those instructions as part of its task, retrieves a document it should not have used and calls the refund function.
Nothing was wrong with the model’s weights. Three things failed. Untrusted content was treated as instruction. The assistant held a tool permission that was too broad. No human approval stood between a manipulated output and a real action.
The layers around the model
A useful way to think about enterprise AI is as a system with several layers, each with its own failure modes and its own control owner.
| Layer | What goes wrong | Where control usually sits |
|---|---|---|
| Model | Evasion, poisoning of training or fine-tuning data, model extraction, leakage of training data | Model provider; your team if you fine-tune or host |
| Data and retrieval | Poisoned or sensitive content in the knowledge base; retrieval that ignores source permissions, so users see what they should not | Data owners and platform team |
| Prompt and context | Direct and indirect prompt injection; leakage of system instructions | Application team |
| Tools and agency | Model-triggered actions that exceed what the task needs | Application and identity teams |
| Identity and access | Agents, API keys and service tokens acting as non-human identities with broad, standing access | Identity security |
| Supply chain | Third-party models, datasets, plugins and connectors; model files in formats that can run code when loaded | Engineering and procurement |
| Output handling | Model output passed unchecked to browsers, databases, shells or other systems | Application security |
| Operations | No logging of prompts, retrievals and tool calls; unsanctioned AI use; uncontrolled consumption | Security operations and IT |
Most of these categories appear in the OWASP Top 10 for LLM Applications, which covers prompt injection, sensitive information disclosure, supply chain weaknesses, data and model poisoning, improper output handling, excessive agency, vector and embedding weaknesses and unbounded consumption. Notice how few of them depend on the model’s internals.
What makes AI different
In a conventional application, code and data travel separately. In a language-model application, instructions and data share one channel: text. The model cannot reliably tell a legitimate instruction from content that merely looks like one. That is why prompt injection is not a bug that a patch removes.
OWASP and NIST both treat this as an open problem. Filtering, system-prompt hardening and detection tools reduce the chance and the impact of manipulation. None removes the condition. The practical answer is to design as though the model can be manipulated, and to limit what a manipulated model can do.
One consequence follows directly. Treat model output the way you treat input from an untrusted user. Validate it before it reaches a browser, a database or an API.
What to do
-
Find the AI that exists. Inventory sanctioned and unsanctioned use. For each system, record the data it can read and the actions it can take.
-
Threat-model the system, not the model. Walk each use case through the layers above. Start with the ones that can act or that touch sensitive data.
-
Cap what an AI can do. Give agents and assistants least-privilege access, narrow and short-lived credentials, and human approval for actions with real consequences.
-
Make retrieval permission-aware. An assistant should only retrieve what the requesting user is allowed to see. A shared index that ignores source permissions turns the assistant into a way around access control.
-
Treat output as untrusted. Validate and constrain what the model produces before another system acts on it.
-
Vet the supply chain. Check where models, datasets, plugins and connectors come from. Prefer safe model file formats, and review third-party connectors as you would any software dependency.
-
Log enough to investigate. Record prompts, retrieved content and tool calls so a security team can reconstruct what an assistant did and why.
-
Govern with a recognized frame. The NIST AI Risk Management Framework and its Generative AI Profile, and ISO/IEC 42001 for an AI management system, give structure. Where the EU AI Act applies, map your use cases to its risk categories and obligations.
What is still moving
Attack techniques, agent frameworks and regulatory timelines are changing fast. OWASP now publishes separate guidance for agentic applications, where models plan and act across several steps and tool permissions and identity matter even more. Treat the controls above as a baseline to revisit regularly, not a finished standard. Where a claim depends on a vendor’s product behavior, test it in your own environment.
Next Step
Sources
-
NIST, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2). NIST Computer Security Resource Center, csrc.nist.gov csrc.nist.gov
-
NIST, AI Risk Management Framework and the Generative AI Profile (NIST AI 600-1). https://www.nist.gov/itl/ai-risk-management-framework
-
OWASP GenAI Security Project, OWASP Top 10 for LLM Applications and OWASP Top 10 for Agentic Applications. https://genai.owasp.org
-
ISO/IEC 42001, Information technology — Artificial intelligence — Management system. International Organization for Standardization. iso.org
-
European Union, Artificial Intelligence Act. Official Journal of the European Union (EUR-Lex). eur-lex.europa.eu
How we label evidence
Facts carry a source. Analysis and recommendations are marked as ours. Examples that are hypothetical are labeled as hypothetical.
Part of the solutionSecure AI & Emerging TechnologyAdopt AI and new platforms without opening new gaps.See how we work →Related service
AI Security & Governance
Use this to assess enterprise AI security risk.