A business team wants to ship an assistant that reads customer tickets, searches an internal knowledge base and drafts replies. It runs on a model from a well-known provider. The vendor questionnaire came back clean, the demo was convincing, and the launch date is fixed. (This scenario is illustrative, not a documented incident.)

None of that answers the questions a security review has to answer. The model is one component. The risk sits in what the surrounding system can read, what it can do, whose authority it uses, and what happens when it is wrong or when someone manipulates it. This note sets out how to run that review in a working week, what to ask for, and which findings should stop a release.

Assess the system, not the model

Draw the use case as a data flow before reading any vendor documentation. Mark every input the model will see: the user message, retrieved documents, tool results, uploaded files, web pages, emails. Mark every output: text shown to a person, text passed to another system, and every action the model can trigger. Then mark the identity each action runs under.

Two things usually surface within the first hour. Some of the inputs are written by people outside the organization; a customer ticket is attacker-controlled text. And the model receives instructions and data through the same channel, so it cannot reliably tell them apart. That is the mechanism behind prompt injection, which OWASP lists first among its risks for LLM applications. Filters and careful prompt wording lower the success rate. They do not remove the condition. Design on the assumption that some injected instruction will eventually be followed, and limit what it can reach.

Seven questions for the project team

Put these to the team in a single working session. Each has a good answer, and each has a red flag worth recognizing.

QuestionWhat a good answer looks likeRed flag
1. What data can it retrieve, and under whose permissions?Retrieval enforces the requesting user’s existing access rights at query time. Sensitive sources are excluded or segmented.One service account indexes everything, so users can retrieve content they could never open directly.
2. Which inputs come from outside the trust boundary?Every untrusted channel is listed, and tool access is reduced when untrusted content is in context.“Only our users” is the answer, ignoring documents, emails, web pages and tool output.
3. What can it do?A written tool inventory separates read from write and reversible from irreversible. Each tool is justified by the use case.Open-ended tools such as a shell, a generic HTTP client or full mailbox access.
4. Whose identity does it act under?Calls carry the user’s delegated identity or a narrowly scoped workload identity. Credentials sit outside the model’s context.A shared, broadly privileged service account, or keys placed in prompts or configuration the model can read.
5. Where does the output go?Output is treated as untrusted by whatever consumes it: encoded for display, validated before it reaches a database, interpreter or API.Model output rendered as HTML, executed as code or interpolated into queries.
6. What does it depend on?Provider terms on retention and training use, hosting location, and the origin and integrity of plugins, tool servers, embedding models and fine-tuning data are documented.Tool servers or packages added by developers without review, and no record of which model version is live.
7. How will you know it is failing, and who can stop it?Prompts, retrieved context and tool calls are logged. Usage and cost limits exist. A named owner can disable the feature quickly, and incident response knows it exists.Logs hold only the final answer, consumption is unlimited, and nobody owns the system after launch.

The questions line up with several entries in the OWASP Top 10 for LLM Applications, including sensitive information disclosure, supply chain, improper output handling, excessive agency and unbounded consumption. The mapping helps with reporting. The questions are better for the conversation, because project teams can answer them.

Match the evidence to the autonomy

Review depth should follow what the system is allowed to do. The tiers below are a working convention for this kind of review, not a published standard.

TierWhat the system doesMinimum evidence before production
1. Drafts for a humanGenerates text. A person reads and decides before anything leaves.Data-flow diagram, data classification, provider terms checked, output handling for any rendered content, logging.
2. Retrieves and answersSearches internal sources and answers users directly.Tier 1, plus proof that retrieval respects source permissions, cross-user leakage tests, and injection tests through each document source.
3. Calls tools, read onlyQueries other systems on the user’s behalf without changing them.Tier 2, plus a tool inventory, scoped credentials, authorization enforced by the downstream system, and rate limits.
4. Calls tools that change thingsSends messages, updates records, moves money or runs code.Tier 3, plus human approval for high-impact actions, tested rollback or compensation, tampering tests on tool parameters, and a kill switch.
5. Acts across systems with little oversightChains steps or agents, or runs without a person in the loop.Tier 4, plus a threat model specific to the agent design, runtime monitoring with defined stop conditions, identity design for the agent itself, and executive risk acceptance. Treat this as a separate agentic AI review.

What to test

Testing before production should cover the failure modes that the questions expose.

  • Indirect injection through every untrusted source. Place instructions in a ticket, a document, a web page and a tool response. Watch whether the system follows them, leaks data or calls a tool.

  • Cross-user retrieval. Query as a low-privilege user for content only a high-privilege user should see.

  • Tool abuse. Push parameters outside intended bounds: another user’s record, a different recipient, a larger amount.

  • Output handling. Confirm that markup or commands in model output are neutralized where the output lands.

  • System prompt exposure. Assume it will be extracted. OWASP’s guidance is that a system prompt should not be treated as a secret or a security control, so no credentials or access logic belong in it. Authorization has to be enforced outside the model.

  • Consumption. Confirm limits on requests, tokens and tool calls per user and per tenant.

Keep the tests as a repeatable set. A change of model version, system prompt, retrieval source or tool should rerun them, because behavior shifts with each. File the results with the release record.

What should block release

OutcomeExamples
BlockA system that can take irreversible actions, reads untrusted content and has no approval step. Retrieval that bypasses source permissions. Credentials or authorization logic placed where the model can read or alter them. No logging of tool calls. No named owner.
Release with conditionsRate and cost limits. Human approval for named actions. Output encoding. Scoped credentials. A reduced tool set. Provider retention terms confirmed in writing.
Record and monitorInaccurate or off-policy answers, bias and content quality. These belong in the AI risk register with an owner and a review cycle.

What to hand back

One page per use case: purpose, tier, data classes, tool inventory, identities, conditions of release, named owner, and the events that trigger a re-review. A use case that cannot be summarized on one page is usually not understood well enough to launch.

Security is one slice of the decision. Privacy review, classification of the use case under applicable AI regulation, and fairness review run alongside it and can block a launch for their own reasons. The NIST AI Risk Management Framework and ISO/IEC 42001 provide the governance structure around this work. This note covers the security assessment inside it.

NEXT STEP

Run the seven questions against your next use case, assign a tier, and decide from the evidence what blocks release. An independent review before launch is a sensible point to bring in outside help if the team lacks the capacity to test its own system.

Sources

How we label evidence

Facts carry a source. Analysis and recommendations are marked as ours. Examples that are hypothetical are labeled as hypothetical.

Part of the solutionSecure AI & Emerging TechnologyAdopt AI and new platforms without opening new gaps.See how we work →

Related service

AI Security & Governance

Use this to assess enterprise AI security risk.

Talk to an ExpertSee this service →Related articles →
Also relevant:Agentic AI SecurityEmerging Technology SecurityApplication Security