Model and Application Security: Where the Boundary Actually Is

The model is not a security boundary. It is an untrusted component inside an application, and the application is where the boundary has to be enforced. Teams that draw the line between “model security” and “application security” usually put it in the wrong place, and the risk collects in the gap.

Why the usual split fails

In many organizations the data science or ML team owns the model, and the application security team owns the code, APIs and dependencies around it. The model team looks at behavior and quality. The AppSec team looks at the application. Nobody owns the seam.

The risks that matter most sit in that seam. Untrusted text goes into the model. Model output then drives something real: a database query, an email, an API call, a rendered web page. OWASP lists prompt injection, improper output handling and excessive agency as separate entries in its Top 10 for LLM Applications, but they describe one chain.

Why the model cannot enforce a boundary

Security engineers know how SQL injection was brought under control. Parameterized queries keep data and instructions apart, so the database never executes user input as a command. The UK NCSC points out that language models have no equivalent. A model receives text and predicts more text. It has no built-in way to tell the developer’s instructions from content it has been handed. The NCSC concludes that prompt injection may never be fully mitigated the way SQL injection can be, and that the realistic goal is to reduce likelihood and impact through design. OWASP is similarly careful: it says it is unclear whether a fully reliable prevention method exists.

The practical consequence is simple. Anything that must hold cannot depend on the model following instructions. A system prompt that says “never reveal customer data” is guidance. It is not a control. Authorization, data access and approval of high-impact actions have to be enforced somewhere the model cannot talk its way past.

Where the boundary goes

These seven rules put the boundary in the application, where it can be enforced. They draw on OWASP and NCSC guidance and on our own reading of how these systems fail.

1. Treat model output as untrusted input. Validate and encode it before it reaches code, browsers, databases or shells. OWASP calls the failure improper output handling.

2. Enforce authorization outside the model. Downstream systems should check whether the user may perform the action, instead of relying on the model to decide. OWASP’s term for this is complete mediation.

3. Run tools with the user’s privileges, not a broad service identity. The NCSC makes the point that a model handling someone’s input should hold only that person’s privileges. It should not read external email while holding access to privileged tools.

4. Keep tools narrow. A tool that looks up order status is safer than one that runs a query. OWASP advises against open-ended extensions such as shell execution or arbitrary URL fetching.

5. Put a human or a deterministic check in front of high-impact actions. Payments, deletions, access changes and outbound data transfers should not rest on model judgment alone.

6. Separate trusted instructions from untrusted content where the platform allows. Marking external content reduces the success rate of attacks. It does not stop them, so it should never be the only layer.

7. Log prompts, outputs, tool calls and failed tool calls. The NCSC notes that failed tool calls can reveal an attacker probing the system.

Who owns what depends on how you consume the model

The split of responsibility changes with the deployment pattern. The table is a planning model. Contract terms and product design vary by provider, so it is not a statement of any vendor’s actual responsibilities.

Watch the first row. It is the easiest one in which to assume the provider has covered risks that sit entirely on the customer side.

Where classic application security still carries the load

AI applications inherit every ordinary web and API weakness. Microsoft’s red team found a server-side request forgery in an AI video-processing application that came from an outdated component. OWASP’s list includes supply chain and unbounded consumption alongside the model-specific entries. Dependencies, secrets, API authentication, rate limiting and error handling all still apply.

The implication for program design: do not stand up a separate “AI security” track that bypasses the existing secure development lifecycle. Extend it. Add AI-specific threat modeling, output handling checks and agent permission reviews to the controls teams already follow.

Where classic application security stops working

• Input validation. You cannot enumerate bad inputs in natural language. The NCSC warns against relying on denylists of known malicious phrases because attackers can rephrase without limit.

• Testing. Model output is not deterministic, so a passing test is evidence, not proof. Behavior tests are statistical. Control tests should target the deterministic layers, where a pass means something.

• Change. A provider can update a model without any change to your code, and behavior shifts with it. Treat model version changes like dependency upgrades that need re-testing.

A test you can run on your own system

Assume the model is fully controlled by an attacker. OWASP recommends treating the model as an untrusted user for exactly this reason. Then ask what the application can still do.

If the answer is that it can still email customers, read the CRM and run queries, the boundary is in the wrong place. If the answer is that it can return text to a screen and nothing else, the design is sound. Most systems fall between those two. The gap between them is your work list.

NEXT STEP

---

Pick one AI-enabled application and run the attacker-controlled-model test on it. Record every action it could still take, then assign each to an owner and a control that does not depend on the model.

Sources

• UK National Cyber Security Centre. Prompt injection is not SQL injection (it may be worse).

• OWASP GenAI Security Project. OWASP Top 10 for LLM Applications.

• OWASP GenAI Security Project. LLM01 Prompt Injection.

• OWASP GenAI Security Project. LLM06 Excessive Agency.

• Microsoft AI Red Team. Lessons From Red Teaming 100 Generative AI Products.