AI Red Teaming: What Should Be Tested?

Test the whole system the model sits inside, and start from what that system can do. A model that refuses every harmful prompt but sits behind an assistant with write access to a mailbox, a ticketing system and a customer database has passed the wrong test.

Many AI red team scopes are built around the model: jailbreaks, harmful output, policy bypass. That work has a place. But an organization that deploys a third-party model has little control over the model itself. What it controls is the application, the data it connects, the tools it exposes and the identities those tools run under. A red team that spends most of its effort on the model is testing the layer the organization controls least.

The Microsoft AI Red Team makes a related point after testing more than 100 generative AI products. Generative AI amplifies existing security risks as well as adding new ones. One of its published case studies traced a server-side request forgery in a video-processing AI application to an outdated FFmpeg component. The model was not the problem.

Six layers worth testing

The OWASP GenAI Red Teaming Guide describes the work across four areas: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. The table below is a working structure that breaks the same ground into six layers an enterprise can scope, assign and retest. It is our own framing, not a published standard.

In our assessment, layers 2 to 5 hold most of the risk an enterprise can actually control. Layer 1 matters most where you build or fine-tune a model, or where harmful output creates direct legal or reputational exposure.

Write test objectives as consequences

A scope that says “test for prompt injection” produces a list of clever prompts. A scope that says what must not happen produces attack paths. OWASP defines indirect prompt injection as instructions hidden in external content, such as a web page or file, that change the model’s behavior when it reads that content. Objectives should start from there:

• A third party who controls a web page or document the assistant reads must not be able to make it send data outside the organization.

• A low-privileged user must not be able to retrieve content they could not open directly.

• A manipulated agent must not be able to approve, pay, delete or change access without a human or deterministic check that sits outside the model.

• Model output must not be able to run code or alter a query in a downstream system.

• A stolen tool credential must not open anything beyond that tool’s task.

Each objective names an attacker position, a boundary and an impact. That also makes the result readable by a system owner who is not an AI specialist.

Methods, and who should do the work

• Use people and automation together. Microsoft reports that automation helps generate prompts, orchestrate attacks and score responses, but cannot replace human judgment, especially where subject knowledge matters. Open-source tooling such as PyRIT exists for the automated part.

• Test from three attacker positions. An ordinary user. The author of third-party content the system ingests. A compromised internal account. Each reaches different layers.

• Name attacks consistently. MITRE ATLAS is a knowledge base of adversary tactics and techniques against AI-enabled systems. NIST AI 100-2 provides a taxonomy and terminology for adversarial machine learning attacks and mitigations. The OWASP Top 10 for LLM Applications gives common weakness classes. Using them makes findings comparable and easier to hand to engineering.

• Test in production-like conditions. Real integrations, real permissions, realistic data. A sandbox with stubbed tools hides layers 3 to 5, where consequential findings are most likely to sit.

What a useful result looks like

A refusal rate across a prompt set is a measurement. It is not a finding. A finding states the attack path, the preconditions, the impact and the control that should change, at the layer where that control can be enforced.

There is a reason to expect fixes to land outside the model. The UK NCSC argues that prompt injection differs from SQL injection because a language model has no built-in separation between instructions and data, and that it may never be fully mitigated in the way SQL injection can be. OWASP likewise says it is unclear whether a fully reliable prevention method exists. So results tend to point to impact reduction: narrower tool scope, least privilege, deterministic checks, approval for high-impact actions, and monitoring. Expect recommendations about permissions and architecture more than about prompts.

When to test again

A red team report describes one configuration at one point. Retest when something that changes the attack surface changes: a new model version, a new tool or data source, a change to permissions, a new user population, or a newly published class of attack. Microsoft describes this as a break-fix cycle: test, measure, mitigate, test again. The trigger is change in the system, not the calendar.

A note on regulation

Article 55(1)(a) of the EU AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing. It applies to those providers. Most enterprises that deploy AI are not in that group, and whether any obligation reaches your organization depends on your role and jurisdiction, so confirm with counsel. Our reading is narrower: documented adversarial testing is moving towards being a normal expectation of responsible AI deployment, and customers and regulators are likely to ask what was tested and what was found.

NEXT STEP

---

Start with one production AI system. List its tools, data sources and identities, then write the five consequences above as test objectives for it. If the team cannot say what the worst reachable action is, that gap is the first finding.

Sources

• OWASP GenAI Security Project. GenAI Red Teaming Guide.

• OWASP GenAI Security Project. OWASP Top 10 for LLM Applications.

• OWASP GenAI Security Project. LLM01 Prompt Injection.

• Microsoft AI Red Team. Lessons From Red Teaming 100 Generative AI Products.

• MITRE. ATLAS.

• NIST. AI 100-2, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations.

• UK National Cyber Security Centre. Prompt injection is not SQL injection (it may be worse).

• European Union. EU AI Act, Article 55: Obligations for providers of general-purpose AI models with systemic risk.