Protecting Sensitive Data in AI Workflows

Sensitive data enters an AI workflow through more doors than the prompt, and most of them sit outside the controls built for files, databases and email. The first control is a data-flow map that follows information through the model call and the systems around it, not past them.

Most data protection programs were designed around data at rest in a known place and data in transit between known systems. An AI workflow adds copies and transformations: text chunked and embedded, summaries generated, prompts logged, results stored in agent memory. Each can hold sensitive content, and none of them inherits the label of the document it came from unless someone built the pipeline to carry it.

Where the data actually travels

Why labels do not survive the pipeline

A classification label usually attaches to a file or a record. Once a document is split, embedded and summarized, the label rarely follows the derived content unless the pipeline is designed to pass it along. A summary of a restricted document can end up less restricted than its source, simply because nothing told the system otherwise. This is interpretation rather than a documented finding, but it is the reason a data inventory built on files will understate what an AI workflow holds.

OWASP’s entry on vector and embedding weaknesses names the practical risks: unauthorized access to embeddings that hold sensitive data, cross-context leaks when different groups share one vector database, and the possibility of inverting embeddings to recover source information. Its recommended mitigations are fine-grained permissions, strict partitioning of datasets, validation of knowledge sources, and detailed immutable logs of retrieval activity.

Instructions are not a control

The common first response is to tell the model not to reveal certain things. OWASP’s guidance on sensitive information disclosure warns that such system-prompt restrictions may not always be honored and can be bypassed through prompt injection. They are useful as a courtesy to the model and worthless as an enforcement boundary.

EchoLeak shows what happens when the boundary is missing. According to the disclosure by Aim Security, hidden instructions in an external email were pulled by Microsoft 365 Copilot’s retrieval into the same context as sensitive internal data, and the data left through URLs the attacker could observe. The researchers called it an LLM scope violation: content the system should have treated as untrusted shared a context with data it should never have reached. Microsoft fixed the flaw and press reporting found no evidence of malicious exploitation, so treat it as a demonstration of the pattern rather than a breach.

NIST’s Generative AI Profile frames this as risk-register material. It lists data privacy among its generative AI risks, covering leakage, unauthorized disclosure and de-anonymization of personal, biometric, health and location data, and separately lists information security, including exposure of model weights, code, training data and outputs.

Controls by position

It helps to think of controls by where they sit relative to the model call.

• Before the model. Minimize what is sent. Redact or tokenize identifiers where the task allows. Check the requester’s entitlements before retrieval, not after generation.

• At the model boundary. Use approved providers and deployments. Confirm retention and training terms with the provider rather than relying on marketing descriptions, and record what was confirmed.

• After the model. Check outputs for sensitive content and restrict where results and agent actions can go.

• Around the model. Apply retention, access control and deletion to logs, caches, embeddings and fine-tuned artifacts.

The decision that ties these together is a tiering of workflows by data class. A practical version, offered as a recommendation, defines which classes of data may enter which kinds of workflow: public-only tools for open information, enterprise-approved deployments with confirmed contractual terms for internal data, and restricted deployments for the most sensitive classes, or a rule that they stay out of generative workflows altogether until controls justify otherwise.

The trade-offs

Each control costs something. Redaction removes information the model may need and can lower answer quality. Tokenization stops the model reasoning about the values it hides. Privacy techniques that OWASP lists, such as differential privacy, involve a loss of utility. Permission-aware retrieval is the most important of these and the most demanding, because it needs the asker’s identity to travel all the way to the retrieval layer, which links data protection directly to how agents and users are identified and authorized.

There is also a limit to any prevention. Users will paste things where they should not. Reducing the damage, through retention limits, deletion paths and detection, matters as much as blocking.

Questions to answer before go-live

1. Which data classes can reach the model, and who decided?

2. Does retrieval enforce the entitlements of the person asking, or those of the system?

3. Where are prompts and completions stored, for how long, and who can read them?

4. Can a deletion or retention requirement reach embeddings, caches, logs and fine-tuned artifacts?

5. What can the workflow send outside the organization, and through which channels?

Where to start

Sources

• OWASP GenAI Security Project: Top 10 for LLM Applications, Sensitive Information Disclosure — OWASP genai.owasp.org

• OWASP GenAI Security Project: Top 10 for LLM Applications, Vector and Embedding Weaknesses — OWASP genai.owasp.org

• Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) — National Institute of Standards and Technology nist.gov

• Zero-Click AI Vulnerability Exposes Microsoft 365 Copilot Data Without User Interaction — The Hacker News, reporting Aim Security’s EchoLeak disclosure thehackernews.com