AI Supply-Chain Security: Models, Data and Dependencies

ARTICLE | AI Security | Secure AI & Emerging Technology | Cross-industry / Technology

An AI system’s supply chain carries the usual software risks plus two that conventional tooling does not see: model files that can run code or carry altered behavior, and data that shapes what the model does. A software bill of materials covers the first layer. It does not cover the rest.

Most organizations can say which libraries their applications use. Far fewer can say which models are running, where those models came from, what data shaped them, or which tool servers an agent is allowed to call.

What is in the chain

Model files: the problem is executable formats

Python’s own documentation warns that the pickle format is not secure and should only be used with data you trust. Pickle-based files remain common for model checkpoints, including PyTorch’s. Loading one can run whatever code an attacker placed inside.

Scanning does not close this gap. ReversingLabs found two models on Hugging Face that used a technique it named nullifAI. The files were PyTorch checkpoints compressed with 7z instead of the default ZIP, which blocked normal loading and let them slip past Picklescan, Hugging Face’s scanner. The payload was a platform-aware reverse shell, placed at the start of a pickle stream that then deliberately broke. Hugging Face removed the models in under a day after being told and updated Picklescan to catch broken pickle files.

The response worked, but the lesson is structural. A scanner looks for patterns it already knows, and attackers change the packaging. The durable control is policy: do not deserialize executable formats from sources you do not control. Formats such as safetensors store tensors without executable code and remove this class of risk where they can be used. Where a pickle-based file is unavoidable, load it only inside an isolated environment with no credentials and no network.

A safe file is not a safe model. Execution risk lives in the file format. Behavior risk lives in the weights, and a clean load says nothing about it.

Behavior risk: data and weights

Poisoning is the data-side equivalent. NIST’s adversarial machine learning taxonomy covers it, and recent research shows the cost to an attacker may be lower than assumed. In a study by Anthropic with the UK AI Security Institute and The Alan Turing Institute, about 250 poisoned documents were enough to implant a backdoor in models from 600 million to 13 billion parameters. The count mattered, not the share of the training data. A hundred documents did not reliably work.

Read that result carefully. The backdoor was narrow: a trigger phrase made the model output gibberish. The authors say it is unlikely to pose significant risk in frontier models, and it is not shown that the pattern holds for larger models or for harmful behaviors such as bypassing safety guardrails. The authors also note that an attacker’s main constraint is getting content into the training data at all.

For an enterprise, that constraint is the useful part. Control who can write to anything you train, fine-tune or retrieve from. Internal fine-tuning data, shared knowledge bases and scraped sources are all attack surface in proportion to how many people and systems can change them. The retrieval path is also the path for indirect prompt injection, covered in the companion article on prompt injection beyond the chat interface.

Can you test your way to safety? Only partly. Behavioral evaluation can find some hidden behaviors. It cannot prove there are none. That is a judgment, not a measured result, and it is why provenance and containment matter as much as testing. Limit what a model can reach, so that an unknown behavior has a bounded cost.

Dependencies around the model

Agent frameworks, vector databases, orchestration libraries and container images follow normal software supply-chain rules. Apply them without exception: pinned versions, vetted sources, and a private mirror for anything that reaches production.

Tool servers deserve separate treatment. Installing one gives the model a new capability and gives a new party influence over the model, through tool descriptions and tool results. That is closer to granting third-party access than to adding a library. Review and allow-list them accordingly.

Hosted models add a different risk. The provider can change model behavior, pricing, retention terms or availability without your release process. Pin versions where the provider allows it, retest on change, and write change notice and data-use terms into the contract.

Inventory: what to record

CycloneDX’s ML-BOM format covers models, datasets, their dependencies and provenance. NIST SP 800-218A adds AI-specific practices to the Secure Software Development Framework for model development. OWASP lists supply chain among the top risks for LLM applications. A bill of materials describes. It does not verify. Treat it as the inventory that lets you answer one question quickly: are we affected?

• Model name, publisher, version or hash, file format, license and base-model lineage.

• Datasets and their sources, and who approved their use.

• Where each model is deployed and what data, tools and outbound channels it can reach.

• Tool servers and plugins in use, including those teams installed without review.

• A named owner who is told when an upstream component is reported as compromised.

An intake gate for new models and datasets

1. Who published it, and can we verify that?

2. What is the file format, and does loading it execute code?

3. Can we pin it by hash and host it internally?

4. What data will it touch, and is it permitted to?

5. What do we know about what it was trained or tuned on, and under what license?

6. What did we test before promotion: task quality, safety behavior, and behavior on unusual inputs?

7. If it misbehaves, what can it do? Which tools, data and outbound channels are in reach, and are they contained?

8. Who owns it after launch?

Decisions for the CISO

Where to start

Start with the inventory. If you cannot list the models, datasets and tool servers running in production, every other control has nothing to attach to. Then stop direct loads from public hubs, set a file-format policy, and put new models through the gate above.

Sources

1. pickle: Python object serialization — Python Software Foundation documentation

2. RL identifies malware ML model hosted on Hugging Face (nullifAI) — ReversingLabs

3. Research post: a small number of poisoned documents can backdoor language models — Anthropic, with the UK AI Security Institute and The Alan Turing Institute. Single experimental setup; authors caution against generalizing

4. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations — NIST

5. Secure Software Development Practices for Generative AI and Dual-Use Foundation Models (NIST SP 800-218A) — NIST

6. CycloneDX ML-BOM — CycloneDX

7. OWASP Top 10 for LLM Applications — OWASP GenAI Security Project