Training data poisoning

A model learns from its data, so whoever influences the data influences the model. Poisoning inserts crafted examples into a training or fine-tuning set: enough mislabelled or trigger-bearing samples to teach the model an association the owner never intended. A backdoor is the sharpest form: the model behaves normally until a specific trigger appears in the input, then produces the attacker’s chosen output. The opportunity comes from where training data originates, which is often scraped from the open web, pulled from public datasets, or contributed by third parties. Each of those is a point where an attacker who does not control the training run can still control some of the training input.

The problem is that a poisoned model passes ordinary testing. Accuracy on the benchmark looks fine, because the malicious behaviour only appears under the trigger, and the trigger is not in the benchmark. From the attacker’s side this is the appeal: the payload ships inside the weights, survives deployment, and is invisible to anyone who evaluates the model the normal way. It also connects to the wider AI supply chain: a poisoned public dataset or a tampered fine-tuning corpus is the same class of risk as a software supply chain attack, and it is treated with far less rigour.

We assess the pipeline, not just the model output. What is the provenance of the training and fine-tuning data, is it integrity-checked, and who can contribute to it. Where the model is exposed through an application, we test for trigger-driven behaviour and for the neighbouring risks that share its surface, such as a retrieval-augmented generation source that can be manipulated at query time. The finding is written against the missing control in the data supply chain. This is part of how we review a model and its data pipeline.