Model inversion
Model inversion is an attack that reconstructs representative features of the data a model was trained on, using only access to the model’s outputs. Where the training set contained personal data, a successful inversion turns a deployed model into a disclosure of that data, which is a privacy problem before it is a security one.
The intuition is that a model which has learned to recognise something has, in the process, encoded what that thing looks like. By optimising an input to maximise the model’s confidence for a particular class or individual, an attacker recovers something that resembles the training examples for that class. The reconstruction is approximate rather than exact, and its quality depends on the task, the model and how much output detail the interface returns.
The reason it matters commercially is regulatory rather than operational. If a model trained on customer records can be made to produce recognisable features of those records, the model is arguably processing personal data everywhere it is deployed, including places the data was never supposed to reach: a copy shipped to a partner, an endpoint published for a mobile application, or a fine tuned checkpoint stored in a repository. That is a question a data protection officer has to be able to answer.
It is a distinct question from membership inference, which asks whether a specific record was in the training set rather than what the records looked like, and both are separate from training data poisoning, which is an attack on the training process instead of on the finished model. Sorting out which of the three applies to a given deployment is part of the AI assessment work that covers what a trained model can be made to reveal.