Adversarial example
An adversarial example is an input crafted with a small, deliberate perturbation that causes a machine learning model to produce a wrong output while looking unchanged to a person. It is the classical attack on classifiers, and it is still what matters wherever a model makes a security or safety decision.
The perturbation is computed rather than guessed. With access to the model’s gradients an attacker can calculate the smallest change that pushes an input across a decision boundary, and iterative versions of that calculation produce examples that are extremely reliable. Without that access the same result is often reached by transfer: craft the example against a substitute model trained for the same task, and a useful proportion of the examples work against the target as well.
In a security context the models that decide things are the interesting ones. A malware classifier, a fraud scorer, a document or identity verification step, a content filter: each is a model whose wrong answer has a direct consequence, and each is a target that gives the attacker feedback with every attempt. The same reasoning applies to the classifier sitting in front of a language model as guardrails, which is why a filter based on a model can be attacked in a way that a rule cannot.
Defences reduce the rate rather than close the class. The honest framing for a client is that a model used as a security control can be optimised against, so it belongs behind something deterministic rather than in front of it. That is separate from a jailbreak, which manipulates instructions rather than numerical input, and both fall inside the AI testing that includes the classifiers, not only the chat interface.