How an AI pentest works

What an AI pentest actually is

An AI pentest is a penetration test whose target happens to contain a model. NIST SP 800-115 defines penetration testing as security testing in which assessors "mimic real-world attacks to identify methods for circumventing the security features of an application, system, or network". Having a model in the stack does not change that.

What changes is the attack surface. The subject is the assistant or agent around the model: the prompt that assembles context, the documents it retrieves, the tools it may call, the identity those tools run as, and the application that renders what comes back. Most findings with real impact live in that plumbing, not in the weights.

It is not a benchmark of your model, and not a safety or bias evaluation; neither of those is what a pentest report answers. The commercial shape sits on the AI pentesting page; this is the method.

Scope, rules of engagement and what you need ready

Nothing is touched until scope is signed. NIST SP 800-115 says the assessment plan, its term for the rules of engagement, answers five questions: scope, authorisation, logistics, handling of sensitive data, and what happens in the event of an incident. Four more are specific to AI:

• Which environment. 800-115 warns that penetration testing can cause loss of system availability or exposure of sensitive data, and advises similarly configured non-production systems. For AI that means a staging copy with a representative corpus; an empty index tests nothing.
• Whose terms apply. If a third party serves the model, adversarial testing meets that provider’s acceptable use policy, which somebody reads first.
• Whether the defences stay on. We ask for two windows: one with filters, WAF and rate limits as production has them, one with the rate limit lifted, which shows what the model does when a filter is bypassed.
• What counts as an incident. A tool call that emails a real customer is an incident, not a finding, and its escalation path is agreed in advance.

Ready on day one: the AI feature switched on in that environment; credentials for every role that matters, two tenants if the product is multi-tenant; the tools the model can call and the permissions each holds; the system prompts for a grey box test; and a named technical contact.

Reconnaissance: mapping the AI attack surface

Discovery is where anything is first touched, and on AI systems it earns its keep, because the inventory you start from is usually wrong.

We enumerate every entry point that reaches a model (chat window, public API, batch job, webhook, IDE plugin), every model behind them, every retrieval source and the route a document takes into it, and every tool the model can invoke with its scopes. This usually surfaces something the platform team did not know was live: an endpoint still serving a previous system prompt, a cheaper fallback model, an internal assistant wired to production data.

The rest of recon maps where untrusted content enters the context window: uploaded files, scraped pages, inbound email, ticket bodies, calendar invitations, and the third-party servers an agent talks to.

The technical phase: four layers

This is not a web test with a chat box bolted on. Four layers, in order, every finding tagged to the OWASP Top 10 for LLM Applications 2025.

The prompt layer

Prompt injection is LLM01, and it splits in two. Direct injection is a user talking to the model; indirect prompt injection is the model reading instructions out of content somebody else controls. The second scales, because the attacker never touches your login page.

OWASP’s own text is blunt: "Given the stochastic influence at the heart of the way models work, it is unclear if there are fool-proof methods of prevention for prompt injection." So the question is not whether a prompt can be injected, but what it can reach: permissions, not wording. System prompt leakage, LLM07, goes in the same pass.

The retrieval layer

Where the system uses retrieval-augmented generation, the corpus is attack surface: LLM08, Vector and Embedding Weaknesses. We test whether one tenant’s documents surface in another tenant’s answers, and whether a document with instructions buried in it changes behaviour for the next reader.

The tool and agent layer

Where the model can act, impact stops being theoretical. LLM06, Excessive Agency, names three causes: excessive functionality, excessive permissions and excessive autonomy. We take the tool registry apart and ask what each function does with attacker-controlled arguments: object level authorisation inside its own API, request forgery from anything that fetches a URL, injection into anything that builds a query or reads a file. Then we chain them: an agent that can read a ticket and call an internal API is two steps from a data path nobody drew. Prompt injection is how the instruction gets in; the tool registry decides how far it travels.

The model and its output

Improper output handling, LLM05, is where AI features reintroduce vulnerabilities the web solved a decade ago: output rendered as HTML, a model-supplied image URL used as an exfiltration channel, generated SQL run unparameterised. For the model itself we work to NIST’s adversarial machine learning taxonomy, AI 100-2 E2025, which separates evasion, poisoning and privacy attacks on predictive systems from poisoning, direct prompting and indirect prompt injection attacks on generative ones. Unbounded consumption, LLM10, closes the layer: on metered inference, denial of service is also an invoice.

Exploitation and verification

800-115 draws the line that matters: "while vulnerability scanners check only for the possible existence of a vulnerability, the attack phase of a penetration test exploits the vulnerability to confirm its existence". An assistant talked into rudeness is not a finding; one that reads another customer’s invoice is.

Two habits are specific here. Chaining: we hunt the path from content an outsider writes, through a tool call carrying your privileges, to data leaving by a channel we control. Repeatability: model output is stochastic, so every finding is re-run a fixed number of times and reported with the rate at which it worked, ten out of ten or two out of ten. A technique that fires once in fifty is still a finding, and you should know which one. Where the fix belongs in telemetry rather than in the prompt, the report says so instead of proposing another filter.

The report, and the walkthrough

800-115 describes the deliverable as a report that identifies the vulnerabilities found, presents a risk rating and gives guidance on mitigating them. Ours lets three audiences read one document:

• an executive summary a board member reads in five minutes, stating exposure as what an attacker could do, not a count of issues;
• one entry per finding: what, where, the evidence, the payload to reproduce it, the success rate, a risk rating and a remediation that names which tool loses which permission, not "implement input validation";
• the attack narrative in order, so a reader sees how three low-severity issues became one high one;
• a mapping from each finding to the OWASP Top 10 for LLM Applications 2025 and to MITRE ATLAS, the public knowledge base of adversary techniques against AI systems;
• what we tested and could not break, the part an auditor asks about.

It is handed over in a working session with the engineers who have to act on it.

How long it takes

The number is agreed in writing before we start, and retest days sit inside it rather than being added afterwards. What moves it:

• How many entry points reach a model, and whether any of them answer without authentication.
• How many tools the model can call, and what the worst of them can do when the call is attacker controlled.
• Whether retrieval is multi-tenant, because isolation has to be proven per boundary rather than once.
• Grey box or black box, and whether system prompts and tool definitions are shared with us.
• Whether a stable environment exists on day one, because a mid-engagement rebuild absorbs a large part of the window.

The working days go in writing once we have seen the surface.

The standards we work to

The frameworks that change what actually gets done:

• NIST SP 800-115 for the shape of the engagement: planning, discovery, attack and reporting, with a loop from attack back to discovery.
• OWASP Top 10 for LLM Applications 2025 as the finding taxonomy, and NIST AI 100-2 E2025 for terminology.
• OWASP GenAI Red Teaming Guide (January 2025) as a coverage check: model evaluation, implementation testing, infrastructure assessment, runtime behaviour analysis.
• MITRE ATLAS for technique identifiers, and PTES for engagement scaffolding, from pre-engagement interactions to reporting.
• OWASP WSTG 4.2 and the OWASP API Security Top 10 2023 for the surface underneath.

If the system is high-risk under the EU AI Act, Article 15 requires that such systems "shall be resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities", with technical solutions expected to address data poisoning, model poisoning, adversarial examples and model evasion. A pentest is evidence towards that duty, not compliance with it.

What an AI pentest does not cover

• It does not make prompt injection go away. OWASP’s wording above is the reason; the engagement bounds the damage.
• It is not a safety, bias or hallucination evaluation. An assistant that gives a customer confidently wrong advice is a real problem, but not this one.
• It does not audit the model provider. Their pipeline, hosting and staff access sit on their side of the shared responsibility model: for that you get attestations, not our test.
• It does not replace the engagements around it. The web front end, the API layer, the cloud account and the identity plane each need their own test. There are companion walkthroughs for how a web application pentest works and how an API pentest works.
• It is a point in time. 800-115 recommends regular scanning with periodic penetration testing interspersed; here a new model version or an edited system prompt can move the result.
• It is not a certificate. No pentest makes an organisation compliant with anything on its own.

The retest

A finding is not closed because a ticket is closed. Among the responsibilities 800-115 assigns to assessors is "conducting additional examinations and tests when needed to validate mitigation actions". That is the retest.

We re-run every finding as originally reproduced, the same class of finding elsewhere, and the full chain, because a fix in the middle sometimes only moves it. On AI findings we repeat the same fixed number of attempts: a filter that takes a success rate from ten out of ten down to two out of ten has reduced the noise, not fixed the problem, and the report says so.

You get an updated report with a status and a date against each finding, which is what security questionnaires ask for. To apply this to your own architecture, start with an AI penetration testing engagement.