Cybersecurity and artificial intelligence: what actually changes

What changed for the attacker: the cost, not the technique

It is worth starting by taking noise out, because this is where there is most of it.

Generative AI has not invented new attacks. What it has done is drop the cost of deception to almost nothing, and that does have practical consequences:

• Bad spelling is finished as a warning sign. An email in flawless English, in the tone of your sector and with the name of the person you deal with, no longer costs time and no longer requires speaking the language. If your internal training still teaches people to spot phishing by the typos, it is teaching something that no longer discriminates.
• Cloning a recognisable voice is cheap. And that breaks the informal control half the company leaned on, which was to phone and confirm. Recognising a voice has stopped being a verification.
• Personalisation is free. The email aimed at one specific person no longer requires the work that used to make it rare.

What has not changed is the way in: a valid credential, an unpatched system, and somebody who clicks. Anybody selling you that AI has changed the nature of the attack is selling you a product.

What is genuinely new: your own AI is attack surface

This is the part that matters and the part that almost nobody looks at. The moment a company connects an assistant to its documentation, to its database or to its tools, it has built a system that:

• takes input from outside,
• has access to data on the inside,
• and can carry out actions.

That is exactly the description of something you have to test before you expose it. And it has an uncomfortable property that no earlier system had: for a language model, data and instructions are the same thing. There is no technical boundary separating "this is content you should summarise" from "this is an order you should obey". All the security of these systems comes out of that.

The reference framework that orders these risks is the OWASP Top 10 for LLM Applications, which is to the world of models what the classic Top 10 is to the web.

Prompt injection, and why the direct kind is the least interesting

Prompt injection is what makes every headline, almost always in its boring version: a user types "ignore your instructions" into the chat and the model does something it should not. That is the jailbreak, and as a company problem it is a minor one, because the user is deceiving himself inside his own session.

The one that matters is the indirect kind. Here the instruction is not written by the user: it arrives inside content that the system processes on its own. An email the assistant summarises. A page it consults. A document somebody outside uploaded to the shared folder. A support ticket a customer opens.

The content carries a hidden instruction, the model reads it as if it came from its owner, and acts. The victim has done nothing, has clicked nothing and does not find out. It is the difference between somebody deceiving his own assistant and somebody deceiving the assistant of another.

System prompt leakage comes out of the same place, and it is usually the first step: knowing the instructions and the tools of the assistant in order to know what can be asked of it.

The real problem: not the model, but what you let it do

A prompt injection on its own makes a model write something odd. Annoying and little else.

It becomes an incident when the model has hands. And that is where companies are today: assistants connected through MCP or other mechanisms to email, to repositories, to databases, to the ticketing tool or to the payment gateway.

The flaws we find are almost never in the model. They are in the design around it:

• Excessive agency. The assistant can do more than its function needs. It was given write access because that was easier than deciding which of the twenty actions were required.
• The permissions are the assistant’s, not the user’s. It is the most repeated flaw and the most expensive: the assistant queries the database with a service account that sees everything, so any user can reach data that is not theirs simply by asking. It is not an AI problem, it is a broken access control with a new interface.
• The tool registry. If a third party can add or describe the available tools, it can steer the model towards the one that suits it. That is tool poisoning, and it is a supply chain problem under another name.
• Actions with effect that ask for no human confirmation. Send, delete, pay, publish.

The practical conclusion: treat your assistant as a user who can be deceived, not as a trusted component. Least privilege, its own credentials, confirmation for anything that has effect, and a log of what it does.

The data: RAG, shadow AI and poisoning

The usual way for an assistant to know your company is RAG: documentation gets indexed and the model queries that index when it answers. It is the right approach and it brings two concrete problems.

The first one is about permissions. Indexing a folder usually means indexing everything in it, and the permissions of the original document rarely travel with the indexed fragment. The result is a search engine that answers anybody about documents that not everybody could open. It gets discovered the day somebody asks about the payroll.

The second one is that the index is an entry surface. If a third party can put content into something that gets indexed, they have put in text that the model will treat as trusted context. That is where indirect injection and data poisoning meet.

And then there is what happens with nobody’s permission: shadow AI. People pasting contracts, code or customer data into tools that the company has neither approved nor reviewed. Banning it by memo never works. What works is giving people an approved alternative that is actually usable, and then the ban holds up on its own.

The regulation: the EU AI Act

The EU AI Act, the artificial intelligence regulation of the European Union, sorts systems by risk level and places different obligations depending on where each one falls. What is worth knowing without going into deadlines or articles, because that is for an adviser with your case in front of them:

• It reaches whoever uses, not only whoever builds. Integrating the model of a third party into your product puts you inside it.
• The first thing is knowing what you have. An inventory of the AI systems in use, including the ones that arrived inside another product without anybody deciding, which tend to be the majority.
• And it overlaps with what already applies to you. If the system processes personal data, the GDPR is still there and goes this way; if you are one of the entities that NIS2 reaches, your AI systems are part of your surface like everything else.

What an AI pentest actually tests

An AI pentest is not about getting the model to say something outrageous. That is a demonstration, not a finding. What gets tested is the system around the model:

• Indirect injection through every input channel. Email, documents, pages, tickets, anything the assistant reads without a human having approved it.
• Authorisation. Whether a user can obtain, by asking, data that their account cannot open. It is the most frequent finding and the most serious.
• The limit of the tools. What actions can be triggered and with what permissions they run.
• Leakage of instructions and of context, including keys and internal paths that end up in the system prompt.
• The RAG index, checking whether it respects the permissions of the original document.
• And the boring part, which is still what comes up most: the API that serves the assistant is an API, and everything that happens to APIs happens to it.

When what you want is to evaluate the behaviour of the model itself rather than the system, that is LLM red teaming, which is a different and complementary exercise.

The whole project, phase by phase, is written up in how an AI pentest works. If this is the first thing you are reading, it is worth starting with what penetration testing is, and for a specific scope, get in touch.