EU AI Act

The EU AI Act classifies AI systems by the risk they present and attaches obligations to each level. Some uses are prohibited outright. A defined set of high-risk uses carries the heaviest duties: risk management, data governance, technical documentation, human oversight, and requirements around accuracy, robustness and security. Systems that interact with people carry transparency duties, and the lightest category carries little. The obligations sit with providers and, in some respects, deployers. For a high-risk system, the security and robustness requirements are not abstract: they call for the system to be tested against the ways it could be manipulated or made to fail, which for a model-backed system means adversarial testing rather than a functional check.

The failure is treating an AI system as ordinary software for compliance and skipping the security work the risk tier demands. A high-risk system deployed without adversarial testing may meet its functional spec and still be trivially manipulated through prompt injection or steered by a connected tool, which is exactly the robustness the regulation asks to be evidenced. The other exposure is on the deployer side: shadow AI, where staff adopt systems outside any governance, makes it impossible to even know which obligations apply, because the organisation cannot document a system it does not know it is using.

Where a system falls into a regulated tier, we test the security and robustness the Act expects and produce evidence of it: LLM red teaming against manipulation, an assessment of what a connected AI agent can actually do, and a check on the data governance behind it. Where the system also processes personal data at scale, this connects to a data protection impact assessment. Findings are framed against both the technical weakness and the obligation. This is part of how we test an AI system against its obligations.