Key takeaways
- AI red teaming stopped being optional in Europe on 2 August 2025, when
Article 55of the EU AI Act began applying to providers of general-purpose AI models with systemic risk. - Enforcement is live: the AI Office gained penalty powers on 2 August 2026, with fines reaching 15 million euros or 3 percent of worldwide annual turnover.
- The high-risk regime moved the other way. The Digital Omnibus approved on 16 June 2026 pushed standalone Annex III duties to 2 December 2027.
- Most organisations are not GPAI providers, so their exposure arrives contractually, through vendor due diligence and customer assurance questions.
- An undocumented exercise proves nothing. What assessors read is the AI red teaming file: scope, threat model, test log, severity triage, remediation owners and re-test.
What AI red teaming actually is, and what it is not
AI red teaming is structured adversarial testing of an AI system, run by people whose job for the duration is to make it fail. The team writes prompts designed to bypass guardrails, poisons a retrieval index, chains tool calls the designer never anticipated, and records what the system does. The goal is not a score. The goal is a list of things the system does that nobody intended.
Read the first page of Google results and you would conclude this is a security discipline and nothing more. Every ranking page describes attack techniques, methodology phases and tooling. That description is accurate and incomplete, because it omits the part that now decides budget: in the European Union, this activity is a documented legal duty for one class of provider, and a documented expectation for several others.
Two boundaries matter before going further.
Red teaming, benchmarking and evaluation are three different things
Benchmarks measure known capabilities against a fixed question set. Evaluations score a system against criteria you defined in advance. Both answer the question “how well does it do the thing we asked about”. Neither answers “what else does it do”.
AI red teaming exists for the second question. IBM Research puts the purpose plainly: the exercise is about overcoming the unknown unknowns. A benchmark suite cannot surface a failure mode nobody thought to write a test for, which is precisely the category that produces incidents.
The practical consequence is that these are complements, not substitutes. If you already run AI benchmarking and publish LLM benchmark results, you have measured capability. You have not yet probed for harm.
It is also not a penetration test with a new label
A penetration test targets infrastructure: the network, the API gateway, the identity layer. Those tests still apply and still matter. AI red teaming targets the model and its behaviour, which fails differently. There is no patch for a model that can be talked into something. Remediation looks like fine-tuning, filtering, refusal policies, permission narrowing and monitoring, not a version bump.
The obligation the search results do not mention
Article 55(1)(a) of the EU AI Act requires providers of general-purpose AI models with systemic risk to perform model evaluation in accordance with standardised protocols and tools reflecting the state of the art, including conducting and documenting adversarial testing of the model, with a view to identifying and mitigating systemic risk. The regulation adds that the testing must be proportionate to the level of risk and the state of the art, and that it may involve independent external experts.
Three words in that sentence carry the weight. “Conducting” makes the exercise mandatory. “Documenting” makes the record mandatory. “Proportionate” means a small provider cannot be held to a frontier lab’s programme, and equally that a frontier lab cannot discharge the duty with a weekend of prompt testing.
A model falls into the systemic-risk class when its cumulative training compute exceeds 10^25 FLOPs, or when the AI Office designates it on other criteria. That threshold is deliberately high. It captures frontier developers, not the median enterprise.
The dates are where this gets interesting. The GPAI obligations began applying on 2 August 2025 for models placed on the market after that date. Models already on the market have until 2 August 2027. And on 2 August 2026, the Commission’s AI Office gained its enforcement powers, with penalties up to 15 million euros or 3 percent of worldwide annual turnover, whichever is higher.
Meanwhile the regime everyone spent two years preparing for slipped. The European Parliament approved the Digital Omnibus amendments on 16 June 2026, moving standalone Annex III high-risk obligations from 2 August 2026 to 2 December 2027, and product-embedded high-risk systems to 2 August 2028. The omnibus left the general-purpose dates untouched.
So the position today is the opposite of what most compliance calendars assume. The AI Act testing duty that is enforceable right now is the general-purpose one, and adversarial testing sits at the centre of it. Our operator’s guide to the EU AI Act sets out how the rest of the timeline now falls.
The operational route is the GPAI Code of Practice. Its Safety and Security chapter, published on 10 July 2025, defines the framework providers are expected to maintain: model evaluations, red-teaming, post-market monitoring, cybersecurity measures, incident reporting and model accountability. Signing is voluntary, but the AI Office has said signatories are treated as acting in good faith, and their commitments are weighed when a penalty is calculated. That is as close to a safe harbour as the Act offers, and it is built on evidence of testing. The detail of who counts as a general-purpose provider is covered in our general-purpose AI guide.
Who actually has to red team
The honest answer for most readers is “not you, directly”. That is worth saying clearly, because the vendor content on this query is written to imply universal obligation.
Providers of GPAI models with systemic risk. Bound by Article 55, enforceable now, penalties attached. This is a short list of organisations and they know who they are.
Providers of high-risk AI systems. Article 9 requires a risk management system with testing throughout the lifecycle, and Article 15 requires systems to perform reliably and to resist errors, faults and attempts to alter their use or performance. Neither article uses the words “red teaming”. Both require testing against conditions that challenge intended operation, with the methodology and results documented. Applies from 2 December 2027 for standalone Annex III systems.
Deployers. No direct statutory testing duty. The obligation arrives through contract instead: procurement questionnaires, assurance schedules, audit rights and warranty terms. In practice this is how AI red teaming reaches the majority of organisations, and it usually arrives as a question you have thirty days to answer. Structuring that answer is a vendor due diligence problem before it is a technical one.
Everyone else. Voluntary, and increasingly expected. Insurers ask. Enterprise buyers ask. Boards ask after the first public incident in their sector.
One more group deserves naming: organisations whose AI estate is not fully known. You cannot red team what you have not inventoried, and shadow AI is the reason many testing programmes cover a fraction of the real surface.
What each framework actually asks for
The frameworks converge on the activity and diverge sharply on how binding it is.
| Framework | What it requires | What it calls it | Binding? |
|---|---|---|---|
EU AI Act Art. 55(1)(a) | Model evaluation including adversarial testing, documented, proportionate to risk | Adversarial testing | Yes, for GPAI with systemic risk, enforceable since 2 Aug 2026 |
EU AI Act Art. 9 and Art. 15 | Lifecycle testing against challenging conditions, documented methodology and results | Testing | Yes, for high-risk, from 2 Dec 2027 |
| GPAI Code of Practice, Safety and Security | Model evaluations, red-teaming, post-market monitoring, incident reporting, model accountability | AI red teaming | Voluntary, but weighed in penalty decisions |
| NIST AI RMF, Measure 1.1 | Red-teams and adversarial testing to surface dangerous capabilities and emergent properties | AI red teaming | Voluntary |
| NIST AI 600-1, Generative AI Profile | Red teaming before and after deployment across twelve risk categories | AI red teaming | Voluntary |
ISO/IEC 42001 | No named red-teaming clause; testing and evaluation records support the management system | Testing and evaluation | Certifiable |
| OWASP GenAI Red Teaming Guide | Phased testing across model, implementation, system and runtime | AI red teaming | Practitioner standard |
| CSA and OWASP Agentic AI Red Teaming Guide | Twelve agentic threat categories on the MAESTRO threat model | AI red teaming | Practitioner standard |
Two observations follow from that table.
First, ISO/IEC 42001 never says “red team”, which surprises people who arrive expecting a clause reference. What the standard wants is evidence that you identified risks, tested treatments and reviewed results. A red team report with a remediation trail satisfies that in a way a capability benchmark does not. Building the mapping once, across all of these, is the point of a governance framework activation rather than a per-standard scramble.
Second, the American and European instruments describe the same practice with different force. The Berkeley CLTC profile for general-purpose AI places AI red teaming under NIST AI RMF Measure 1.1, to surface dangerous capabilities, vulnerabilities and emergent properties, and repeats it as a lifecycle control under Manage 1.3, 2.3 and 2.4. NIST AI 600-1, the Generative AI Profile, recommends AI red teaming both before and after deployment across twelve risk categories. Neither carries a fine. Both are what a European assessor will accept as “state of the art” when Article 55 asks what standard you tested to.
Scoping so the findings hold up
Most AI red teaming reports fail at scoping, not at testing. The exercise finds real problems, and then nobody can say which system version they applied to.
Name the target precisely. Base model, fine-tuned variant, retrieval layer, tool and agent surface, deployed application. State which are in scope and which are not, and give the reason. A test of the base model does not cover the product you shipped on top of it.
Version-pin everything. Model build, system prompt version, retrieval index snapshot, guardrail configuration, tool permissions. A finding against an unnamed build cannot be re-tested, and an untestable finding is not evidence.
Define the harms from your own taxonomy. Generic lists produce generic findings. If your risk register says the top exposures are unauthorised financial commitment and disclosure of client data, those go in scope first, and the report should say why the others were deprioritised. That reasoning is also where your risk appetite becomes visible to an assessor.
Decide on independence deliberately. Article 55 explicitly contemplates independent external experts. Internal teams know the system better and cost less. External teams are harder to dismiss and carry more evidentiary weight. Write down which you chose and why, because the choice will be questioned.
Extend scope for agentic systems. Single-turn prompt testing is inadequate once a system has autonomy, memory and tools. The Cloud Security Alliance and OWASP guide organises this around the MAESTRO threat model and twelve categories, among them agent authorisation and control hijacking, checker-out-of-the-loop, goal manipulation, memory and context manipulation, multi-agent exploitation, impact chain and blast radius, and agent untraceability. That last one matters more than it sounds: if an agent’s actions cannot be attributed after the fact, you have an accountability failure before you have a security one.
The evidence a red team has to leave behind
This is the part the search results skip entirely, and it is the part that decides whether the money was well spent. Picture the file an assessor opens eighteen months from now.
Scope statement. Dated, version-pinned, with exclusions and their justification.
Threat model. The reference framework named explicitly: MITRE ATLAS, MAESTRO, the OWASP GenAI taxonomy, or your own, mapped to one of them.
Test log. What was attempted, by whom, with which tooling, on which date, against which build. Automated runs get their configuration recorded, not just their output.
Findings with severity and a triage rationale. The rationale matters more than the rating. An assessor can disagree with “high” and still accept the file if the reasoning is legible.
Remediation decisions with an owner and a date. Every finding needs a disposition: fixed, mitigated, accepted, or deferred with a review date.
Residual risk accepted at the right level. Someone with authority signs that the remaining exposure is tolerable, against a written threshold.
Re-test evidence. Proof that the fix works, against the same test, on the named build. Without this the remediation column is an assertion.
Two failure states are worth naming. An undocumented AI red teaming exercise is not evidence of anything, however good the testing was. And a red team report with findings and no remediation trail is worse than no report at all, because it is durable proof that you knew. Regulators and litigants both read that document the same way.
The results belong in the technical file, not in a drive folder. What goes into the Annex IV documentation set is covered in our guide to AI system documentation requirements.
Where AI red teaming plugs into the rest of the governance file
AI red teaming that terminates in a PDF has failed, regardless of what it found. Four connections make it operational.
The risk register. Findings become register entries with owners and review dates, in the same system as every other risk. If red team findings live in a separate security tracker, the governance file has a hole where the link should be.
Incident reporting. A finding that later materialises in production becomes an Article 73 serious-incident question, and the first thing anyone will ask is whether you knew. Our AI incident reporting guide covers the reporting clocks.
Post-market monitoring. AI red teaming is a point-in-time probe. Monitoring is the continuous half, and the Code of Practice names both. An AI red teaming programme with one and not the other is half a programme.
Transparency reporting. The Berkeley profile routes red-team output into model cards and system cards, while responsibly omitting details that would raise misuse potential. That caveat is not optional: publishing a working jailbreak is a disclosure failure, not transparency.
Five failures that void the evidence
Testing one build and shipping another. The most common defect. The report describes a system that no longer exists.
Scoping to prompt injection only. Prompt injection is the famous failure mode, not the expensive one. Tool permissions and agent autonomy produce the incidents with financial consequences.
No severity model. When everything is medium, nothing is prioritised, and the triage rationale an assessor wants to read does not exist.
Findings without owners. An unassigned finding is a note. It will still be open at the next review, and the dates will prove it.
No re-test. The fix is asserted rather than demonstrated. This is the single most common gap in otherwise competent AI red teaming programmes, and the cheapest one to close.
FAQ
Is AI red teaming legally required?
In the European Union, yes, for one specific group. Article 55(1)(a) of the EU AI Act requires providers of general-purpose AI models with systemic risk to conduct and document adversarial testing, and those obligations have been enforceable since 2 August 2026. For providers of high-risk systems, Article 9 and Article 15 require documented testing against challenging conditions from 2 December 2027, without using the term “red teaming”. For everyone else it is voluntary in law and increasingly mandatory in contracts.
Who has to do AI red teaming under the EU AI Act?
Providers of general-purpose AI models whose cumulative training compute exceeds 10^25 FLOPs, or which the AI Office designates as carrying systemic risk. That is a small population of frontier developers. Deployers of those models inherit expectations through contract rather than statute, which is how the requirement reaches most organisations.
What is the difference between AI red teaming and penetration testing?
A penetration test attacks infrastructure and produces vulnerabilities with patches. AI red teaming attacks model behaviour and produces failure modes with no patch. Remediation is fine-tuning, filtering, refusal policy, permission narrowing or monitoring. Both are needed, and neither substitutes for the other.
How often should you red team an AI system?
Tie the cadence to change rather than to the calendar. A new base model, a fine-tune, a new tool or integration, a material system-prompt change, or a new deployment context each warrant a fresh exercise. Add a periodic baseline, annually for most systems, and treat post-deployment testing as separate from pre-deployment testing, as NIST AI 600-1 recommends.
Can AI red teaming be automated?
Partly, and the split matters for evidence. Automated probing gives breadth, repeatability and a re-test mechanism, which is exactly what the documentation needs. Human red teamers find the novel path that no probe library contains. A programme built only on automation will produce a tidy file with a predictable blind spot.
What does an auditor ask for after a red team exercise?
The scope statement with versions, the threat model and its reference framework, the test log, findings with severity and rationale, remediation owners and dates, the residual-risk acceptance and who signed it, and re-test evidence. Auditors assess the trail, not the exercise, because the trail is the only part that survives.
Conclusion
The search results treat AI red teaming as a security service, and on that reading the decision is a budget line owned by a security team. The regulatory position says something different. Adversarial testing is the AI Act obligation that is enforceable today, while the high-risk regime everyone prepared for sits more than a year out, and the thing that discharges the duty is not the exercise but the record it leaves.
That reframing changes who owns the work. Testing stays with the people who are good at breaking systems. The scope statement, the severity rationale, the remediation ledger and the residual-risk signature belong with governance, because those are the artefacts anyone will ever read. Start with the file you would need to produce, then commission the AI red teaming that fills it. See how the wider obligations fit together in our AI regulatory landscape overview.