AI Security: A Governance and Compliance Guide to Securing AI Systems

Key takeaways

• AI security means protecting the data, models, pipelines and interfaces of an AI system across its full lifecycle, which is a different discipline from using AI to run your security operations.
• Under the EU AI Act, AI security is no longer optional: Article 15 requires high-risk systems to achieve and maintain accuracy, robustness and cybersecurity, including resilience to adversarial attacks and data poisoning.
• The threats are specific to machine learning: adversarial examples, data and model poisoning, model theft, prompt injection and insecure output handling, catalogued in the OWASP Top 10 for LLM Applications and MITRE ATLAS.
• You cannot secure what you have not inventoried, so an AI system registry and shadow AI discovery are the first control, not an afterthought.
• Organizations that win treat AI security as a governed control set mapped to the EU AI Act, ISO/IEC 42001 and the NIST AI RMF, backed by evidence an auditor will accept rather than a live dashboard.

What is AI security?

AI security is the practice of protecting an AI system and everything it depends on: training and inference data, the model itself, the pipelines that build and serve it, and the interfaces that expose it to users and other systems. It spans the full lifecycle, from data collection through training, deployment and monitoring, to retirement.

One distinction trips up most first conversations, so it is worth drawing plainly. Using AI for security means applying machine learning to defensive work, for example anomaly detection or alert triage. AI security, the subject of this guide, means securing the AI system itself against attacks and failures. IBM and Fortinet both open their treatments with the same split, and it matters because the two require different owners, different controls and different evidence.

Why does the distinction carry weight for compliance? Because regulators now treat the security of the AI system as a named obligation. The EU AI Act binds accuracy, robustness and cybersecurity together in a single article for high-risk systems (EU AI Act Article 15). Once security is a legal requirement rather than an engineering preference, it needs governance: defined controls, assigned ownership and records that prove the controls work.

Why AI security is now a compliance requirement, not just a security problem

For most of the last decade, securing a model was a good idea that lived inside engineering. That has changed. Three instruments now make AI security a documented obligation.

First, the EU AI Act. Article 15 requires providers of high-risk AI systems to design and develop them to reach an appropriate level of accuracy, robustness and cybersecurity, and to keep those properties stable across the system’s lifetime. It names the adversary directly: systems must resist attempts by unauthorized third parties to alter their use, outputs or performance, including adversarial attacks and data poisoning.

Second, ISO/IEC 42001. The 2023 standard is the first certifiable AI management system, and its Annex A controls fold AI security, data governance and adversarial-attack mitigation into a management system an external body can audit (ISO/IEC 42001). It gives security a home inside a governed cycle rather than a backlog.

Third, the NIST AI Risk Management Framework. NIST organizes the work into four functions, Govern, Map, Measure and Manage, and the Manage function is where security controls and treatments live (NIST AI RMF). Read together with the EU AI Act cross-map in our frameworks guide, it turns security into a repeatable, evidenced routine.

The gap between obligation and capability is wide. In one Microsoft assessment of 28 organizations, 25 lacked the right tools to secure their AI systems. Compliance is what closes that gap, because it forces inventory, control selection and evidence rather than leaving security to whoever remembers to ask.

The AI security threat landscape

Securing an AI system starts with knowing what you are defending against. AI attacks do not map cleanly onto traditional application security, because the model, the data and the prompt are all attack surfaces. It helps to group threats by what they target. The MITRE ATLAS knowledge base catalogues these techniques in an ATT&CK-style matrix, and our MITRE ATLAS guide walks the mapping from technique to control.

Attacks on the model

Adversarial examples are inputs crafted to make a model misclassify while looking normal to a human. Evasion attacks use them at inference time to slip past a classifier. Model extraction, or model theft, queries a deployed model enough times to reconstruct a functional copy, stealing intellectual property and creating a shadow model an attacker can probe offline.

Attacks on the data

Data poisoning corrupts the training set so the model learns the wrong thing, from subtle accuracy degradation to hidden backdoors that trigger on a chosen input. Membership inference attacks work in the other direction, testing whether a specific record was in the training data, which can expose personal or confidential information.

Attacks on the application layer

For systems built on large language models, the application layer is now the softest target. The OWASP Top 10 for LLM Applications is led by prompt injection, where an attacker hides instructions in a prompt, a document or a tool output to override the model’s guardrails, leak data or trigger actions through connected tools. Insecure output handling, where a model’s response is trusted and passed downstream without validation, turns a language bug into a system compromise.

Supply chain and agentic risks

Models, datasets and libraries arrive from third parties, so a poisoned open-weight model or a compromised dependency is a supply chain risk. Agentic systems, which can call tools and act on their own outputs, widen the blast radius: a single successful injection can chain into real-world actions. Our autonomous AI agents guide covers the governance controls these systems need.

You cannot secure what you cannot see: AI inventory and shadow AI

Every control below assumes you know which AI systems you run. Most organizations do not. Teams adopt copilots, embed third-party models and wire up agents faster than any central function can track, producing shadow AI: AI in use that governance has never seen.

Shadow AI is a security problem before it is a compliance problem. An unrecorded model has no owner, no threat assessment, no monitoring and no incident path. So the first control in any AI security program is not a firewall, it is an inventory. A living registry of AI systems, their data, their models and their risk tier is what makes every later control possible. Our shadow AI guide explains how hidden AI use becomes a governance gap, and why discovery is control zero.

Mapping AI security to frameworks: EU AI Act, ISO 42001, NIST AI RMF

The move that no competitor page makes is connecting each threat to the control that answers it and the clause that requires it. This is what turns AI security from a shopping list of tools into a defensible compliance position.

Build the map once and every audit becomes a lookup rather than a scramble. The ISO 42001 explainer and the NIST AI RMF guide go deeper on each side, and the EU AI Act operator’s guide covers where Article 15 sits among the wider high-risk obligations.

From threats to auditable controls

A threat you cannot evidence is a threat you cannot defend in an audit. So each threat class should resolve into a named control with an owner and a record.

• Adversarial and evasion attacks: scheduled adversarial testing and red-teaming, with results logged and re-run when the model changes.
• Data poisoning: data lineage, provenance checks and integrity controls on the training pipeline.
• Model theft: rate limiting, query monitoring and access control on inference endpoints.
• Prompt injection and insecure output: input and output validation, tool-use allowlists, and isolation between the model and downstream actions.
• Drift and decay: continuous monitoring of performance and behavior, with thresholds that raise an alert.

The point of writing controls this way is not the list, it is the evidence. A dashboard shows a state now; an auditor needs a record over time. Each control should emit a dated artifact: a test report, a monitoring log, an access review. Our auditability guide sets out what makes an AI system auditable and how to prove it, and continuous compliance monitoring is how those artifacts accrue without a manual scramble before each review.

Who owns AI security? Roles and governance

AI security fails when everyone assumes someone else owns it. Traditional cybersecurity sits with the CISO, but AI security spans data science, MLOps and compliance, so ownership has to be explicit.

A workable split assigns accountability to a named AI governance lead or committee, with the CISO responsible for the security controls, data science responsible for model testing, and compliance responsible for evidence and framework mapping. A short responsibility matrix, agreed once, prevents the common failure where a model ships with no security owner. This governance layer is the subject of our AI governance operating system guide, which shows how these roles connect to the wider control environment.

How to operationalize an AI security program

Turning the ideas above into a running program is a sequence, not a project with an end date.

1. Inventory every AI system, including shadow AI, and assign each a risk tier.
2. Run a threat and risk assessment per system, using the OWASP LLM Top 10 and MITRE ATLAS as checklists.
3. Select controls from your framework mapping and assign each an owner.
4. Test the controls: adversarial testing, red-teaming and validation, sized to the risk tier.
5. Capture evidence as dated artifacts for each control.
6. Monitor continuously for drift, abuse and new vulnerabilities.
7. Wire in incident handling, including regulatory reporting where required. The EU AI Act sets serious-incident reporting duties, covered in our Article 73 incident reporting guide.

Run the sequence as a loop. Each new model re-enters at step one, and each material change re-triggers testing and evidence.

FAQ

What is AI security in simple terms?

AI security is protecting an AI system, its data, its model, its pipelines and its interfaces, from attacks and failures across the system’s lifecycle. It is distinct from using AI for security work such as threat detection. In a compliance context it also means meeting named obligations, such as the EU AI Act Article 15 requirement that high-risk systems maintain accuracy, robustness and cybersecurity.

What is the difference between AI security and cybersecurity?

Traditional cybersecurity protects networks, applications and endpoints. AI security adds attack surfaces that classic controls do not cover: the training data, the model’s learned behavior and the prompt. A firewall does not stop data poisoning or prompt injection. AI security therefore extends cybersecurity with model-aware controls such as adversarial testing, data lineage and output validation, and ties them to AI-specific frameworks.

What are the main AI security threats?

The recurring classes are adversarial examples and evasion attacks on models, data and model poisoning, model theft or extraction, membership inference, prompt injection and insecure output handling at the application layer, and supply chain risks from third-party models and libraries. The OWASP Top 10 for LLM Applications and MITRE ATLAS are the two reference catalogues most teams use to confirm they have not missed a category.

Does the EU AI Act require AI security?

Yes. Article 15 requires providers of high-risk AI systems to achieve and maintain an appropriate level of accuracy, robustness and cybersecurity, and to keep those properties stable across the lifetime of the system. It explicitly names resilience to adversarial attacks and data poisoning. Security is therefore a documented obligation for high-risk systems, not an optional engineering practice.

How do ISO 42001 and NIST AI RMF relate to AI security?

ISO/IEC 42001 is a certifiable AI management system whose Annex A controls include AI security, giving security a governed, auditable home. The NIST AI RMF organizes risk work into Govern, Map, Measure and Manage, with security treatments living in the Manage function. Used together, they let you map each security control to a recognized clause, which is what makes an audit a lookup rather than a scramble.

Where should an AI security program start?

Start with inventory. You cannot secure, test or attest an AI system you have not recorded, and most organizations run shadow AI that governance has never seen. Build a living registry of AI systems with their data, models and risk tiers, then run threat assessments, select framework-mapped controls, test them, and capture dated evidence. Inventory is control zero; everything else depends on it.

Conclusion

AI security has moved from an engineering nicety to a named compliance obligation, and the pages that rank for the term have not caught up: they describe threats and sell tools, but they do not connect security to the frameworks that now require it. Organizations that get this right treat AI security as a governed control set, mapped to the EU AI Act, ISO 42001 and the NIST AI RMF, starting from an honest inventory and ending in evidence an auditor will accept. AI Sigil is built to run exactly that loop, from AI discovery to control mapping to audit-ready evidence. See how the AI governance operating system turns AI security into a defensible, continuous capability.