Zooming in on AI Cybersecurity Requirements for AI Systems
The Artificial Intelligence Act (AI Act) is the world’s first comprehensive legal framework for AI regulation, which entered into force on August 1, 2024. The AI Act aims to ensure that AI systems are trustworthy, safe, and respect fundamental rights and values. As part of this objective, the AI Act imposes specific requirements for high-risk AI systems, which are those that pose significant risks to health, safety, or fundamental rights in certain sectors or use cases. One of these requirements is to achieve an appropriate level of cybersecurity, meaning that high-risk AI systems must be resilient against malicious attacks that could alter their use, outputs, or performance.
However, cybersecurity is not only relevant for high-risk AI systems but for all AI systems that process data, interact with users, or influence physical or virtual environments. This article explains what the AI Act requires for high-risk AI systems in terms of cybersecurity and why cybersecurity should be considered across all AI systems, irrespective of their risk level.
Cybersecurity Requirements for High-Risk AI Systems under the AI Act
According to Article 15 of the AI Act, high-risk AI systems must be designed and developed to achieve an appropriate level of accuracy, robustness, and cybersecurity, performing consistently throughout their lifecycle. This means that high-risk AI systems must be resilient to errors, faults, or inconsistencies that may occur within the system or its operating environment and must be protected against unauthorized attempts to exploit system vulnerabilities.
The AI Act provides guidance on the technical aspects of measuring and ensuring the appropriate levels of accuracy and robustness. It encourages the development of benchmarks and measurement methodologies in cooperation with relevant stakeholders such as metrology and benchmarking authorities. The levels of accuracy and relevant metrics for high-risk AI systems must be declared in the accompanying instructions for use.
Technical measures to ensure the cybersecurity of high-risk AI systems should be appropriate to the relevant circumstances and risks. Possible solutions include:
- Technical redundancy solutions, including backup or fail-safe plans.
- Measures to prevent, detect, respond to, resolve, and control attacks, including those carried out using:
- Data poisoning: Threat actors manipulate training data.
- Model poisoning: Threat actors manipulate pre-trained components used in training.
- Model evasion: Threat actors manipulate input data to trick the model into unintended actions.
Risk Assessments for High-Risk AI Systems
The AI Act requires providers of high-risk AI systems to conduct a risk assessment before placing the system on the market or putting it into service. This assessment must document potential risks to health, safety, and fundamental rights, alongside measures taken to prevent or mitigate those risks. It must also consider cybersecurity risks and measures to ensure resilience against malicious attacks.
Risk assessments must be updated regularly throughout the AI system’s lifecycle, and the technical documentation must be made available to competent authorities upon request. Providers must ensure their quality control and assurance processes for high-risk AI systems create and record this documentation appropriately, expecting it to be disclosed in any subsequent enforcement action.
The Cyber Resilience Act (CRA)
The cybersecurity of AI systems is also influenced by the Cyber Resilience Act (CRA). The CRA imposes several cybersecurity requirements on “products with digital elements” (i.e., connected products including Wi-Fi routers and IoT devices, as well as certain forms of software). Key requirements include:
- Protecting against unauthorized access through tools like authentication and identity management.
- Minimizing the collection of data to only what is adequate and relevant for the device’s intended use.
The CRA also contains specific provisions for high-risk devices, as defined under the AI Act. Connected devices containing AI models that are in-scope of the CRA and fulfill its security-by-design requirements will be deemed compliant with the cybersecurity requirements of the AI Act.
Why Cybersecurity Should Be Considered Across All AI Systems
The AI Act sets specific cybersecurity requirements for high-risk AI systems in the EU but will also provide influential guidance on reasonable cybersecurity standards for riskier AI solutions elsewhere. Importantly, all AI systems that process data, interact with users, or influence environments are exposed to cybersecurity threats and should be designed with security in mind.
Cybersecurity is not just a compliance issue; it encompasses trust, reputation, and competitiveness. Cyberattacks against AI systems can have serious consequences, including:
- Compromising the confidentiality, integrity, or availability of data.
- Causing harm or damage to users or third parties.
- Undermining the performance or reliability of the AI system.
- Violating fundamental rights or ethical principles.
Additionally, cyberattacks can erode the trust and confidence of users, customers, and stakeholders, damaging the provider’s reputation and market position. Cybersecurity is a dynamic concept that requires constant monitoring, updating, and improvement in response to evolving threats and technological advancements.
Best Practices for AI System Providers
Providers of AI systems should adopt a risk-based security-by-design and security-by-default approach. This includes:
- Early risk assessment processes to identify potential security risks.
- Integrating security into the AI system’s design and development process, ensuring default settings provide the highest level of security.
Furthermore, because the safety and security of AI products are mostly defined during the design and development phase, deploying quality control and assurance processes and retaining documentation is essential for demonstrating adequate cybersecurity risk management.
Providers should also:
- Conduct regular risk assessments.
- Implement appropriate technical and organizational measures.
- Follow best practices and standards to ensure their AI systems’ cybersecurity.
- Comply with existing laws and regulations related to cybersecurity, such as the General Data Protection Regulation (GDPR) and the Digital Operational Resilience Act (DORA).
Additionally, cooperation with relevant authorities and stakeholders, such as the European Union Agency for Cybersecurity (ENISA), which offers guidance and support on cybersecurity policy and AI system-related issues, is encouraged.
Conclusion
Cybersecurity is a key requirement for high-risk AI systems under the AI Act, but it is also crucial for all AI systems that process data, interact with users, or influence environments. It is not merely a compliance issue but a matter of trust, reputation, and competitiveness. Providers of AI systems should integrate security into their design processes, conduct regular risk assessments, and comply with existing regulations to contribute to the development of trustworthy, safe, and respectful AI.
