Taming Generative AI: Regulation, Reality, and the Road Ahead

The rise of generative AI has unlocked unprecedented creative potential, but it’s also raised critical questions about regulation and accountability. As algorithms blur the lines between human and machine-generated content, upcoming legislation like the EU AI Act aims to establish clear boundaries and ensure transparency. However, achieving meaningful oversight presents a complex challenge, forcing us to confront fundamental questions about how we classify, detect, and manage artificially created realities. Our research delves into the practical realities of this evolving landscape, analyzing current practices and dissecting the hurdles that stand in the way of effective regulation.

What are the key characteristics of the current landscape of generative AI systems?

As generative AI continues its rapid evolution, several key characteristics define the current landscape, impacting both its potential and the challenges it poses.

Core Technological Insights

Generative AI models, particularly text-to-image (TTI) systems, have achieved remarkable photorealism, often making it difficult even for experts to distinguish AI-generated content from human-created images. This leap in quality stems from advancements in deep learning techniques and the ability to learn from vast, unlabeled datasets. These models analyze patterns in training data to generate new content with similar characteristics.

However, this capability comes at a significant cost. Training these advanced models requires immense computational resources and access to massive datasets. Consequently, development is concentrated among a select few large tech companies and well-funded academic institutions.

Regulatory and Compliance Concerns

The EU AI Act is the most comprehensive regulatory initiative to date that tackles the risks posed by AI-generated content. Its key requirements include:

Machine-readable watermarking: All outputs from generative AI systems must be marked in a machine-readable format, detectable as artificially generated or manipulated. Interoperability, robustness, and reliability are key considerations.
Visible deepfake disclosures: Deployers of generative AI systems that create or manipulate image, audio, or video content constituting a “deep fake” must clearly disclose its artificial origin. This information must be provided at the time of first interaction or exposure.

Non-compliance can result in substantial fines, up to €15 million or 3% of global annual turnover. The AI Act’s transparency rules will take effect on August 1, 2026.

Practical Implications and Challenges

Despite the impending regulations, watermarking implementation remains limited. An analysis of 50 widely used generative AI systems reveals several concerning trends:

Inconsistent Watermarking: Implementation is inconsistent and varies significantly across different providers and deployment methods.
Metadata Vulnerability: Embedding metadata is the most common approach, yet it’s easily removable, undermining its effectiveness as a robust marker.
End-to-End System Focus: Machine-readable marking practices are primarily found in end-to-end systems (Category 1 systems) and large-scale providers (Category 2 systems). This suggests that large organizations are mostly concerned with data provenance or copyright, not necessarily with detecting “non-truth.”
Visible Markings Avoidance: Visible watermarks remain rare, largely due to concerns about user experience and potential impact on business models. When used, they often apply to all generated images, not specifically to deep fakes, as required by the AI Act. Most visible markings can also be removed after payment.

Implementing appropriate disclosure for deep fakes presents a distinct challenge. It requires a separate system, likely NLP-based, to classify prompts as deep fakes. Smaller organizations would find this difficult and computationally expensive to implement.

The ecosystem is also dominated by a few major players. Many system providers rely on a handful of models, primarily from Stability AI, Black Forest Labs, and OpenAI. Although these model providers strive to incorporate watermarking, these watermarks are often easily disabled or are not applied consistently across API-based systems. Hugging Face should enforce watermarking requirements for the open-source models hosted by their platform.

The AI Act introduces ambiguities surrounding the roles of providers, deployers, models, and systems, potentially impacting compliance effectiveness and dividing responsibility across the complex generative AI supply chain.

How do the transparency obligations of the EU AI Act apply in the context of watermarking and disclosure for AI-generated content?

The EU AI Act mandates transparency for AI-generated content through two key measures: embedding machine-readable markings in AI-generated outputs and visibly disclosing the artificial origin of AI-generated deep fakes. Non-compliance can result in significant fines, up to 3% of global annual turnover or €15 million. These rules are set to be enforced beginning August 1, 2026. However, challenges persist around practical application, responsibility allocation within the generative AI supply chain, and precise definitions, such as that of a “deep fake.”

Key Obligations Under Article 50

Article 50 of the EU AI Act specifically addresses generative AI systems, outlining these obligations:

Machine-readable markings: Providers of generative AI systems must ensure that outputs are marked in a machine-readable format, detectable as artificially generated or manipulated. Technical solutions must be effective, interoperable, robust, and reliable, within technical feasibility and considering content type and implementation costs. Techniques such as watermarks, metadata identifications, cryptographic methods, logging, and fingerprints are listed as potential means of compliance.
Visible disclosure for deep fakes: Deployers of generative AI systems generating or manipulating image, audio, or video content that constitutes a deep fake must disclose that the content has been artificially generated or manipulated. This information must be provided to individuals in a clear and distinguishable manner upon their initial interaction or exposure. This is typically interpreted as a directly embedded visible watermark or label, and not necessarily a separate disclosure message.

While the Act targets AI systems (apps or web tools), it doesn’t directly mandate transparency measures for the underlying models. However, the second draft of the code of practice for GPAI models does call for systemic-risk models to use watermarks for incident identification and reporting.

Practical Scenarios and Implications

To better understand how these regulations play out in practice, consider these common generative AI deployment scenarios:

End-to-end integrated systems: Organizations developing AI models and integrating them into their applications. These entities act as both provider and deployer, enabling robust watermarking during model development. Examples are OpenAI and Adobe.
Systems using API model access: Systems leveraging APIs from large-scale GPAI model providers. These providers must either rely on existing API features or implement their own post-processing measures for marking and disclosure.
Open-source systems deployed on Hugging Face: Systems built upon open-source models, often deployed via Hugging Face. Determining responsibility for AI Act compliance in these cases is complex, potentially involving Hugging Face itself.
Systems using other (open-source) models under their own trademark: Organizations deploying AI models under their own brand, potentially without disclosing underlying models. These providers are subject to full transparency requirements.

These varying scenarios highlight the complexities in assigning responsibility and ensuring consistent application of the AI Act’s transparency requirements.

Implementation Challenges

The distinctions established by the EU AI Act between providers, deployers, models and systems create grey areas that need clarification.
The burden of compliance with the machine-readable marking and deep fake disclosure requirements may disproportionally affect small-scale app and website developers, as larger, better-funded organizations are typically the organizations developing and deploying the models. Implementing machine-readable solutions would be better carried out at the model development stage. Enforcing regulations at the model developer level would be simpler given the relatively small number of actors developing advanced image models.

What are the practical implications of watermarking implementation and visible disclosure requirements for various types of generative AI systems?

The EU AI Act mandates two key measures for generative AI outputs: machine-readable watermarks for automated detection and visible disclosure of AI-generated deep fakes. Failure to comply could result in heavy fines. However, practical application ambiguities exist around responsibility allocation and deep fake definitions.

Key Implications of the AI Act:

Legal Requirements: By August 1, 2026, all generative AI outputs must have machine-readable markings, and deep fakes must be visibly disclosed as artificially generated.
Liability Risks: Social media companies like Meta could face liability under the EU Digital Services Act for distributing AI-generated content that poses societal risks. This incentivizes larger providers to mark AI-generated content.
Financial Risk: Non-compliance can result in fines of up to 15 million Euros or 3% of global annual turnover.
Model Collapse: Implementing stringent watermarking helps mitigate the risk of model collapse, as AI models could degrade trained on AI-generated content.

The implementation of the AI Act’s transparency rules differs across generative AI system scenarios:

Deployment Scenarios and Implications:

The responsibilities for watermarking and disclosure don’t always stick to neat categories, but here’s a breakdown of common scenarios:

Scenario 1: End-to-end Integrated Systems:

Providers developing AND deploying AI systems (like OpenAI or Adobe) can implement robust watermarking during model development.
Many also provide API access without necessarily extending watermarking features, resulting in inconsistent application across the supply chain.
Social media giants in this category must also consider obligations under the EU’s Digital Services Act and the possibility of GPAI models being classified as high-risk.

Scenario 2: Systems Using API Model Access:

Providers leverage APIs from large-scale GPAI model providers and must comply with transparency.
They can rely on built-in features the model providers give or implement their post-processing measures. Creating a visible marking solution could be difficult if they want to single out deep fake prompts.

Scenario 3: (Open-source) Systems deployed on Hugging Face:

The open-source models deployed via the Hugging Face platform face compliance requirement challenges.
The platform offers an interface but has no control over the model. This makes it difficult to determine who bears compliance responsibility.

Scenario 4: Systems using other (open-source) models under their own trademark:

Organizations use AI models from other organizations to create their own products under their own trademarks without proper notice. Because of this system it makes it difficult to track where the open-source models came from.
This makes their product subject to the same transparency obligations since they now deploy their product without disclosing their source.

These scenarios highlight how blurred lines between developers, providers, and deployers complicate AI Act enforcement. Automated compliance inspection methods are essential as the AI Act takes effect.

What were the findings of the empirical analysis of watermarking and disclosure practices?

Our analysis of 50 widely used generative AI systems reveals a landscape still far from fully compliant with upcoming regulations like the EU AI Act. While the Act mandates machine-readable markings and visible disclosures for certain AI-generated content, adoption is uneven and implementation varies widely.

Key Findings:

Limited Machine-Readable Watermarking: Only a minority of providers (18 out of 50) currently implement machine-readable marking, with metadata embedding being the most common but easily removable technique. Hidden watermarking, a more robust approach, is rarer.
End-to-End Systems Lead in Watermarking: End-to-end integrated systems (Category 1), where the same organization develops the model and the interface, are more likely to implement machine-readable markings.
Visible Disclosures Are Rare: Visible watermarks or disclosures indicating AI-generated content were found in only 8 of the 50 systems analyzed. Notably, these were often applied to all generated images, not just deep fakes.
Challenges with Deep Fake Detection: Restricting visible disclosures to deep fakes presents a significant challenge, particularly for smaller organizations needing to implement NLP-based prompt classification.
Ecosystem Dominated by a Few Players: System providers often rely on a handful of base models from key players like Stability AI, Black Forest Labs, and OpenAI. Watermarking solutions, even when implemented by these core providers, aren’t consistently extended to API-based systems.

Implications:

These findings underscore the need for clearer regulatory guidance and more robust, interoperable watermarking solutions to ensure effective enforcement of the EU AI Act and similar legislation. The current reliance on easily removable metadata and the inconsistent application of watermarking across the generative AI supply chain raise serious concerns about the ability to detect and label AI-generated content effectively.

Regulatory Concerns:

The AI Act’s definitions of “provider” and “deployer” in the context of generative AI remain ambiguous, leading to potential loopholes and uneven distribution of compliance burdens. Smaller app and website developers may bear a disproportionate burden compared to the large, well-funded model developers responsible for the technology’s core.

Actionable Takeaways:

Prioritize Robust Watermarking at the Model Level: Model developers should implement robust watermarking directly in the generation phase to ease implementation for downstream system providers and increase the cost for malicious actors to remove these markings.
Consider GPAI Designation for Advanced Models: The EU and other regulatory bodies should consider designating the most advanced image models as General Purpose AI (GPAI) models with systemic risks, requiring developers to take proactive mitigation measures like watermarking.
Utilize APIs and Licensing Terms for Enforcement: Model developers can leverage APIs and licensing terms to enforce the use of watermarking by downstream system providers.
Develop Automated Compliance Inspection Tools: As the number of generative AI systems grows, automated methods for compliance inspection will be crucial for detecting watermarking and ensuring effective enforcement of AI regulations.

What are the primary challenges and potential future developments concerning the effective implementation of AI Act regulations in the context of generative AI?

The EU AI Act aims to address risks from AI-generated content with mandatory machine-readable markings and visible disclosures for deepfakes. Non-compliance could result in significant fines, kicking in from August 1, 2026. However, ambiguities persist around practical application, especially concerning responsibility allocation and definition of “deepfake.”

Key Implementation Challenges:

Defining ‘Deepfake’: The lack of clarity around what constitutes a “deepfake” creates uncertainty for deployers.
Responsibility Allocation: The division of responsibilities across the complex generative AI supply chain (model developers, downstream developers, system providers, and deployers) needs clarification.

Transparency Rule Complexities:

Specifically, Article 50 of the AI Act outlines transparency rules that apply to generative AI systems:

Article 50(2): Providers must ensure outputs are “marked in a machine-readable format and detectable as artificially generated or manipulated.” Solutions must be effective, interoperable, robust, and reliable.
Article 50(4): Deployers generating or manipulating image, audio, or video content constituting a deepfake must disclose that the content has been artificially generated or manipulated.”
Visibility: Under Article 50(5), information must be provided “in a clear and distinguishable manner at the latest at the time of the first interaction or exposure.” For deep fakes, this is interpreted as including a visible mark on the output.

Future Developments and Considerations:

Efficient enforcement will hinge on:

Automated Compliance Checks: To handle increased volume, automated methods for compliance inspection are needed, which integrate various detection mechanisms.
Model-Level Watermarking: Machine-readable watermarking would be best implemented at the model development stage, where they could be enforced via API and licensing terms.
GPAI Model Designation: The most advanced image models could be designated as GPAI models with systemic risk.
API Provider Classification: It could be considered to classify providers that offer ready-to-use model APIs as providers of AI systems.

Ultimately, the EU AI Act aims for trustworthy implementation and mitigates risks of generative AI content, but it poses substantial ambiguity in translating compliance from legal requirements into actual technical products that may create issues for deployers and the distribution of responsibilities with the different components of the AI generative chain.

As generative AI rapidly reshapes our digital world, the path to responsible innovation lies in bridging the gap between regulatory ambition and practical implementation. While the EU AI Act sets a crucial precedent for transparency and accountability, its effectiveness hinges on addressing critical ambiguities and fostering collaborative solutions across the complex AI ecosystem. Moving forward, focusing on robust, model-level watermarking, clarifying responsibility across the supply chain, and developing automated compliance mechanisms will be essential to unlocking the transformative potential of generative AI while safeguarding against its inherent risks. Successfully navigating these challenges is paramount to fostering a future where AI benefits society as a whole.