AI-Generated Content: Bridging the Gap Between Transparency and Reality

The digital world is grappling with a transformative shift: the rise of artificial intelligence capable of generating remarkably realistic content. This groundbreaking technology, while offering unprecedented creative potential, also introduces significant risks. From the erosion of trust in online information to the potential for malicious manipulation, the societal implications are profound. As jurisdictions worldwide begin to address these challenges, understanding the complexities of AI-generated content and the measures being developed to mitigate its harms is crucial. This analysis delves into the emerging landscape of AI transparency, exploring the technical and practical hurdles that lie ahead in ensuring a trustworthy digital environment.

What are the risks and challenges associated with AI-generated content and how do jurisdictions respond?

The surge in generative AI has blurred the lines between authentic and synthetic content, creating societal risks, especially with realistic deepfakes. This erosion of trust is prompting jurisdictions, like the EU, to introduce AI transparency regulations.

Watermarking, machine-readable markings, and visible disclosures are emerging as key mechanisms to combat these risks. However, conflicting incentives exist, as providers may want to offer users the ability to create content without signs of artificiality. Model collapse, where training on synthetic data degrades AI quality, adds another layer of complexity. Social media platforms also face liability for distributing harmful AI-generated content.

The EU AI Act mandates two main safeguards:

Machine-readable watermarks that facilitate detecting generated or manipulated content.
Visible disclosures of deepfakes to explicitly reveal their artificial origin.

Non-compliance carries substantial penalties, reaching up to €15 million or 3% of global turnover. The rules will apply from August 1, 2026. Despite good intentions, ambiguities persist around responsibility allocation within the AI supply chain, and definitions (like ‘deepfake’) need clarification.

Key Challenges & Risks:

Erosion of Trust: The ease of creating convincing synthetic content undermines trust in media and information sources.
Conflicting Incentives: AI providers balance satisfying customer desires for unrestricted creativity with societal needs for transparency.
Model Collapse: AI training on AI-generated content degrading model quality, is a serious concern for AI developers.
Liability: Social media platforms face increased pressure to manage and flag deep fakes, leading to liability concerns.
Ambiguity in Regulations: The AI Act needs clearer guidance on applying watermarking and disclosure requirements, especially in complex AI supply chains.

What are the core ideas of the generative AI supply chain?

The generative AI landscape, particularly in text-to-image systems, involves a complex supply chain with interconnected steps and diverse participants. A simplified view identifies four primary players:

Base Model Developers: Create the foundational AI models, requiring vast amounts of data and computing resources. Examples include OpenAI (DALL-E 3) and Stability AI (Stable Diffusion).
Downstream Developers: Fine-tune base models for specific applications (e.g., artistic styles). They may distribute these models for a fee or via open-source platforms (e.g., Juggernaut XL).
System Providers: Transform models into functioning systems with user interfaces (web, mobile, desktop). They often offer access to end-users (deployers) but can implement different degrees of sharing and integration.
System Deployers: Deploy AI systems to end-users. In many cases, the system provider also acts as the deployer.

Regulatory Concerns and the EU AI Act

The EU AI Act introduces transparency requirements for generative AI, particularly around watermarking and disclosing AI-generated content (specifically deep fakes). Article 50 mandates:

Machine-readable markings in AI-generated outputs for automated detection of synthetic content. These markings should be effective, interoperable, robust, and reliable. This can be achieved through watermarks, metadata IDs, cryptographic methods, logging, or fingerprinting techniques.
Visible disclosure that content has been artificially generated or manipulated, especially when its a “deep fake”, which the AI Act defines as AI-generated content that resembles existing persons, objects, places, entities or events and would falsely appear to a person to be authentic or truthful. This information must be presented clearly at the point of first interaction.

The regulations apply to AI systems (apps, web tools), not just the underlying models. Non-compliance can lead to penalties of up to €15 million or 3% of global annual turnover. These rules commence on August 1, 2026.

Practical Implications and Implementation Challenges

The AI Act introduces complexities in assigning responsibility across the supply chain. It is not always clear whether the burden of compliance is appropriately distributed, such as for AI models that are API enabled.

Deployment scenarios can vary, resulting in different approaches to watermarking. The document identifies 4 such scenarios:

End-to-End Integrated Systems
Systems Using API Model Access
Open-Source Systems Deployed on Hugging Face
Systems Using Other (Open-Source) Models Under Their Own Trademark

Visible marking of *only* deep fakes is another practical hurdle under the AI Act. To accurately target deep fakes, providers need separate NLP-based solutions for prompt classification. This raises concerns, especially for smaller organizations.

The implementation of effective compliance mechanisms involves challenges in verifying “non-truth” (detecting AI-generated content) and ensuring robust, interoperable solutions. Given the growing number of AI image generation tools, automated compliance inspection methods and third-party solutions will be essential for effective enforcement.

How does the EU AI Act address watermarking and disclosure for AI-generated content?

The EU AI Act, slated to be enforced starting August 1, 2026, introduces key transparency obligations for generative AI systems, particularly concerning watermarking and disclosure.

Legal Requirements

Machine-Readable Watermarking: Article 50(2) mandates that providers of generative AI systems ensure their outputs are marked in a machine-readable format, detectable as artificially generated or manipulated. The Act emphasizes that technical solutions should be effective, interoperable, robust, and reliable.
Visible Disclosure for Deep Fakes: Article 50(4) requires deployers of generative AI systems to disclose when they generate or manipulate image, audio, or video content constituting a “deep fake.” This disclosure must be clear and distinguishable at the time of first interaction or exposure.

The regulations do not specify the exact methods for implementing visible disclosures but emphasizes the need for a clear label within the AI-generated output itself, rather than a separate message. Penalties for non-compliance can reach up to €15 million or 3% of a global annual turnover.

Ambiguities and Implementation Challenges

The AI Act’s language leaves room for interpretation, particularly concerning the allocation of responsibility along the complex generative AI supply chain. Defining “deep fake” also presents challenges.

Limited Scope for Model Developers

The transparency requirements apply specifically to AI systems (e.g., apps or web tools), not to the base models underpinning them.

Although the second draft of the code of practice for GPAI models includes a committment for providers of GPAI models wih systemic risks to use, methods such as watermarks for identifying and reporting incidents related to the use of their model.

Exemptions

Notably, providers of AI systems released under free and open-source licenses are not exempt from these transparency requirements under article 50 of the AI act, unlike some other sections. This is key for the implementation of the Act throughout the ecosystem.

What methodology was utilized to analyze the practical implementation of watermarking?

To gauge the real-world adoption of watermarking and disclosure practices, a multi-pronged approach was employed, analyzing 50 widely used generative AI systems.

Selection of AI Systems

A diverse selection of 50 generative AI systems was curated based on different business models and distribution channels, mirroring the four deployment categories: end-to-end integrated systems, systems using API model access, open-source systems deployed on Hugging Face, and systems using other (open-source) models under their own trademark. The systems in category 1 were selected by filtering the Stanford Foundation Models ecosystem table [26] and selecting organizations that offer free image generation tools using their own foundational models. The systems in category 3 were selected by filtering the Hugging Face ‘model’ section on the five most downloaded open-source text-to-image generation models that offered the Hugging Face API widget tool [21]. The web and mobile app-based systems from categories 2 and 4 were selected using the search query “AI image generation” in the Apple App Store and Google Search. The top 14 systems from each modality (28 in total) that offered free text-to-image generation (if needed: by using an account or by starting a free trial) were included.

Image Generation

For each selected system, at least two images were generated. Standard settings were used with a neutral prompt (“A PhD student”) and a potentially risky “deep fake” prompt (“A beautiful deep fake photograph of Donald Trump in the McDonald’s”). The aim was to evaluate whether watermarks were specifically applied to content flagged as a potential deepfake. Generated images were stored from the “save” or “download” option in the system’s interface.

Watermark and Disclosure Detection

Evaluation metrics focused on machine-readable markings and visible disclosures. A combination of techniques, as outlined below, was utilized to identify their adoption:

Documentation Analysis: System documentation (app descriptions, FAQs, terms of use, privacy policies, and ReadMe files) was reviewed to identify any disclosures related to watermarking, metadata usage, or digital fingerprinting practices.
Code Analysis: For open-source systems (primarily categories 2 and 3), model information pages and source code on Hugging Face and GitHub were inspected for mentions of watermarking libraries, detection tools, or metadata adjustments.
Image Inspection:
Tools were leveraged to examine the generated images:
- Metadata: An online metadata inspection tool was used to extract and analyze metadata from EXIF, IPTC, and XMP standards, searching for AI-generation mentions.
- Watermarking: If watermarking solutions and detection tools were found in documentation or code analysis, the corresponding detection tool was applied to the images. Public detection tools were used for Google’s watermarking [28] and the open-source watermark library used by Stability AI and Black Forest Labs [10, 48, 51]. An algorithm was created to run the said public detection tools on all of the generated images.
- Digital Fingerprinting: The C2PA technical specification, an industry standard for provenance, was analyzed [25, 43]. The C2PA detection tool was used to verify the presence of a digital fingerprint, metadata embedding, or watermark.
Manual Inspections: Manual checks verified visible watermarks and disclosures indicating AI-generated content on both neutral and deep fake prompts. The focus was on whether visible markings were restricted to deep fakes, not on judging whether images qualified as a “deep fake.”

What are the main findings of the empirical analysis regarding watermarking practices?

Our investigation into 50 widely used generative AI image systems reveals a landscape still in its nascent stages concerning watermarking adoption, especially as we approach the EU AI Act’s August 2026 enforcement date. The study primarily aimed not at assessing current compliance, but at outlining the state of machine-readable watermarking and visible disclosure solutions, and the hurdles in implementing and enforcing regulations related to them.

Here are the key findings:

Limited Implementation of Machine-Readable Watermarks: Only a minority of systems (18 out of 50) include some form of machine-readable watermark.
Metadata Embeddings are Common, But Not Robust: While metadata is the most frequently used method, it’s easily stripped, making it a less robust solution. Ten systems used this approach.
Hidden Watermarks are Rare: Sophisticated hidden watermarking techniques are found in only a small subset of systems (6).
Visible Disclosures are Also Rare: Only 8 out of 50 systems used visible watermarks or other disclosure solutions embedded within the image to indicate its AI-generated nature.
End-to-End Systems Lead in Implementation: Machine-readable marking solutions are more prevalent among end-to-end providers (category 1) and large-scale providers (category 2). Many of these are also social/digital media platform or search engine operators (Meta, Google, Adobe, Canva, Microsoft).

These findings point to a “wild west” scenario where robust protections are not uniformly applied.

The Ecosystem is Concentrated

A notable concentration exists in the AI ecosystem. A small number of providers of advanced (open-source) models significantly influence the field. Many system providers rely on base models or fine-tuned versions from just a few sources (e.g., Stability AI, Black Forest Labs, and OpenAI). While they may incorporate solutions, they can be easily disabled or inconsistently applied.

Challenges in Restricting to Deep Fakes

Applying visible disclosures specifically to deep fake images poses challenges. Providers would need a sophisticated system to classify prompts as deep fakes, which can be difficult for smaller organizations. Visible disclosures are mostly not used yet.

Concerns About Long-Term Compliance

The variety of marking techniques and the rise of systems using others’ models (categories 2, 3, and 4) necessitate automated compliance inspection methods. These must integrate various detection mechanisms to ensure effective enforcement when the AI Act takes effect.

What limitations are associated with the investigation?

It’s essential to acknowledge several limitations and constraints in our analysis.

First, there is a possibility that we failed to identify certain watermarking and digital fingerprinting techniques that are used in practice. We tried our best to find any disclosures of watermarking from the documentation of the providers of the systems, but there is a chance that they might have not disclosed them and used non-standard techniques that we did not check for.

Second, although we used prompts that we clearly see as deep fakes, we admit that the definition of a deep fake might be subject to debate.

Finally, some AI systems that we analysed did not offer a save button for the images within the systems’ interface environment, which required us to use other downloading methods. This may have caused certain metadata to be lost, potentially affecting our findings regarding the embedding of machine-readable watermarks.

What are the implications and challenges of watermarking in the context of AI generation systems?

The rise of generative AI has created a pressing need for watermarking and labeling AI-generated content, especially with regulations like the EU AI Act mandating these practices. However, implementation presents a complex web of challenges and implications.

Conflicting Incentives and Regulatory Mandates

While society benefits from identifying AI-generated content, providers often face conflicting incentives. They want to offer users the ability to create content without visible signs of artificial generation. Large companies face the risk of “model collapse,” where AI models degrade when trained on AI-generated content. Also, social media companies like Meta face liability under the EU Digital Services Act for distributing harmful content, further incentivizing watermarking.

The EU AI Act, set to apply from August 1, 2026, mandates two key measures:

Embedding machine-readable markings in AI-generated outputs for automated detection.
Visibly disclosing the artificial origin of AI-generated “deep fakes.”

Non-compliance can result in fines up to €15 million or 3% of a company’s global annual turnover. However, ambiguities persist regarding the practical application of these requirements, including responsibility allocation and the definition of “deep fake.”

Transparency Challenges Across the AI Supply Chain

Generative AI’s supply chain involves base model developers, downstream developers, system providers, and system deployers. The AI Act’s transparency rules apply specifically to AI systems (apps or web tools), not necessarily to the underlying model developers.

Different deployment scenarios create varying implications for compliance:

End-to-end Integrated Systems: Organizations developing and deploying AI models are responsible for implementing robust watermarking.
Systems Using API Model Access: These systems leverage APIs from large-scale model providers. Compliance depends on utilizing built-in watermarking features or implementing post-processing measures.
(Open-Source) Systems Deployed on Hugging Face: Determining responsibility for AI Act compliance is not clear in these cases, especially as Hugging Face provides the user interface.
Systems Using Other (Open-Source) Models Under Their Own Trademark: These organizations deploy AI models under their own brand without disclosing the source, requiring full compliance with transparency obligations.

Practical Considerations and Implementation Gaps

Currently, only a minority of providers implement machine-readable marking practices, largely driven by large organizations who want to prevent the degradation of their AI training sets and protect copyrighted content.

Limited Watermarking: Robust watermarking methods focusing on detecting AI-generated images remain rare, especially those which cannot be easily removed. Many solutions rely on post-generation techniques like metadata embedding, which are easily stripped.
Visible Disclosures: Visible watermarks for deep fakes are seldom used, often due to their impact on user experience.
Deep Fake Detection: Restricting labeling to deep fakes requires complex methods, potentially challenging for smaller organizations.
Ecosystem Concentration: A handful of model providers heavily influence the ecosystem, making their actions critical for wider adoption of watermarking practices.

Challenges exist in fairly distributing compliance burdens along the AI supply chain, and the EU is considering classifying large-scale models as GPAI models with systemic risk, thus requiring developers to have strict watermarking implementation via API.

Practical Implications Under the New EU AI Act

The EU AI Act mandates specific measures to address risks from AI-generated content. Here’s a breakdown for compliance officers and legal-tech professionals:

Key Requirements:

Machine-Readable Markings (Article 50(2)): All AI-generated outputs must have embedded, detectable machine-readable markings. This aims to facilitate automated detection of synthetic content.
Visible Disclosures (Article 50(4)): Deployers of generative AI systems creating or manipulating “deep fakes” (resembling real people, objects, etc.) must disclose that the content is artificial. This disclosure needs to be clear and distinguishable at the time of first “interaction or exposure.”

Enforcement and Penalties:

These rules take effect from August 1, 2026.
Non-compliance can result in fines up to €15 million or 3% of global annual turnover.

Ambiguities and Challenges

The AI Act faces practical application issues like responsibility allocation throughout the complex generative AI supply chain, and the definition of a “deep fake.” This section examines those practical deployment scenarios of generative AI systems to clarify application, and they would serve for designing AI measurements to clarify the translation of the legal into technical artifacts.

Deployment Scenarios and Responsibilities

To clarify the AI Act’s transparency rules’ application, the paper identifies four deployment scenarios, analyzing how the rules apply in each:

End-to-End Integrated Systems: Organizations developing and deploying AI models internally need compliance with both machine-readable markings and visible deep fake disclosures.
Systems Using API Model Access: Systems integrating APIs from large model providers (like OpenAI) require compliance. They can either rely on built-in features from model providers or implement their own measures (post-processing watermarks, metadata). Detection of deep fake prompts presents a significant challenge, especially for smaller companies. A simpler approach could involve visible disclosures to all the generated images, although this may negatively impact the user experience.
(Open-Source) Systems Deployed on Hugging Face: It’s unclear who bears responsibility for compliance where AI models are hosted on Hugging Face. Hugging Face offers the user interface but has no control over the model, it could be the provider and deployer.
Systems Using Other (Open-Source) Models under Their Own Trademark: This category includes organisations requiring to comply with both transparency obligations.

Practical Takeaways

Clarify Roles: Businesses must clearly define their role in the AI supply chain (model developer, system provider, deployer) to understand their specific obligations.
Technical Artifact Translation: Businesses must translate the legal requirements technical artifacts by the final ruling and/or implementation of the AI Act.
Evaluate Existing Solutions: Assess existing watermarking capabilities in AI models or APIs used. If absent, implement post-processing techniques.
Deep Fake Detection: Develop or acquire capabilities to detect deep fake prompts or consider applying visible disclosures broadly.

As the regulatory landscape for AI sharpens, the current state of watermarking in image generation reveals a fragmented reality. While awareness of the need for transparency is growing, tangible implementation, particularly of robust and consistently applied solutions, remains limited. The path forward requires collaborative efforts across the AI supply chain, alongside the development and adoption of automated verification tools. Successfully navigating the evolving regulatory environment necessitates a shift from ad-hoc measures to standardized, verifiable, and scalable methods for ensuring the responsible deployment of AI-generated content. The ultimate success rests on bridging the gap between regulatory intent and practical execution within a rapidly advancing technological domain.