Grok is an AI-tool developed by the provider of X. In recent years, Grok has been deployed in several ways, such as allowing users to generate images and texts as well as providing them with contextual info to other user’s posts.
This deployment has recently prompted the European Commission to launch an investigation against X under the Digital Services Act (DSA), as Grok was utilised to disseminate illegal content in the EU, such as deep fakes with sexual content.
The Grok incident has demonstrated the current regulatory framework's vulnerability in addressing deep fake incidents, particularly those involving sexual content.
This blog post provides an overview of the current regulatory landscape, specifically the AI Act and Digital Services Act, highlighting the existing risks. Finally, the blog post concludes with a summary of the next steps to be taken by the EU institutions in light of the recent incident.
Overview of the legal frameworks
AI Act
Under the AI Act, for content to be considered as a deep fake, four criteria must be met.
Content must be AI-generated or manipulated;
The content must be in image, audio or video form;
It must resemble existing persons, objects, places, entities or events. Monsters or unicorns can thus not be considered as deep fakes;
The must falsely appear to a person to be authentic or truthful.
It is important however to examine the relationship between deep fakes and the existing risk-based approach of the AI Act which divides AI systems into different risk categories. These include the categories of (1) unacceptable risks, (2) high risks, (3) limited risks and (4) minimal to no risks. The first three risk categories are regulated by the AI Act. Nevertheless, in view of this risk-based approach, the qualification of deep fakes with sexual content under the AI Act remains unresolved.
Prohibited AI systems
Article 5 of the AI Act provides a list of AI systems that constitute an unacceptable level of risk and are therefore prohibited. In the context of deep fake creation, there is one particular AI system that could potentially be considered as prohibited, namely the one mentioned in Article 5 (1) (a).
Article 5 (1) (a) of the AI Act states the following:
The following AI practices shall be prohibited: (a) the placing on the market, the putting into service or the use of an AI system that deploys subliminal techniquesbeyond a person’s consciousnessor purposefully manipulative or deceptive techniques, with the objective, or the effect of materially distorting the behaviour of a person or a group of persons by appreciably impairing their ability to make an informed decision, thereby causing them to take a decision that they would not have otherwise taken in a manner that causes or is reasonably likely to cause that person, another person or group of persons significant harm.
However, according to the prevailing opinion in academic lecture, deep fakes do not generally appear to fall under aforementioned prohibition. This is due to the fact that, in most instances where deep fakes are being created, the AI system merely serves an aid to the use of manipulative or deceptive techniques by an individual or entity. According to Martina J. Block however, if an AI system were to specifically analyse individuals and create targeted deep fakes based on that analysis, and deliver them at the ‘right’ moment, Article 5 (1) (a) could be applicable due to the analysis and specific adaptation of the content.
Nevertheless, under the current framework, deep fakes generally cannot be classified as an Article 5 (1) (a) prohibited practice, nor as one of the other prohibited practices under the AI Act.
High-risk AI systems
For an AI system to be qualified as a high-risk system it should fulfil one of the following conditions:
The AI system is intended to be used as a safety component of a product, or the AI system is itself a product, covered by the Union harmonisation legislation listed in Annex I and is required to undergo a third-party conformity assessment. E.g.: lifts and safety components of lifts.
The AI system is being referred to in one of the specific ‘use cases’ in Annex III. E.g.: AI systems intended to be used for influencing the outcome of an election or referendum or the voting behaviour of natural persons in the exercise of their vote in elections or referenda.*
* Side note: although deep fakes are generally suited for influencing elections, as demonstrated in the case of the far-right AfD-party sharing a deep fake video about the German future with and without the party, very few AI systems will be specificallyintended for the purpose of influencing elections. In the majority of cases, AI systems will be merely misused for this purpose.
In the specific case of AI systems used to create deep fakes containing sexual content, such as in the Grok incident, it is evident that these systems are not generally part of a product or themselves are products that are covered by the legislation covered in Annex I. The creation of deep fakes by AI systems does not appear to be encompassed within the Annex III scope either, unless the AI system is specifically intended for the purpose of influencing elections. In contexts involving the utilisation of such systems for the creation of deep fakes that contain sexual content, the exclusion is even more evident.
Transparency requirements
In the case of AI systems containing limited risks, the transparency requirements under Article 50 of the AI Act may be triggered.
Providers
According to Article 50(2) AI Act providers of AI systems, including general-purpose AI systems, generating synthetic audio, image, video or text content, must ensure that the outputs of the AI system are marked in a machine-readable format and detectable as artificially generated or manipulated. Despite the fact that the article does not explicitly refer to deep fakes, it is evident that AI systems that create deep fakes fall within the scope of the article as they generate synthetic audio, image and video content.
This obligation does not apply in the following cases: (1) where AI systems perform an assistive function for standard editing or do not substantially modify the input data provided by the deployer or the semantics thereof, or (2) where authorised by law to detect, prevent, investigate or prosecute criminal offences.
Deployers
Based on Article 50(4) AI Act deployers of an AI system that generates or manipulates image, audio or video content constituting a deep fake, must disclose that the content has been artificially generated or manipulated.
There are two exceptions to this:
Where the use is authorised by law to detect, prevent, investigate or prosecute criminal offence;
Where the content forms part of an evidently artistic, creative, satirical, fictional or analogous work or programme. In this case the transparency obligations set out are limited to disclosure of the existence of such generated or manipulated content in an appropriate manner that does not hamper the display or enjoyment of the work.
Digital Services Act
The DSA’s main goal is to strengthen the protection of citizen’s fundamental rights online by giving them more control and choices while navigating online platforms. On the 25th of April 2023 the European Commission released a list of Very Large Online Platforms (VLOPs), which included Twitter (now known as X).
These VLOPs have specific obligations under the DSA given their importance due to their reach, such as the number of recipients of the service, the facilitation of public debate, economic transactions, the influence of how recipients obtain and communicate information online and more.
Specifically, providers like X must identify, analyse and assess any systemic risks in the Union stemming from the design or functioning of their service or from the use made of their services.
There are four categories of systemic risks that should be assessed by the providers of VLOPs. These categories are the following:
Category 1: the risks associated with the dissemination of illegal content, such as the dissemination of child sexual abuse material or illegal hate speech and the conduct of illegal activities, such as the sale of products or services prohibited by Union or national law. The DSA determines that such dissemination may constitute a significant systemic risk where access to illegal content may spread rapidly and widely through accounts with a particularly wide reach or other means of amplification.
Category 2: this category concerns the actual or foreseeable impact of the service on the exercise of fundamental rights, as protected by the Charter of Fundamental rights of the European Union, including but not limited to human dignity, freedom of expression and of information, including media freedom and pluralism, the right to private life, data protection, the right to non-discrimination, the rights of the child and consumer protection.
Category 3: this category concerns the actual or foreseeable negative effects on democratic processes, civic discourse and electoral processes, as well as public security.
Category 4: this risk category stems from concerns relating to the design, functioning or use, including through manipulation, of very large online platforms and of very large online search engines with an actual or foreseeable negative effect on the protection of public health, minors and serious negative consequences to a person's physical and mental well-being, or on gender-based violence.
These risk assessment should be carried out at least once every year and should take into consideration the severity and probability of the risks mentioned above.
Based on the specific systemic risks, providers must put in place reasonable, proportionate and effective mitigation measures. One of these measures may include the labelling of deep fakes.
Specifically the DSA determines that providers should ensure that an item of information, whether it constitutes a generated or manipulated image, audio or video that appreciably resembles existing persons, objects, places or other entities or events and falsely appears to a person to be authentic or truthful is distinguishable through prominent markings when presented on their online interfaces, and, in addition, providing an easy to use functionality which enables recipients of the service to indicate such information. This is very similar to the definition of deep fakes in the AI Act, although the DSA does not explicitly refer to them as “deep fakes”.
Finally, providers of very large online platforms, specifically those primarily used for the dissemination of pornographic content, need to fulfil all their obligations under the DSA diligently in respect of illegal content constituting cyber violence, including illegal pornographic content. This includes ensuring that notices of this content are processed rapidly as well and that the content in question is swiftly removed.
Current risks of the legal frameworks
AI Act
As outlined above, it is evident that the current framework fails to specifically address the creation of deep fakes with sexual content. At present, the provisions for both the prohibited and high-risk AI systems do not offer any definitive solutions.
Subsequently, there are also some concerns about the feasibility of implementing Article 50 (4). Although the article stipulates an obligation for deployers to disclose deep fakes, the question of whether these deployers (i.e. the users of Grok) are able to invoke the household exemption remains unresolved.
The AI Act defines a deployer as ‘a natural or legal person, public authority, agency or other body using an AI system under its authority except where the AI system is used in the course of a personal non-professional activity’.
As mentioned by Mateusz Łabuz, the AI Act generally doesn’t apply to end users, meaning it will not cover ‘nonprofessional’ activities aimed at creating and disseminating deep fakes. This exclusion causes significant issues, making it a subject of discussion by various legal professionals. Martina J. Block, for example, argues that the creation of a deep fake is essentially for the purpose of publication, although the specific platform on which it is published may vary (a small WhatsApp group or a bigger social media platform). Block asserts that, in exceptional cases in which the deep fake is not created for publication, the household exemption applies. In any instances where deep fakes are created and published, the inherent characteristic of being intended for publication remains, regardless of the size of the group that will receive it. In such cases, the household exemption does not appear to be applicable. Consequently, in the majority of cases, the individual who created the deep fake will be also typically be the person deploying the AI system in question. Therefore, according to Block, they must also be in the position of disclosing the deep fake.
However, the practical implications of this remains to be clarified. Furthermore, it is yet to be determined whether the publication of deep fakes, as Block has suggested, would constitute as a sufficient ground to exclude end users from the household exemption, thereby categorising them as deployers.
Lastly, although the European Commission has the competence under Article 112 (1) to assess the need for amendment of the list set out in Annex III, as well as the list of prohibited AI practices laid down in Article 5, once a year, the Grok incident seems to have proven that the article is inadequate in addressing pressing threats posed by AI-generated deep fakes.
Digital Services Act
As demonstrated by the Grok incident, the effective implementation of the DSA is dependent on whether or not the VLOP (in this case X) is willing to carry out an actual risk assessment or not.
As stated by Elisabetta Stringhi and often observed in the context of GDPR compliance, the imposition of fines is often more convenient for online platforms than a radical altercation of their business models. Much like the GDPR, an efficient coordination between supervisory authorities is crucial for the effective enforcement of the DSA. This coordination is even more imperative in the context of the creation and publication of deep fakes with sexual content.
Breaches of the DSA can result in fines of up to 6 percent of the global annual turnover and, in cases of continuous non-compliance, even lead to the platform’s temporary suspension in the EU. However, the regulatory framework is not designed to impose a permanent ban on services from the market. Even as a last resort measure, the Commission can only request a temporary suspension of the service and this is only if (1) the infringement persists and causes serious harm to users and (2) entails criminal offences involving threat to persons’ life of safety. This in contrast with the new legislation proposed by the UK, triggered by the Grok incident, whereby companies would have 48 hours to remove illegal content, such as non-consensual sexual images, once it is reported to them. Failure in complying, could lead to 10% of qualifying worldwide revenue, or even having their services blocked in the UK.
Next steps of the EU institutions
Investigation
According to the Commission their investigation will evaluate whether X ‘properly assessed and mitigated risks associated with the deployment of Grok’s functionalities into X in the EU’. One of these risks relate to the circulation of illegal content in EU, like manipulated sexually explicit images, including content that may amount to child sexual abuse material.
Currently the Commission will continue to gather evidence on the incidents. If necessary, it also may impose interim measures, like for example specific keywords or hashtags or orders to terminate alleged infringements.
Transparency requirements
Against this backdrop of rapid changes, the European Commission is also working on the Code of Practice on Transparency of AI-Generated Content. This Code of Practice (CoP) which is a soft law instrument, aims to support compliance with the transparency obligations of the AI Act related to the marking and labelling of AI-generated content. The First Draft of the CoP, published on 17 December 2025, consists, among other things, of measures on the ‘Clear and distinguishable disclosure of deep fakes’. Examples are:
For real-time deep fake video, displaying the icon in a non-intrusive way consistently throughout the exposure where feasible, as well as inserting a disclaimer at the beginning of exposure that explain that this display content includes deep fake.
For non-real-time deep fake video, Signatories of the CoP will disclose that the video contain deep fakes with the icon. the Signatories may choose among various disclosure options, individually or combined, as deemed appropriate according to the specific context.
A disclaimer which is at the beginning of the exposure. In case of an oral disclaimer, the icon needs to appear simultaneously with the audio disclosure.
Placing the icon continuously throughout the exposure in an appropriate fixed place, ensuring that the disclosure is clearly visible to the viewer without any extra interaction. For online platforms, this means placing the icon consistently just outside the video frame integrated into the user interface or interface overlay.
A disclaimer which is in the credits at the end of the video. This measure always needs to be accompanied by other measures as mentioned in the Code.
It is important to note that this first published version of the Code of Practice is still a draft. The final version is expected to be finalised in May–June 2026.
While the Code may help to establish shared standards and outline practical self-regulatory measures prior to the introduction of binding regulations, it remains a form of soft law and is therefore not legally binding. Therefore, the Code does not clarify the difficult legal classification of deep fakes containing sexual content; rather, it provides exemplary measures that providers and deployers can take to increase transparency about them.
Although enforcement strategies such as voluntary codes of practice are welcomed, in the past they have been also the subject of critique by the European Parliament as they ‘prevent a democratic control in the adoption of transparent rules, equally applied all over the Union and which might be challenged in courts by citizens and economic operators’.
Closing remark: will the Digital Omnibus Proposal bring solace?
On 19 November 2025, the European Commission proposed measures in order to simplify the implementation of various AI Act provisions (the Digital Omnibus on AI Regulation Proposal).
In their recent opinion on this Proposal, the JURI Committee (the European Parliament’s Committee on Legal Affairs), proposed adding two new prohibitions to the Article 5 list of prohibited AI systems.
Specifically the following prohibitions are proposed: ‘the placing on the market, the putting into service or the use of an AI system that can generate, alter or reproduce sexually or nude content in the violation of the dignity, sexual integrity or consent of natural persons, through the use of deep fake or other synthetic media techniques’, as well as ‘the placing on the market, the putting into service or the use of an AI system that generates child sexual abuse material, regardless of the nature or origin of the underlying content’.
It thus remains to be seen whether the upcoming months will result in a resolution to this ongoing issue. The Grok incident, however, seems to have triggered the topic's resurgence, with privacy watchdogs all over the world asserting that deep fakes consititute a significant global risk to children and other vulnerable groups.
