X Tightens Grok AI Controls To Prevent Obscene Deepfakes In India And Worldwide

Microblogging platform X has introduced new restrictions on its AI chatbot Grok to prevent the generation of images of real people in revealing clothing in jurisdictions where it is illegal, after facing widespread criticism over obscene deepfakes created using the AI.

According to PTI, the platform announced that this restriction applies to all users, including paid subscribers, through a post on its official 'Safety' handle. Image creation and editing via the Grok account on X are now restricted to paid subscribers only, which the platform said adds an extra layer of accountability against misuse.

"We now geoblock the ability of all users to generate images of real people in bikinis, underwear, and similar attire via the Grok account and in Grok in X in those jurisdictions where it's illegal," the post said.

X further stated that technological measures have been implemented to prevent the editing of images of real people in revealing clothing. "This restriction applies to all users, including paid subscribers," the post added.

The platform reiterated that all AI-generated content and prompts on X must continue to follow its rules. "However content is created or whether users are free or paid subscribers, our Safety team are working around the clock to add additional safeguards, take swift and decisive action to remove violating and illegal content, permanently suspend accounts where appropriate, and collaborate with local governments and law enforcement as necessary," X said.

According to PTI, X affirmed its commitment to maintaining a safe platform, with zero tolerance for child sexual exploitation, non-consensual nudity, and unwanted sexual content. "We take action to remove high-priority violative content, including Child Sexual Abuse Material (CSAM) and non-consensual nudity, taking appropriate action against accounts that violate our X rules," it said, adding that accounts attempting to access child sexual exploitation materials are reported to law enforcement.

The move comes after India’s IT Ministry raised concerns over obscene content linked to Grok, prompting X to remove around 3,500 pieces of content and delete over 600 accounts. X had assured authorities that it would comply with Indian laws, PTI reported.

Governments worldwide, including India, have stepped up scrutiny of Grok over non-consensual sexually explicit images, content moderation, and data safety issues. On January 2, the IT Ministry directed X to remove all vulgar and unlawful content generated by Grok immediately or face legal action.

The ministry also asked the platform to submit a detailed action taken report (ATR), outlining technical and organisational measures for Grok, the oversight role of the Chief Compliance Officer, actions against offending content, and mechanisms to ensure compliance with mandatory reporting under Indian law.

The ministry highlighted that Grok AI was being misused to generate obscene images or videos of women, both via fake accounts and by manipulating content hosted or published by real users. It warned that such misuse reflects a serious failure of platform safeguards and enforcement mechanisms and constitutes a gross violation of applicable laws.

Compliance with the IT Act and related rules, the ministry emphasised, is mandatory. Exemptions under section 79 of the IT Act, which provide safe harbour protection for online intermediaries, are conditional on strict adherence to due diligence obligations. Failure to comply could lead to the platform losing immunity and facing action under other provisions of the IT Act and Bharatiya Nyaya Sanhita.

The ministry also directed X to enforce user terms and AI restrictions, including suspension and termination of violating accounts, and to remove or disable access to unlawful content promptly, in accordance with the IT Rules, 2021, without affecting evidence.

X’s initial response, though detailed, was viewed by the ministry as largely reiterating compliance commitments and takedown policies, without providing sufficient specifics on action taken regarding obscene Grok content. The platform later acknowledged the lapse and confirmed it would adhere to Indian laws, PTI reported.

Beyond India, regulators in the UK and EU have also raised concerns over the misuse of Grok AI for generating deepfake images.

(With inputs from PTI)