X moves to restrict Grok after outcry over sexualised deepfakes

Regulators mull next steps

X has moved to restrict AI chatbot Grok's image editing capabilities to paying subscribers only, after the social media site was flooded with non-consensual sexualised deepfakes of women and children.

The move comes after authorities and regulators in several countries promised to crack down on Elon Musk’s social media site.

Grok, developed by Musk-owned xAI, recently introduced a feature that allowed any user to edit pictures posted on X, formerly Twitter, without requiring consent. The feature was quickly used to “nudify” pictures of women and children, bypassing the app’s flimsy guardrails by specifying the subject should appear in a “transparent bikini” or similar.

This feature has now been restricted to paying subscribers.

So far, no-one at X has issued an apology or acknowledged the harm caused. While the new restrictions are welcome, the tool should never have been given this capability in the first place, and limiting access does not undo the harm already caused, Hannah Swirsky, head of policy at the Internet Watch Foundation, told the BBC. "We do not believe it is good enough to simply limit access to a tool which should never have had the capacity to create the kind of imagery we have seen in recent days," she said.

Simply limiting the feature to paying subscribers would likely be insufficient to satisfy most regulators if the images were then shared.

The UK government has urged regulator Ofcom to use its powers under the Online Safety act, to address the unlawful AI-generated images published on X. Keir Starmer called the situation "disgraceful" and "disgusting," and said there should be zero tolerance for such content.

In the UK, sharing of non-consensual intimate images is illegal. The creation of such images is also illegal under the Data (Use and Access) Act (DUA) - but this law is not yet in force despite having been passed 6 months ago.

It’s also an offence under the Protection of Children Act (1978) and the Sexual Offences Act (2003) to distribute and publish an indecent photograph or pseudo-photograph of an under-18-year-old.

However, it may also prove difficult to prosecute an organisation based overseas.

Under the Online Safety Act, regulator Ofcom could issue substantial fines (up to £18 million or 10% of global turnover) against X, while other countries have threatened to ban the app from app stores or prevent third parties from supporting it financially.

In the EU, the Digital Services Act (DSA) requires sites to mitigate risks associated with harmful content, including non-consensual sexual imagery. Non-compliance could result in fines of up to 6% of global turnover.

The way the world’s regulators act on this issue will have important repercussions on the future of AI and social media.