Child sexual abuse material discovered in popular AI image dataset

Rush to publish means safeguards are being overlooked

A study by researchers at the Stanford Internet Observatory has found that LAION-5B, one of the largest image datasets used to train AI systems like Stable Diffusion, contains thousands of instances of child sexual abuse material (CSAM).

The suspect CSAM images were identified through a combination of perceptual and cryptographic hash detection.

The non-profit LAION (Large-scale Artificial Intelligence Open Network), which compiled the model, identifies links to suspect images through a "not safe for work" classification.

It is illegal to view CSAM, so the Stanford researchers sent suspected matches via Microsoft's PhotoDNA image checker to project Arachnid Shield API and the Canadian Centre for Child Protection (C3P), which is permitted to analyse them.

More than 2,000 of the images were identified as CSAM with a high probability, with almost 1,000 being manually verified by the C3P. This may not sound like much in a dataset of more than 5 billion images, but the researchers point out, it allows models to generate realistic child pornography and can have a disastrous effect on victims already traumatised from having the images available online.

Moreover, the researchers say, their results are certainly a significant undercount due to practical limitations and the inaccuracy of LAION's classifiers.

The excitement and commercial buzz around generative AI has meant that models, which generally scrape training data from the web, are being released to the public domain before effective safeguards are in place. LAION could have used similar processes to the researchers in order to effectively root out CSAM, but it did not, the study's lead author David Theil said.

"Ideally, such datasets should be restricted to research settings only, with more curated and well‐sourced datasets used for publicly distributed models," said Theil.

Unfortunately, it is not an easy problem to fix retrospectively, because the material is now part of publicly available models, including some versions of Stable Diffusion.

"We are now faced with the task of trying to mitigate the material present in extant training sets and tainted models; a problem gaining urgency with the surge in the use of these models to produce not just generated CSAM but also CSAM and NCII of real children often for commercial purposes," said Theil.

LAION has now taken down its LAION-5B dataset, with pages on Hugging Face where it was hosted, saying "access disabled on dataset author's request."

Theil argues that LAION‐5B‐derived training sets should be deleted or referred to experts to have the offensive and illegal material removed.

"Models based on Stable Diffusion 1.5 that have not had safety measures applied to them should be deprecated and distribution ceased where feasible," he wrote.

Stability AI said in a statement that it only hosts filtered versions of Stable Diffusion adding "since taking over the exclusive development of Stable Diffusion, Stability AI has taken proactive steps to mitigate the risk of misuse".

Google, whose Imagen model is based on a LAION dataset, said it decided not to make it public after it "uncovered a wide range of inappropriate content including pornographic imagery, racist slurs, and harmful social stereotypes".