Amnesty calls for ban on AI models that use illegal data scraping

Many GenAI systems are 'illegal by design’ and break international human rights law, charity claims

Amnesty International has called for GenAI systems that violate international law to be prohibited.

In a briefing report the human rights charity claims that "generative AI systems, based on unlawful web scraping, are in conflict with international human rights law (IHRL) and standards through their design, development and deployment."

Referencing conventions including the International Covenant on Civil and Political Rights (ICCPR) and the International Convention on the Elimination of All Forms of Racial Discrimination (ICERD), Amnesty calls out “standalone generative AI systems” (such as AI chatbots and image/video/audio/text generators) on several fronts, describing them as "unlawful by design".

Among the models analysed were ChatGPT, Gemini, DALL-E, Llama and DeepSeek. Amnesty stops short of naming individual models as "illegal", but argues that some flagship systems rely on practices that make them unlawful “by design”.

Invasion of privacy

AI training data frequently includes personal data collected without consent. Amnesty says this constitutes “mass invasions of privacy" in contravention of the ICCPR, adding that they fail legal tests of necessity, proportionality and consent.

"Notably, the bulk and mass collection of training data through web scraping constitutes a mass invasion of privacy by design because such collection includes personal data and is happening without knowledge and consent," the report says.

Bias, discrimination and inequality

The report takes aim at the widely acknowledged biases in the output of LLMs and other models due to the data they have been trained on. These biases tend to reinforce existing stereotypes and normalise discriminatory practices. A further issue is caused by the disproportionate use of the English language. The use of AI to generate porn and CSAM, disproportionately of women and girls, violates the Rights of the Child (CRC), the right to equality and non-discrimination as well the right to privacy, Amnesty argues.

Threats to freedom of expression

Generative AI-based moderation systems risk over-censoring content, especially in non-English contexts as well as misinterpreting cultural or political speech, Amnesty says. In addition, new norms are imposed owing to biases in the training data, creating a “shrinking civic space”. The report also notes that some models are created with inbuilt censorship, including Chinese LLMs that refuse to comment on the Tiananmen Square massacre.

Threats to freedom of thought

There is a danger that through using AI systems people are becoming less able to think critically, instead trusting AI outputs implicitly, a phenomenon known as "automation bias" or "techno-chauvinism".

"The manipulation of user intentions and thought processes through predictive suggestions may constitute coercion ... while repeated exposure to synthetic content and algorithmic biases can shape beliefs and mental models without user awareness as a form of manipulation." the briefing says, warning this could undermine independent thinking and autonomy.

Environmental and labour impacts

The physical requirements of training and inference including high energy and water usage and the resultant shortages and pollution are having a disproportionate impact on communities where the datacentres are located, Amnesty says: "Around the world, datacentres are already being built at the expense of historically marginalised communities facing the brunt of the catastrophic consequences of the infrastructure."

The briefing notes a lack of accountability and transparency about the impacts of this infrastructure on behalf of the AI companies, including the toll on low-paid workers engaged in data labelling.

It urges governments to mandate transparency and human rights impact assessments and regulate AI supply chains and environmental impact, and calls on companies to stop non-consensual data scraping and address bias and discrimination directly.

"Without significant changes to how these systems are developed and deployed, they will continue to undermine international human rights laws and standards." it concludes.