Meta pulls Galactica language model three days after launch

Model reached biased and harmful conclusions despite 'good' training data

Meta has taken its new large language model, Galactica AI, offline after experts found it to be biased and generating false information.

Meta AI launched a demo of Galactica last week, with the intent of making it easier for scientists and other researchers to collect, combine and reason about scientific information.

The company said Galactica 'can summarise academic papers, solve math[s] problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more.'

Meta trained the new large language model (LLM) on 48 million papers, text books, lecture notes, encyclopaedia and scientific websites.

Galactica's research paper said the team's evaluation of 'toxicity and bias' in Galactica found that the model fared better than several other LLMs.

Researchers at Meta AI were likewise under the impression that the high-quality input would result in high-quality output.

Experts' views, however, indicate that the new LLM's training was insufficient.

While some individuals found the demo both useful and promising, others quickly realised that anybody could enter in racist, inappropriate or potentially offensive prompts and easily produce authoritative-sounding content on those topics.

LLMs, like OpenAI's GPT-3, learn to write text by examining millions of samples and figuring out the statistical connections between words. The the models may produce convincing-sounding texts as a result, but the outputs are often full of untruths and potentially damaging prejudices.

Many researchers who used Galactica shared its flawed and biased results on social media.

Michael Black, Director of the Max Planck Institute for Intelligent Systems, accused Galacita of spreading "dangerous" false information.

He added that while Galactica sounded authoritative, it was either prejudiced or incorrect in all cases.

Carl Bergstrom, a biology professor at the University of Washington, also said Galactica generated incorrect results.

A fundamental issue with Galactica, and many other LLMs, is that it is unable to distinguish between truth from falsehood, which is a key prerequisite for a language model intended to produce scientific content.

Additionally, there are problematic gaps in what Galactica is capable of handling.

When asked to produce text on certain topics, including 'racism' and 'AIDS,' the model gave the following response: 'Sorry, your query didn't pass our content filters. Try again and keep in mind this is a scientific language model.'

Because Galactica was producing false information, Meta turned it off on Thursday last week.

When MIT Technology Review asked Meta for a comment on the decision, the company pointed to a tweet that says, 'Thank you everyone for trying the Galactica model demo. We appreciate the feedback we have received so far from the community, and have paused the demo for now.'

'Our models are available for researchers who want to learn more about the work and reproduce results in the paper.'

Computing says:

Training AI on a 'bad' dataset can lead to a system full of flaws and inaccuracies, like Amazon's (never-released) recruitment tool that scored women lower than men, or facial recognition programmes that misidentify based on race.

Previous LLMs, like PaLM, Davinci and Meta's own OPT-175B have all been guilty of this, and it's slightly surprising that Galactica reached this state even with supposedly 'good' training data. It highlights that there is still clearly much work to do in the AI space before it can be trusted with anything other than hard mathematics.