Google's Gemini AI hacked three companies during security test

AI model used exposed credentials to access real companies

Google's Gemini AI model accessed real-world computer systems and used exposed credentials to enter three companies during a cybersecurity test.

The incidents took place in May during an evaluation conducted by Israel-based AI security firm Irregular.

Google said there was no evidence that the affected companies suffered any damage and that Gemini stopped after gaining access.

The incident was first reported by The Wall Street Journal.

According to Google, Gemini was participating in a "capture the flag" exercise designed to assess its ability to locate information within systems operated by fictional companies. The testing environment was intended to be isolated from the wider internet. However, an unintended connection gave Gemini access to real-world information and computer systems.

In one case, the model attempted to access a fictional company with the same name as a real business, with AI model eventually reaching the real company's software.

In two other cases, the model discovered credentials that were publicly available online and used them to access real corporate systems.

"In all three of these instances, the model stopped," Google vice-president of security engineering Heather Adkins said, adding that the affected organisations were informed.

Google changes security testing process

Google said it has worked with Irregular to change its testing procedures and strengthen safeguards around evaluation environments.

A spokesperson for Irregular said the Gemini incidents were related to an issue that had already been reported and did not constitute a separate, materially different incident.

The company said relevant AI laboratories had been informed in late July and that affected organisations had been contacted during the investigation. Irregular said the problems on its side had been addressed and resolved within weeks.

Moving beyond the boundaries

The Gemini incidents are part of a series of cases in which AI models have moved beyond the boundaries intended by developers during security evaluations - although how firmly those boundaries are set is very much a matter of opinion.)

Irregular has also been involved in investigations concerning models from OpenAI and Anthropic.

OpenAI has reported incidents in which its AI systems interacted with external services, including one where a model uploaded malicious packages to RubyGems and another involving Hugging Face.

Anthropic has similarly reported cases involving its Claude models.

Meta said in August that one of its AI models gained internet access and breached another organisation's computer systems during an evaluation. The company said the incident did not involve a sandbox escape or a sophisticated cyberattack.

These incidents have raised questions about how AI systems should be tested when they are capable of independently finding information, obtaining credentials and taking actions using external tools.

Google said the Gemini incident demonstrated why safeguards and responsible behaviour need to be built into evaluations of powerful AI systems.

Potential risks from LLMs

The concerns come as technology companies and governments debate the potential risks of agentic LLMs.

Former Anthropic researcher Jacob Coxon recently warned that rapid AI development could carry serious risks. Last week, King Charles called for safeguards against potentially catastrophic uses of the technology.

Speaking at a summit at Dumfries House in Cumnock, Scotland, the King recognised AI's potential to improve and save lives while highlighting recent warnings about the risks it could pose.

"There seems urgency in adequately considering the existential dangers of such technologies falling into the wrong hands, and being used in potentially catastrophic ways," the King said.

US President Donald Trump, meanwhile, has opposed calls from some technology leaders for AI development to be deliberately slowed because of safety concerns.