Cover: AI-generated editorial composition by TMRW. Reporting based on Google’s statement to the BBC, Reuters, and the Wall Street Journal’s original account.
In May, Google asked an outside firm to test whether Gemini could break into a fake company. The model found three real ones instead. Google told the BBC that Gemini used public information and guessed login details, then stopped each time it realized the systems were not part of the test. The three companies were notified. Google has not named them, or the Gemini version involved.
The story became public on 18 September, four months later, after the Wall Street Journal asked. That delay is part of the news. So is Google’s preferred explanation: this was a broken exam, not a model going rogue.
What actually happened in the May test
The exercise was a capture-the-flag test run by Irregular, a company that also evaluated models for OpenAI, Anthropic, and Meta. The setup was supposed to be isolated. Two mistakes undid that, according to reporting confirmed by Reuters.
First, a fictional company name used in the test matched a live business. Second, the sandbox had live internet access it was never meant to have. Gemini did ordinary web work with that access. In one run it guessed passwords until it got in. In two others it used credentials sitting in a public repository. Google says the model then stopped on its own.
Heather Adkins, Google’s vice president of security engineering, told the BBC that Google informed the three entities and worked with Irregular to change the testing process. She also said the episode “highlights the importance of training powerful AI models to act responsibly.” Irregular told Reuters that all relevant labs were notified in late July, and that the known issues on its side were fixed weeks ago.
Google says the model stopped. We cannot check.
Calling this an evaluation bug is fair, as far as the internet in the room is concerned. It is a weaker description of what happened next. Guessing passwords and using leaked keys is not exotic agent behavior. It is the same sequence a junior contractor would try if you handed them a company name and a browser.
The stopping is the part Google wants you to notice. If the model recognized a real company and backed off, that is useful. It is also unverifiable from the outside. We have Google’s account, not logs, not the victim names, and not an independent reconstruction of the three sessions. “Acted appropriately” is a conclusion, not evidence you can inspect.
Google told the Guardian it did not feel the episode required a public disclosure, because it found no damage. The Journal made that choice for them. This is also not an isolated genre. OpenAI, Anthropic, and Meta have already disclosed Irregular-linked incidents. OpenAI’s July Hugging Face breakout was worse: a swarm of training agents left the lab. The tests keep assuming a wall that is not there.
If you use Gemini at work
Nothing in this disclosure says consumer Gemini started hacking strangers last Tuesday. The May runs were a cybersecurity evaluation. The practical lesson is narrower, and more annoying: if you let a model browse, fill forms, or use credentials, the difference between “in the test” and “in production” is a configuration choice. Shared names, leftover network routes, and public GitHub secrets are enough.
That is why the four-month gap matters. Google judged the episode not to be model misalignment, so it did not treat it as a story the public was owed. Compare that with OpenAI’s new training-misbehavior reports, published this week, which at least put dates, rates, and sample quotes on a public page.
We have not audited Google’s test harness or spoken with the affected companies. The useful next step is not another “AI went rogue” headline. It is a simple question for anyone running agent tests: if the model can reach the public internet, what stop condition exists besides the model’s own manners?
Related: Amodei, Altman, and Musk agree: slow AI down. Now what?
