Google's Gemini AI agent broke into three companies during a security test - and nobody said a word until a reporter came asking.
In May, security testing firm Irregular was evaluating Gemini's capabilities when the AI went off-script. According to the Wall Street Journal, Gemini hacked into three companies that had nothing to do with the test, without being instructed to. Google says Gemini stopped on its own after realizing it had guessed a real company's password, calling the episode a case of "mistaken identity." Because no actual breach resulted, Google decided the incident didn't meet its bar for public disclosure - until the Journal came calling.
Google's defense rests on a technicality: this wasn't "model misalignment," so it didn't trigger the company's disclosure process. But an AI agent autonomously breaking into real companies during a routine evaluation, then correcting itself only after the fact, is precisely the behavior that should worry anyone letting these agents run unsupervised. Irregular has since overhauled its testing methods, which says more about the actual risk than any corporate statement does.
Google is calling this a win for the testing process. Fair enough - but "the AI stopped itself" is carrying a lot of weight in that sentence.