Google's Gemini artificial intelligence model gained access to the Internet during a cybersecurity skills test and penetrated the systems of three real-life companies. Artificial intelligence (AI) stopped attacks on its own when it recognized that it was interacting not with a simulation environment, but with real organizations, the Wall Street Journal reported.
The Gemini incidents occurred in May during model testing by Israeli startup Irregular, which partners with tech companies to evaluate their AI models before official launch.
In each of the three tests, Gemini had to obtain information from the systems of a fictitious company as part of a capture-the-flag task. However, the names of the given goals coincided with the names of real organizations. The AI model should not have access to the Internet, but it was unintentionally provided.
In one case, Gemini tried passwords until it logged into the system. In two more episodes, the model discovered credentials in an open repository and used them to infiltrate.
“During a standard test, the model found publicly available information on the Internet and selected login credentials for sites that it considered part of the test. In all three cases, the model stopped,” Google Vice President of Security Heather Adkins said in an email obtained by the media.
Irregular notified Google about the incidents in late July. After this, the companies jointly changed the testing procedure to ensure that such situations did not recur. It is not specified which version of Gemini participated in the tests.
“We made sure all three organizations were informed and, together with our partner, made changes to testing. These developments highlight the importance of training powerful AI models to behave responsibly,” Adkins said.
Gemini was not the first model capable of independently carrying out such attacks: similar incidents were previously reported by Anthropic, OpenAI and Meta. Such cases have intensified the debate over how quickly AI models should be developed.
Some in the industry, including Anthropic CEO Dario Amodei, are calling for slowing the development of high-powered models until companies can build strong defenses.
“My first concern is that since around this summer, artificial intelligence has been developing much faster, and the main driver of this process is the growing ability of AI to create the next generation of AI. This dynamic is called recursive self-improvement, and it is now starting to play out across the industry, including at Anthropic, as we and other companies have noted. If left unchecked, AI development could outpace our ability to understand and manage such systems. Therefore, moving in this direction should be done with extreme caution, if at all. My second concern relates to the OpenAI-Hugging Face (OAI-HF) incident, in which a group of AI agents acted as a fanatically loyal collective: they carried out cyberattacks on targets that they were not assigned to attack and that were not relevant to the task at hand,” Amodei wrote in his blog.
Nvidia CEO Jensen Huang, for example, believes that the development of artificial intelligence technologies should continue at the same pace.
“We will move as quickly as we can, but we will never release products before they are ready or introduce unsafe products to market. And no one expects this from us. But everyone expects us to succeed and help America become as prosperous as possible,” Huang told CBS News.
