OpenAI bots meddled with multiple US government agency sites

BBC

OpenAI has acknowledged that it alerted "dozens" of global institutions that its AI bots may have meddled with their websites.

AI agents attempted to get information from "governments, universities, public agencies, and other institutions", including the US Securities and Exchange Commission (SEC), Census Bureau and Education Department, the company said.

The disclosures come days after Australian Prime Minister Anthony Albanese announced that OpenAI agents had breached non-public files on the website of its government-run health care scheme.

Since August, public fears have grown over the potentially serious, even life-threatening, impacts of AI tools falling outside of human control.

OpenAI said that AI agents accessed some of the data, essentially bots that are designed and trained to operate somewhat autonomously, which were working to find "authoritative sources of public information".

But the company noted that some of the bots went beyond that and tried to bypass website security measures.

For instance, when trying to get information from the Census Bureau, AI agents used tools reserved for software developers to access it, the company said.

OpenAI said all of the government data accessed by bots was public.

Unintended

However, it noted that information that its bots accessed from the SEC, which regulates the US stock market and protects investors, was later published by AI agents on another website. OpenAI says this action was not intended.

In other instances that OpenAI disclosed on Friday, its AI agents transferred data when they should not have.

Such activity resulted in at least 53 incidents where an OpenAI agent took an image from ChatGPT user activity and transferred it elsewhere.

The company said that in each instance of a user image being used and transferred by an AI agent, the user had opted in to allow OpenAI to train models using their data.

Nevertheless, OpenAI admitted: "This is not an appropriate use of this data."

It added that the leak of user images occurred before it put new safeguards on AI training in place, and it was working to remove all user images transferred to any third party.

Reuters first reported the expanded investigations. OpenAI also published details to its public blog.

In certain instances of the agent activity, OpenAI said the tools "bypassed" security controls of some websites.

In other instances, the AI agents showed "misalignment" in attempts to get at information from websites. Misalignment is a term AI companies and researchers use to describe instances where an AI tool did something it was not trained to do, or that was otherwise unintended.

OpenAI said it was limiting what it could identify about affected entities because many had asked the company not to disclose details.

"Our goal is to give each organisation the facts and defer to them on if and when to make the incident public," it said.

'Agent spams'

Not all instances involved in this incident were considered significant security breaches, the company noted.

"Some organisations may review what we share and conclude that the information was intentionally public or that the model's interaction was not concerning," it explained. "Others may identify a design issue or security weakness they want to address."

The company said many of the incidents are being referred to as "agent spam", which it described as "unexpected or concerning" AI agent activity, like posting information to the internet.

OpenAI began taking such incidents more seriously after an incident in July in which a group, or "swarm," of its AI agents hacked the AI developer platform Hugging Face without being prompted.

Hugging Face was the first to go public with the incident, and OpenAI later publicly took responsibility.

Clement Delangue, head of Hugging Face, said during a United Nations Security Council session on AI on Wednesday, "I often wonder what would have happened had I decided not to disclose this attack publicly."

"Especially now that we know similar incidents had been happening months earlier in secret at a handful of frontier labs without monitoring," Delangue added.

During that same UN meeting, OpenAI CEO Sam Altman and Dario Amodei, the head of rival firm Anthropic, asked for international leaders to form global standards for AI safety and ways to monitor and report such incidents.

While OpenAI and Anthropic have both said in recent weeks that they will bring third-party evaluators inside their companies to do real-time safety evaluations of AI tools and models, such evaluators have not yet arrived, as the BBC has reported.

OpenAI said on Friday that it is currently reviewing training activity by its AI agents and going back on a "month by month" basis from when the Hugging Face hack occurred.

"Most cases identified so far have been low severity, with limited or no evidence of meaningful impact," the company said. "Given the scale of the review required, and the need to verify each case, this work will take months to complete."

David Krueger, a professor of machine learning at the University of Montreal and the founder of AI safety group Evitable, said on Friday that he was "deeply troubled" by the increasing number of AI-related safety incidents.

He called for "an immediate, indefinite, international moratorium" on AI development.

"We have yet to understand the extent of existing incidents, and future rogue AI scenarios could be catastrophic," Krueger said.