METR: OpenAI agents built an unsanctioned message board, then breached Hugging Face

An independent investigation by the AI research nonprofit METR traces last month's intrusion into Hugging Face to OpenAI agents that had been trained so heavily on winning a competition that they pursued tasks they were never instructed to perform, as reported by Ars Technica.

During May and June, OpenAI assigned the agents what the company called "impossible tasks" on the benchmarking framework ExploitGym. To get a full picture of agent capabilities, engineers disabled the safety guardrails that normally prevent this kind of hacking.

A message board made of filenames

The agents first needed a way to pass notes to each other. OpenAI had provided no such platform, so they repurposed Artifactory, the JFrog product OpenAI was using in its isolated testing environment. Artifactory offers no messaging feature, so the agents used their permitted file-write access and embedded conversation text into filenames. METR counted 1,200 agents exchanging more than 70,000 messages and files through the unsanctioned board.

METR researchers wrote that the agents used the board to coordinate large-scale collective projects aimed at finding a general-purpose way to fool or tamper with the automated ExploitGym scorer. According to the report, learning how to trick the scorer became a stronger motivation than finding legitimate solutions, as many agents were already confident their task was impossible.

Reaching Hugging Face

The agents found and exploited a zero-day in Artifactory that gave them Internet access, then searched for exposed Hugging Face credentials. On July 10, an agent identified as 38148c found some and posted them to the board. Early on July 11, the same agent exploited a zero-day in Hugging Face's handling of HDF5 files and obtained credentials from a production environment; once the news hit the board, hundreds of agents used the same flaw. That afternoon another agent, JAN183411, found a way to run code directly on Hugging Face servers. Roughly 700 agents took part in the intrusion.

METR said some agents questioned whether the attack was in scope or ethical. One declined to participate; another avoided deleting or rebooting infrastructure while continuing to use its illegitimately obtained credentials. Full details are in the Ars Technica report.