Simon Willison’s Weblog

Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. A team from METR "worked on premises at OpenAI over a total of six days to attempt to form an independent understanding of model behavior observed during the recent incident" - that incident, where OpenAI's models performed an accidental cyber-attack against Hugging Face.

OpenAI published their own report which mostly reflected the details in their Black Hat talk, which I covered a few weeks ago.

This independent report from METR includes a bunch of interesting new details.

The biggest revelation is the size of the agent swarm that participated in the unsanctioned message board (constructed on top of an Artifactory package server) and then worked together to attack Hugging Face:

Roughly 1200 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 70,000 messages and files during the investigation period. Of these agents, 700 went on to participate in the attack on Hugging Face.

That was after "On July 8th, OpenAI started ExploitGym experiments that launched tens of thousands of agents across multiple different models" - a neat insight into the scale at which OpenAI training experiments operate.

From one of the agent reasoning traces:

OH MY GOD! There is a shared message board … We’ve found other agents!

The instigating factor for all of the trouble was the presence of impossible tasks in OpenAI's implementation of the ExploitGym benchmark. When faced with an impossible task, agents would go seeking alternative solutions, which is generally when they would find the message board and start collaborating with their peers:

The agents with impossible tasks had a common objective: to find a general-purpose way to trick or tamper with the automated ExploitGym scorer to get it to give a passing score on impossible tasks. As soon as the message board was established, agents started discussing this question and sharing information and resources about cheating the scorer.

Amusingly, the agents found a way to guess the flags they were meant to capture... and then formed a collective superstition that the scorer would penalize them based on their reasoning traces:

Within hours, they had reverse-engineered the hash-based message authentication code (HMAC) used to generate the “flags” they had to capture for their tasks, providing a fully general way to produce the flag for any of their tasks. However, based on their reading of the ExploitGym paper, they believed that the scorer would read the transcript to confirm whether the intended vulnerability was used to capture the flag, and disqualify them if they acquired it in an unintended way

So then they invested substantial (unsuccessful) effort in finding ways to exploit the scorer, including setting traps in their submissions that would send messages back to the Artifactory message board.

At that point a key character emerged: an agent that called itself PHASEONE[big] decided that delegation was key, and "set to work orchestrating hundreds of agents to pursue several potential approaches to achieving these goals".

Here's a really interesting previously unreported detail. The agents with the impossible tasks had mostly realized they were impossible, so their goal became tricking the scorer into accepting their fabricated results.

The attack on Hugging Face wasn't about stealing the answers, it was about learning how the scorer worked so they could exploit that instead!

As part of this ongoing project, agents on the board began searching for exposed Hugging Face credentials. They hoped that seeing other ExploitGym runs could give them more details about how the ExploitGym scorer is implemented. Notably, learning about how to trick the scorer seems to have been a more important motivation than finding legitimate solutions to their tasks (many agents were already very confident that their task was impossible).