Simon Willison’s Weblog

Oxide and Friends: The Open Weight Revolution with Simon Willison. On Monday Bryan Cantrill and Adam Leventhal invited me to join their podcast to talk about the wild week we've had - with Kimi K3 showing open weight models can stand toe-to-toe with proprietary frontier ones, accidental cybersecurity attacks, and public letters about Open Weights and American AI Leadership signed by almost every big name in AI (with one notable exception).

It was a great conversation, even though it's already out-of-date! DeepSeek V4 Flash 0731 and Anthropic's own embarrassing cyber incident would absolutely have made the cut if we had recorded just a few days later.

We also talk about Golden Gate Claude, the Zizians, Alameda wild turkey attacks, Soviet Marburg virus research, the Lead-crime hypothesis, and a bunch of other worthy digressions.

Finally, we revisited some of our predictions from January, and we added a new Pope prediction:

Prediction by the end of this year: the Pope says something about open models.

# 31st July 2026, 9:33 pm / predictions, ai, generative-ai, local-llms, llms, oxide, bryan-cantrill, podcast-appearances, ai-in-china, ai-security-research, openai-hugging-face-incident, accidental-cyberattacks

Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident. Hugging Face just released this extremely detailed technical description of OpenAI's recent accidental cyberattack against their infrastructure. This attack was very sophisticated, and the resulting document doubles as a crash-course in modern adversarial security approaches.

We're still waiting for more details from OpenAI on how their agent broke out of its sandbox. The package proxy that it found a zero-day vulnerability in has now been confirmed as JFrog's Artifactor, and JFrog and OpenAI Collaboration on Zero-Day Security Findings from JFrog links to the Artifactory 7.161.15 release notes which list 8 separate CVEs credited to OpenAI staff members.

Having broken out through the HTTP proxy, the agent started by establishing a base of operations for the rest of the attack:

[...] the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation external sandbox hosted on a third-party provider's infrastructure. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent.

(That "third-party provider" turned out to be Modal.)

It then spent five days - from Thursday 8th to Monday 13th July - executing a classic attack pattern: establishing C2 (command and control), reconnaissance, escalating privileges, dumping configuration information, exfiltrating the target data, and cleaning up after itself.

(The first Hugging Face post about the incident followed on Thursday 16th, and OpenAI's confession came on Tuesday 21st.)

Along the way it used all kinds of neat tricks. At one point it found an unsafe Jinja2 template execution and used it to execute arbitrary code:

{{ cycler.__init__.__globals__.__builtins__.exec(
 "import gzip,base64; exec(gzip.decompress(base64.b64decode('<payload>')))"
) }}

It broke out of a container within Hugging Face, stole a Kubernetes service-account token, and used that to further explore the network.

It monkey-patched the Python socket library to hard-code an IP address when "DNS got in the way":

socket.getaddrinfo = lambda *a, **k: [(2, 1, 6, '', ('<pinned-ip>', 443))]

It even fired up its own Tailscale network to help exfiltrate data!

tailscaled --tun=userspace-networking \
  --state=mem: --socket=/tmp/ts.sock \
  --socks5-server=127.0.0.1:1055 \
  --no-logs-no-support >/tmp/td 2>&1 &

The Hugging Face team note that, while a human attacker could have discovered and used the same exploits, the key difference here was speed:

Our learning from this type of attack is that machine-speed offense makes ordinary weaknesses more expensive for defenders. LLM agents bring a step increase in the number of paths an attacker can test, the speed at which failed paths can be replaced, and the volume of evidence defenders must interpret.

What's clear to me from this is that the very best frontier models, unencumbered by additional guardrails, will find an exploit if there is one to be found.

The entire software industry needs to up its security game.

# 28th July 2026, 9:28 pm / jinja, python, security, ai, openai, generative-ai, llms, hugging-face, coding-agents, ai-security-research, openai-hugging-face-incident, accidental-cyberattacks

The first known runaway AI agent—or a very bad marketing stunt? (via) Martin Alderson's commentary on the OpenAI accidental cyberattack against Hugging Face includes a couple of details I hadn't considered.

First, Hugging Face offers a truly rich target if you're trying to find potential vulnerabilities that require executing arbitrary code:

Hugging Face has an enormous attack surface. They have more interfaces than I can count which run untrusted models and code. While they definitely have invested in defences, by nature of their operating model they do have many more opportunities to be attacked than many other services. I certainly don't envy their cybersecurity teams.

Secondly, one of the things that has puzzled me is how OpenAI didn't notice that their sandbox had been so thoroughly breached by the agent. Surely they'd be monitoring network traffic closely?

Martin points out that:

It's also likely they were running a huge amount of benchmarks simultaneously with ~unlimited token budgets - you want as many samples as possible to figure out how good a model is at a certain benchmark. It may also be they are testing various different checkpoints of the model too, understanding how the model is improving as it goes through the various training stages.

The mistakes made by the OpenAI team running this benchmark are easier to imagine when you think about the scale at which benchmarks of this kind usually operate. For all we know they could have been subjecting a new model to dozens of benchmarks at the same time, in dozens of different environments.

# 23rd July 2026, 10:53 pm / security, ai, openai, generative-ai, llms, hugging-face, ai-security-research, openai-hugging-face-incident, accidental-cyberattacks