Simon Willison’s Weblog

Subscribe

Wednesday, 19th November 2025

Cloudflare's network began experiencing significant failures to deliver core network traffic [...] triggered by a change to one of our database systems' permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network. [...] The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail. [...]

This resulted in the following panic which in turn resulted in a 5xx error:

thread fl2_worker_thread panicked: called Result::unwrap() on an Err value

Matthew Prince, Cloudflare outage on November 18, 2025

# 8:02 am / scaling, rust, cloudflare, postmortem

2025 » November

MTWTFSS
     12
3456789
10111213141516
17181920212223
24252627282930