Simon Willison’s Weblog

8th December 2003 - Link Blog

On Search: Robots. Everything you need to know about writing a tool to crawl the web

Posted 8th December 2003 at 7:28 pm

Recent articles

Now we have a timeline of the OpenAI accidental attack against Hugging Face - 7th August 2026
One-shotting a Raccoon Heist game using Claude Fable 5 - 5th August 2026
New release of LLM adds support for reasoning traces, OpenAI Responses, server-side tools, and smarter logging - 4th August 2026

This is a link post by Simon Willison, posted on 8th December 2003.

Monthly briefing

Sponsor me for $10/month and get a curated email digest of the month's most important LLM developments.

Pay me to send you less!

Sponsor & subscribe