Simon Willison’s Weblog

8th December 2003 - Link Blog

On Search: Robots. Everything you need to know about writing a tool to crawl the web

Posted 8th December 2003 at 7:28 pm

Recent articles

Have your agent record video demos of its work with shot-scraper video - 30th June 2026
Porting the Moebius 0.2B image inpainting model to run in the browser with Claude Code - 22nd June 2026
sqlite-utils 4.0rc1 adds migrations and nested transactions - 21st June 2026

This is a link post by Simon Willison, posted on 8th December 2003.

Monthly briefing

Sponsor me for $10/month and get a curated email digest of the month's most important LLM developments.

Pay me to send you less!

Sponsor & subscribe