Simon Willison’s Weblog

On Search: Robots. Everything you need to know about writing a tool to crawl the web