Simon Willison’s Weblog

robots.txt Adventure. Interesting notes from crawling 4.6 million robots.txt, including 69 different ways in which the word “disallow” can be mis-spelled.

Posted 22nd September 2007 at 12:36 am

Recent articles

Tips on prompting ChatGPT for UK technology secretary Peter Kyle - 3rd June 2025
How often do LLMs snitch? Recreating Theo's SnitchBench with LLM - 31st May 2025
Talking AI and jobs with Natasha Zouves for News Nation - 30th May 2025

robots-txt 10 crawling 11 andrew-wooster 1

Monthly briefing

Sponsor me for $10/month and get a curated email digest of the month's most important LLM developments.

Pay me to send you less!

Sponsor & subscribe