1 item tagged “andrewwooster”
robots.txt Adventure. Interesting notes from crawling 4.6 million robots.txt, including 69 different ways in which the word “disallow” can be mis-spelled. 
22nd September 2007, 12:36 am
Sign in with 
OpenID
robots.txt Adventure. Interesting notes from crawling 4.6 million robots.txt, including 69 different ways in which the word “disallow” can be mis-spelled. 22nd September 2007, 12:36 am
andrewwooster on del.icio.us, Flickr, Technorati
