Simon Willison’s Weblog

Subscribe

Calendars and crawlers

20th February 2003

Douglas Bowman has been having some amusing problems with robots and his calendar. The calendar, visible on every page of the site, automatically adds a “next month” and “previous month” link to allow surfers to browser through the archive in both directions. Unfortunately, Doug ommitted the logic to stop showing a “previous month” link when there were no earlier entries. An enterprising crawler started following the links, and didn’t stop until it had reached 1542!

I’ve written a few dynamic calendars in the past and I’m pretty sure at least one of them was susceptible to this kind of bug. Definitely one to watch out for.

This is Calendars and crawlers by Simon Willison, posted on 20th February 2003.

Next: Get a better browser!

Previous: DNS mess

Previously hosted at http://simon.incutio.com/archive/2003/02/20/calendarsAndCrawlers