Simon Willison’s Weblog

Monday, 3rd February 2003

Mechanize the web

Via Keith Devens, Screen-scraping with WWW::Mechanize describes how Perl’s WWW::Mechanize module can be used to grab information from sites that require a user login. I’ve always dismissed screen scraping as something of a wasted effort, given the fact that a major rewrite of the scraper is required whenever the target site tweaks its HTML. This article has encouraged me to reconsider—some of the functionality in WWW::Mechanise is fantastic:

[... 262 words]

2003 » February

MTWTFSS
     12
3456789
10111213141516
17181920212223
2425262728