Simon Willison’s Weblog

Subscribe

What is currently the best technology stack for web scraping?

10th December 2013

My answer to What is currently the best technology stack for web scraping? on Quora

PhantomJS combined with CasperJS is pretty fantastic—it runs a full, headless copy of a Webkit browser so it can operate against a real DOM, execute JavaScript properly, even grab full rendered screenshots of areas of the page but is still easy to automate.
Posted 10th December 2013 at 5:32 pm · Follow me on Mastodon, Bluesky, Twitter or subscribe to my newsletter

More recent articles

This is What is currently the best technology stack for web scraping? by Simon Willison, posted on 10th December 2013.

Next: What is the Y Combinator cycle?

Previous: Is it fair for someone who calls themselves a "seed" investor to require traction?

Monthly briefing

Sponsor me for $10/month and get a curated email digest of the month's most important LLM developments.

Pay me to send you less!

Sponsor & subscribe