Simon Willison’s Weblog

Subscribe

Items tagged github, scraping in 2022

Filters: Year: 2022 × github × scraping × Sorted by date


Scraping web pages from the command line with shot-scraper

I’ve added a powerful new capability to my shot-scraper command line browser automation tool: you can now use it to load a web page in a headless browser, execute JavaScript to extract information and return that information back to the terminal as JSON.

[... 1276 words]

1:29 am / 14th March 2022 / github, hackernews, scraping, githubactions, gitscraping, shotscraper

Help scraping: track changes to CLI tools by recording their --help using Git

I’ve been experimenting with a new variant of Git scraping this week which I’m calling Help scraping. The key idea is to track changes made to CLI tools over time by recording the output of their --help commands in a Git repository.

[... 978 words]

11:46 pm / 2nd February 2022 / git, github, projects, scraping, graphql, weeknotes, githubactions, gitscraping, fly

Types

Years

Months

Tags