shot-scraper: automated screenshots for documentation, built on Playwright
10th March 2022
shot-scaper is a new tool that I’ve built to help automate the process of keeping screenshots up-to-date in my documentation. It also doubles as a scraping tool—hence the name—which I picked as a complement to my git scraping and help scraping techniques.
Update 13th March 2022: The new
Update 14th October 2022: Automating screenshots for the Datasette documentation using shot-scraper offers a tutorial introduction to using the tool.
As software changes over time, screenshots get out-of-date. I don’t like the idea of stale screenshots, but I also don’t want to have to manually recreate them every time I make the tiniest tweak to the visual appearance of my software.
shot-scraper is a tool for automating this process. You can install it using
pip like this:
pip install shot-scraper shot-scraper install
shot-scraper install line will install the browser it needs to do its job—more on that later.
You can use it in two ways. To take a one-off screenshot, you can run it like this:
shot-scraper https://simonwillison.net/ -o simonwillison.png
Or if you want to take a set of screenshots in a repeatable way, you can define them in a YAML file that looks like this:
- url: https://simonwillison.net/ output: simonwillison.png - url: https://www.example.com/ width: 400 height: 400 quality: 80 output: example.jpg
And then use
shot-scraper multi to execute every screenshot in one go:
% shot-scraper multi shots.yml Screenshot of 'https://simonwillison.net/' written to 'simonwillison.png' Screenshot of 'https://www.example.com/' written to 'example.jpg'
The documentation describes all of the available options you can use when taking a screenshot.
Each option can be provided to the
shot-scraper one-off tool, or can be embedded in the YAML file for use with
The default behaviour for
shot-scraper is to take a full page screenshot, using a browser width of 1280px.
For documentation screenshots you probably don’t want the whole page though—you likely want to create an image of one specific part of the interface.
--selector option allows you to specify an area of the page by CSS selector. The resulting image will consist just of that part of the page.
What if you want to modify the page in addition to selecting a specific area?
The combination of these two options—also available as
selector: keys in the YAML file—should be flexible enough to cover the custom screenshot case for documentation.
A complex example
To prove to myself that the tool works, I decided to try replicating this screenshot from my tutorial.
I made the original using CleanShot X, manually adding the two pink arrows:
This is pretty tricky!
- It’s not this whole page, just a subset of the page
- The cog menu for one of the columns is open, which means the cog icon needs to be clicked before taking the screenshot
- There are two pink arrows superimposed on the image
I decided to do use just one arrow for the moment, which should hopefully result in a clearer image.
I started by creating my own pink arrow SVG using Figma:
And ran this command to generate the screenshot:
shot-scraper multi shot.yml
annotated-screenshot.png image looks like this:
I’m pretty happy with this! I think it works very well as a proof of concept for the process.
How it works: Playwright
Then I noticed that the puppeteer-cli package I was using hadn’t had an update in two years, which reminded me to check out Playwright.
I’ve been looking for an excuse to learn Playwright for a while now, and this project turned out to be ideal.
Playwright is Microsoft’s open source browser automation framework. They promote it as a testing tool, but it has plenty of applications outside of testing—screenshot automation and screen scraping being two of the most obvious.
Playwright is comprehensive: it downloads its own custom browser builds, and can run tests across multiple different rendering engines.
subprocess.run( [ "npx", "playwright", "screenshot", "--full-page", url, output, ], capture_output=True, )
This could take a full page screenshot, but that CLI tool wasn’t flexible enough to take screenshots of specific elements. So I needed to switch to the Playwright programmatic API.
pip install playwright
I was curious how they pulled this off, so I dug inside the
playwright Python package in my
site-packages folder... and found it bundles a full Node.js binary executable and uses it to bridge the two worlds! What a wild hack.
Thanks to Playwright, the entire implementation of
shot-scraper is currently just 181 lines of Python code—it’s all glue code tying together a Click CLI interface with some code that calls Playwright to do the actual work.
I couldn’t be more impressed with Playwright. I’ll definitely be using it for other projects—for one thing, I think I’ll finally be able to add automated tests to my Datasette Desktop Electron application.
Hooking shot-scraper up to GitHub Actions
shot-scraper very much with GitHub Actions in mind.
My shot-scraper-demo repository is my first live demo of the tool.
Once a day, it runs this shots.yml file, generates two screenshots and commits them back to the repository.
One of them is the tutorial screenshot described above.
The other is a screenshot of the list of “recently spotted owls” from this page on owlsnearme.com. I wanted a page that would change on an occasional basis, to demonstrate GitHub’s neat image diffing interface.
I may need to change that demo though! That page includes “spotted 5 hours ago” text, which means that there’s almost always a tiny pixel difference, like this one (use the “swipe” comparison tool to watch 6 hours ago change to 7 hours ago under the top left photo).
Storing image files that change frequently in a free repository on GitHub feels rude to me, so please use this tool cautiously there!
I had ambitious plans to add utilities to the tool that would help with annotations, such as adding pink arrows and drawing circles around different elements on the page.
So really, my next step is to start using this tool for my own projects—to generate screenshots for my documentation.
I’m also very interested to see what kinds of things other people use this for.
More recent articles
- Weeknotes: the Datasette Cloud API, a podcast appearance and more - 1st October 2023
- Things I've learned about building CLI tools in Python - 30th September 2023
- Talking Large Language Models with Rooftop Ruby - 29th September 2023
- Weeknotes: Embeddings, more embeddings and Datasette Cloud - 17th September 2023
- Build an image search engine with llm-clip, chat with models with llm chat - 12th September 2023
- LLM now provides tools for working with embeddings - 4th September 2023
- Datasette 1.0a4 and 1.0a5, plus weeknotes - 30th August 2023
- Making Large Language Models work for you - 27th August 2023
- Datasette Cloud, Datasette 1.0a3, llm-mlc and more - 16th August 2023
- How I make annotated presentations - 6th August 2023