Simon Willison’s Weblog

Weeknotes: CDC vaccination history fixes, developing in GitHub Codespaces

28th September 2021

I spent the last week mostly surrounded by boxes: we’re completing our move to the new place and life is mostly unpacking now. I did find some time to fix some issues with my CDC vaccination history Datasette instance though.

Fixing my CDC vaccination history site

I started tracking changes made to the CDC’s COVID Data Tracker website back in Feburary. I created a git scraper repository for it as part of my five minute lightning talk on git scraping (notes and video) at this year’s NICAR data journalism conference.

Since then it’s been quietly ticking along, recording the latest data in a git repository that now has 335 commits.

In March I added a script to build the collected historic data into a SQLite database and publish it to Vercel using GitHub. That started breaking a few weeks ago, and it turnoud out that was because the database file had grown in size to the point where it was too large to deploy to Vercel (~100MB).

I got a bug report about this, so I took some time to move the deployment over to Google Cloud Run which doesn’t have a documented size limit (though in my experience starts to creak once you go above about 2GB.)

I also started publishing the raw collected data directly as a CSV file, partly as an excuse to learn how to publish to Google Cloud Storage.

datasette-template-request

I released an extremely simple plugin this week called datasette-template-request—all it does is expose Datasette’s request object in the context passed to custom templates, for people who want to update their custom page based on incoming request parameters.

More notable is how I built the plugin: this is the first plugin I’ve developed, tested and released entirely in my browser using the new GitHub Codespaces online development environment.

I created the new repo using my Datasette plugin template repository, opened it up in Codespaces, implemented the plugin and tests, tried it out using the port forwarding feature and then published it to PyPI using the publish.yml workflow.

Not having to even open a text editor on my laptop (let alone get a new Python development environment up and running) felt really good. I should turn this into a tutorial.

Releases this week

• datasette-template-request: 0.1—2021-09-23
  Expose the Datasette request object to custom templates
• datasette-notebook: 0.1a1—(2 releases total)—2021-09-22
  A markdown wiki and dashboarding system for Datasette
• datasette-render-markdown: 2.0—(8 releases total)—2021-09-22
  Datasette plugin for rendering Markdown
• sqlite-utils: 3.17.1—(87 releases total)—2021-09-22
  Python CLI utility and library for manipulating SQLite databases
• twitter-to-sqlite: 0.22—(28 releases total)—2021-09-21
  Save data from Twitter to a SQLite database

TIL this week

• Publishing to a public Google Cloud bucket with gsutil
• Loading lit from Skypack