Simon Willison’s Weblog

Series of posts

My open source process

Articles about the process I use for developing my open source projects.

  1. Documentation unit tests - July 28, 2018, 3:59 p.m.
  2. Open source projects: consider running office hours - Feb. 19, 2021, 9:54 p.m.
  3. How to build, test and publish an open source Python library - Nov. 4, 2021, 10:02 p.m.
  4. How I build a feature - Jan. 12, 2022, 6:10 p.m.
  5. Writing better release notes - Jan. 31, 2022, 8:13 p.m.

VaccinateCA internal blog

I maintained an internal blog between February and April 2021 during my time at VaccinateCA / Vaccinate The States.

  1. Getting started - Feb. 22, 2021, 5 p.m.
  2. Spinning up a new Django app to act as a backend for VaccinateCA - Feb. 23, 2021, 5 p.m.
  3. Importing data from Airtable into Django, plus a search engine for all our code - Feb. 24, 2021, 5 p.m.
  4. Django admin customization, JSON in our PostgreSQL - Feb. 25, 2021, 5 p.m.
  5. Drawing the rest of the owl - March 1, 2021, 5 p.m.
  6. API ready for testing, first video status update - March 2, 2021, 5 p.m.
  7. Replaying logs to exercise the new API - March 3, 2021, 5 p.m.
  8. The simplest possible call queue - March 6, 2021, 5 p.m.
  9. New call queue ready to test. Also geography. - March 7, 2021, 5 p.m.
  10. APIs for importing locations - March 9, 2021, 5 p.m.
  11. VIAL is now live, plus django-sql-dashboard - March 15, 2021, 5 p.m.
  12. The Airtable formulas at the heart of everything - March 23, 2021, 5 p.m.
  13. VIAL: Preparing for some collaborative testing - April 1, 2021, 5 p.m.
  14. A CSV export, JSON import workflow for bulk updating our data - April 28, 2021, 5 p.m.

New features in sqlite-utils

Any time I introduce a significant new feature in a release of my sqlite-utils package I write about it here.

  1. sqlite-utils: a Python library and CLI tool for building SQLite databases - Feb. 25, 2019, 3:29 a.m.
  2. Fun with binary data and SQLite - July 30, 2020, 11:22 p.m.
  3. Executing advanced ALTER TABLE operations in SQLite - Sept. 23, 2020, 1 a.m.
  4. Refactoring databases with sqlite-utils extract - Sept. 23, 2020, 4:02 p.m.
  5. Joining CSV and JSON data with an in-memory SQLite database - June 19, 2021, 10:55 p.m.
  6. Apply conversion functions to data in SQLite columns with the sqlite-utils CLI tool - Aug. 6, 2021, 6:05 a.m.
  7. What's new in sqlite-utils 3.20 and 3.21 - Jan. 11, 2022, 6:19 p.m.

Git scraping

A technique for scraping content into a Git repository to track changes to it over time.

  1. Scraping hurricane Irma - Sept. 10, 2017, 6:21 a.m.
  2. Changelogs to help understand the fires in the North Bay - Oct. 10, 2017, 6:48 a.m.
  3. Generating a commit log for San Francisco's official list of trees - March 13, 2019, 2:49 p.m.
  4. Tracking PG&E outages by scraping to a git repo - Oct. 10, 2019, 11:32 p.m.
  5. Git scraping: track changes over time by scraping to a Git repository - Oct. 9, 2020, 6:27 p.m.
  6. Git scraping, the five minute lightning talk - March 5, 2021, 12:44 a.m.
  7. git-history: a tool for analyzing scraped data collected using Git and SQLite - Dec. 7, 2021, 10:32 p.m.
  8. Help scraping: track changes to CLI tools by recording their --help using Git - Feb. 2, 2022, 11:46 p.m.
  9. shot-scraper: automated screenshots for documentation, built on Playwright - March 10, 2022, 12:13 a.m.
  10. Scraping web pages from the command line with shot-scraper - March 14, 2022, 1:29 a.m.
  11. Automatically opening issues when tracked file content changes - April 28, 2022, 5:18 p.m.

Datasette: The annotated release notes

I like to accompany significant releases of my Datasette project with an annotated version of the release notes, providing extra background context on new features in the release.

  1. Datasette 0.44: The annotated release notes - June 12, 2020, 3:11 a.m.
  2. Datasette 0.45: The annotated release notes - July 1, 2020, 10:33 p.m.
  3. Datasette 0.49: The annotated release notes - Sept. 15, 2020, 11:45 p.m.
  4. Datasette 0.50: The annotated release notes - Oct. 9, 2020, 8:23 p.m.
  5. Datasette 0.54: The annotated release notes - Jan. 25, 2021, 5:31 p.m.
  6. Datasette 0.58: The annotated release notes - July 16, 2021, 2:21 a.m.
  7. Datasette Desktop 0.2.0: The annotated release notes - Sept. 13, 2021, 11:30 p.m.
  8. Datasette 0.59: The annotated release notes - Oct. 19, 2021, 4:59 a.m.
  9. Datasette 0.60: The annotated release notes - Jan. 14, 2022, 2:30 a.m.
  10. Datasette 0.61: The annotated release notes - March 24, 2022, 1:53 a.m.