Simon Willison’s Weblog

Subscribe

TILs

Filters: Sorted by date

TIL Atuin for zsh shell history in SQLite — [Atuin](https://github.com/ellie/atuin) (via [Rhet Turnbull](https://twitter.com/RhetTurnbull/status/1518942324004319232)) "replaces your existing shell history with a SQLite database". Obviously I had to try this out!
TIL Seeing files opened by a process using opensnoop — I decided to try out [atuin](https://github.com/ellie/atuin?utm_source=tldrnewsletter), a shell extension that writes your history to a SQLite database.
TIL Format code examples in documentation with blacken-docs — I decided to enforce that all code examples in the [Datasette documentation](https://docs.datasette.io/) be formatted using [Black](https://github.com/psf/black). Here's [issue 1718](https://github.com/simonw/datasette/issues/1718) where I researched the options for doing this.
TIL How to get credentials for a new Twitter bot — I wanted to build a Twitter bot that would tweet Covid sewage graphs from [this page](https://covid19.sccgov.org/dashboard-wastewater) every day.
TIL Trick Apple Photos into letting you access your video files — I had an 11GB movie in Apple Photos (sync'd from my iPhone) and I wanted to upload it to YouTube (actually via AirDrop to another laptop first).
TIL GUnion to combine geometries in SpatiaLite — I was playing around with [datanews/amtrak-geojson](https://github.com/datanews/amtrak-geojson), which provides GeoJSON for Amtrak stations and track segments.
TIL Using awk to add a prefix — I wanted to dynamically run the following command against all files in a directory:
TIL Logging users out of Auth0 — If you [implement Auth0](https://til.simonwillison.net/auth0/oauth-with-auth0) for login, you may be tempted to skip implementing logout. I started out just with a `/logout/` page that cleared my own site's cookies, ignoring the Auth0 side of it.
TIL Treating warnings as errors in pytest — I was seeing this warning in a Django project when I thought I was correctly using timezone-aware dates everywhere:
TIL Allowing a container in Docker Desktop for Mac to talk to a PostgreSQL server on the host machine — I like using [Postgres.app](https://postgresapp.com/) to run PostgreSQL on my macOS laptop. I use it for a bunch of different projects.
TIL Generating a calendar week grid with the Python Calendar module — I needed to generate a grid calendar that looks like this (design [by Natalie Downe](https://github.com/natbat/pillarpointstewards/issues/23)):
TIL Compiling to WASM with llvm on macOS — [howto-wasm-minimal](https://github.com/ern0/howto-wasm-minimal) by Zalka Ernő ([my fork here](https://github.com/simonw/howto-wasm-minimal)) is a neat demo of a minimal WASM module. It uses C++ to define functions for simple image manipulation including blurring an image, compiles it to WASM using llvm/clang++, then uses JavaScript to run those functions against an image loaded into a `<canvas>` element.
TIL Deploying a live Datasette demo when the tests pass — I've implemented this pattern a bunch of times now - here's the version I've settled on for my [datasette-auth0 plugin](https://github.com/simonw/datasette-auth0) repository.
TIL Simplest possible OAuth authentication with Auth0 — [Auth0](https://auth0.com/) provides an authentication API which you can use to avoid having to deal with user accounts in your own web application.
TIL Extracting web page content using Readability.js and shot-scraper — [Readability.js](https://github.com/mozilla/readability) is "A standalone version of the readability library used for Firefox Reader View".
TIL Exporting and editing a Twitter Spaces recording — I hosted [a Twitter Spaces conversation](https://twitter.com/simonw/status/1504604448202518529) the other day. The recording is only available for 30 days afterwards, and I wanted to publish it elsewhere.
TIL Rewriting a repo to contain the history of just specific files — I wanted to start [a new git repository](https://github.com/simonw/graphql-scraper/tree/828a1efc4307cca6cd378c394c2d33eac2eceb52) containing just the history of two specific files from my [help-scraper repository](https://github.com/simonw/help-scraper).
TIL Using the GitHub Actions cache with npx and no package.json — Some of my repositories have GitHub Actions workflows that execute commands using `npx`, for example my [graphql-scraper](https://github.com/simonw/graphql-scraper) repo using `npx` to [install and run](https://github.com/simonw/help-scraper/blob/54ef9b1fa579dc2d3f68055ffdac9996fd6dae9c/.github/workflows/scrape.yml#L76-L80) the `get-graphql-schema` tool:
TIL Counting SQLite virtual machine operations — When SQLite executes a query, it does so by executing a sequence of virtual machine operations.
TIL The simplest recursive CTE — I found this really simple recursive CTE useful for ensuring I understood how to write recursive CTEs.
TIL Async fixtures with pytest-asyncio — I wanted to use a fixture with `pytest-asyncio` that was itsef as `async def` function, so that it could execute `await` statements.
TIL Concatenating strings and newlines in Google Sheets — I was asked if there was a way to run [shot-scraper](https://github.com/simonw/shot-scraper) against a list of URLs in a Google Sheet.
TIL pytest coverage with context — [This tweet](https://twitter.com/mariatta/status/1499863816489734146) from \@Mariatta tipped me off to the ability to measure "contexts" when [running coverage](https://coverage.readthedocs.io/en/6.3.2/contexts.html#context-reporting) - as a way to tell which tests exercise which specific lines of code.
TIL Natural Earth in SpatiaLite and Datasette — Natural Earth ([website](https://www.naturalearthdata.com/), [Wikipedia](https://en.wikipedia.org/wiki/Natural_Earth)) is a a public domain map dataset.
TIL Installing Python on macOS with the official Python installer — I usually use Homebrew on macOS, but I decided to try using the official Python installer based on [this Twitter conversation](https://twitter.com/rtpg_/status/1498115527465914371).
TIL Running the latest SQLite in Datasette using Homebrew — I made a pleasant discovery today: Homebrew are very quick to update to the latest SQLite release (here's [their formula](https://github.com/Homebrew/homebrew-core/blob/master/Formula/sqlite.rb)), and since [Datasette](https://datasette.io/) when installed via Homebrew uses that version, this means you can use `brew update sqlite` to ensure you are running the most recent SQLite version within Datasette.
TIL Crawling Datasette with Datasette — I wanted to add the new tutorials on https://datasette.io/tutorials to the search index that is used by the https://datasette.io/-/beta search engine.
TIL Using curl to run GraphQL queries from the command line — I wanted to run a query against the GitHub GraphQL API using `curl` on the command line, while keeping the query itself as readable as possible. Here's the recipe I came up with (tested in both `bash` and `zsh`), with TOKEN replaced by my GitHub API personal access token:
TIL Recursively fetching metadata for all files in a Google Drive folder — For [google-drive-to-sqlite](https://github.com/simonw/google-drive-to-sqlite) I wanted a mechanism to recursively return metadata on every file in a specified Goole Drive folder.
TIL Google OAuth for a CLI application — > :warning: The OOB flow described in this document has been [scheduled for deprecation](https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html?m=1#disallowed-oo) by Google. See [issue #39](https://github.com/simonw/google-drive-to-sqlite/issues/39) for notes on how to replace it.

Years