Simon Willison’s Weblog

Subscribe

TILs

Filters: Sorted by date

TIL Storing files in an S3 bucket between GitHub Actions runs — For my [git-history live demos](https://github.com/simonw/git-history/issues/30) I needed to store quite large files (~200MB SQLite databases) in between GitHub Actions runs, to avoid having to recreate the entire file from scratch every time.
7th Dec 2021, 7:52 pm
TIL __init_subclass__ — David Beazley [on Twitter](https://twitter.com/dabeaz/status/1466731368956809219) said:
3rd Dec 2021, 7:19 pm
TIL Ignoring a line in both flake8 and mypy — I [needed to tell](https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982133970) both `flake8` and `mypy` to ignore the same line of code.
1st Dec 2021, 2:16 am
TIL Reusing an existing Click tool with register_commands — The [register_commands](https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli) plugin hook lets you add extra sub-commands to the `datasette` CLI tool.
29th Nov 2021, 5:32 pm
TIL Publishing a Web Component to npm — I tried this for the first time today with my highly experimental [datasette-table](https://www.npmjs.com/package/datasette-table) Web Component. Here's [the source code](https://github.com/simonw/datasette-table/tree/0.1.0) for version 0.1.0.
28th Nov 2021, 6:47 pm
TIL Pausing traffic and retrying in Caddy — A pattern I really like for zero-downtime deploys is the ability to "pause" HTTP traffic at the load balancer, such that incoming requests from browsers appear to take a few extra seconds to return, but under the hood they've actually been held in a queue while a backend server is swapped out or upgraded in some way.
25th Nov 2021, 1:18 am
TIL Assigning a custom subdomain to a Fly app — I deployed an app to [Fly](https://fly.io/) and decided to point a custom subdomain to it.
20th Nov 2021, 8:46 pm
TIL Using build-arg variables with Cloud Run deployments — For [datasette/issues/1522](https://github.com/simonw/datasette/issues/1522) I wanted to use a Docker build argument in a `Dockerfile` that would then be deployed to Cloud Run.
20th Nov 2021, 12:24 am
TIL Using cog to update --help in a Markdown README file — My [csvs-to-sqlite README](https://github.com/simonw/csvs-to-sqlite/blob/main/README.md) includes a section that shows the output of the `csvs-to-sqlite --help` command ([relevant issue](https://github.com/simonw/csvs-to-sqlite/issues/82)).
18th Nov 2021, 4:27 pm
TIL Planning parallel downloads with TopologicalSorter — For [complicated reasons](https://github.com/simonw/datasette/issues/878) I found myself wanting to write Python code to resolve a graph of dependencies and produce a plan for efficiently executing them, in parallel where possible.
16th Nov 2021, 9:41 pm
TIL Using Tesseract.js to OCR every image on a page — Pasting this code into a DevTools console should load [Tesseract.js](https://github.com/naptha/tesseract.js) from a CDN, loop through every image loaded by that page (every PNG, GIF, JPG or JPEG), run OCR on them and output the result to the DevTools console.
9th Nov 2021, 8:27 pm
TIL Annotated code for a demo of WebSocket chat in Deno Deploy — Deno Deploy is a hosted Deno service that promises [a multi-tenant JavaScript engine running in 25 data centers across the world](https://deno.com/blog/deploy-beta1/).
7th Nov 2021, 1:34 am
TIL Basic Datasette in Kubernetes — This recipe for deploying the official `datasetteproject/datasette` container in Kubernetes just worked for me. It uses an interesting (possibly nasty?) trick to install plugins and download a SQLite database file on container startup, without needing to bake a brand new container image.
5th Nov 2021, 8:20 pm
TIL Quick and dirty mock testing with mock_calls — I needed to write a test that checked for a really complex sequence of mock calls for [s3-credentials#3](https://github.com/simonw/s3-credentials/issues/3).
3rd Nov 2021, 1:35 am
TIL Using VCR and pytest with pytest-recording — [pytest-recording](https://github.com/kiwicom/pytest-recording) is a neat pytest plugin that makes it easy to use the [VCR library](https://vcrpy.readthedocs.io/), which helps write tests against HTTP resources by automatically capturing responses and baking them into a YAML file to be replayed during the tests.
3rd Nov 2021, 12:51 am
TIL Understanding Kristofer Joseph's Single File Web Component — [Via Brian LeRoux](https://twitter.com/brianleroux/status/1453472609518034944) I found [single-file-web-component.html](https://gist.github.com/kristoferjoseph/c4e47389ae0f0447db175b914e471628) by Kristofer Joseph. It's really clever! It demonstrates how to build a `<hello-world></hello-world>` custom Web Component in a single HTML file, using some neat tricks.
28th Oct 2021, 12:33 am
TIL Removing a git commit and force pushing to remove it from history — I accidentally triggered a commit which added a big chunk of unwanted data to my repository. I didn't want this to stick around in the history forever, and no-one else was pulling from the repo, so I decided to use force push to remove the rogue commit entirely.
22nd Oct 2021, 7:25 pm
TIL Using the sqlite3 Python module in Pyodide - Python WebAssembly — [Pyodide](https://github.com/pyodide/pyodide) provides "Python with the scientific stack, compiled to WebAssembly" - it's an incredible project which lets you run a full working Jupyter notebook, complete with complex packages such as numpy and pandas, entirely in your browser without any server-side Python component running at all.
18th Oct 2021, 6:04 pm
TIL Using Fabric with an SSH public key — Inspired by [this tweet](https://twitter.com/driscollis/status/1445772718507376646) by Mike Driscoll I decided to try using Fabric to run commands over SSH from a Python script, using a public key for authentication.
6th Oct 2021, 6:22 pm
TIL Loading lit from Skypack — [Lit 2](https://lit.dev/blog/2021-09-21-announcing-lit-2/) stable was released today, offering a tiny, feature-full framework for constructing web components using modern JavaScript.
22nd Sep 2021, 1:03 am
TIL Publishing to a public Google Cloud bucket with gsutil — I decided to publish static CSV files to accompany my https://cdc-vaccination-history.datasette.io/ project, using a Google Cloud bucket (see [cdc-vaccination-history issue #9](https://github.com/simonw/cdc-vaccination-history/issues/9)).
20th Sep 2021, 8:13 pm
TIL Configuring auto-update for an Electron app — This is _almost_ really simple. I used [electron/update-electron-app](https://github.com/electron/update-electron-app) for it, the instructions for which are:
14th Sep 2021, 3:57 am
TIL Cumulative total over time in SQL — This is a quick trick for creating a cumulative chart of the total number of items created over time based just on their creation date.
13th Sep 2021, 3:33 pm
TIL Bundling Python inside an Electron app — For [Datasette Desktop](https://datasette.io/desktop) I chose to bundle a full version of Python 3.9 inside my `Datasette.app` application. I did this in order to support installation of plugins via `pip install` - you can read more about my reasoning in [Datasette Desktop—a macOS desktop application for Datasette](https://simonwillison.net/2021/Sep/8/datasette-desktop/).
8th Sep 2021, 11:38 pm
TIL Signing and notarizing an Electron app for distribution using GitHub Actions — I had to figure this out for [Datasette Desktop](https://github.com/simonw/datasette-app).
8th Sep 2021, 5:41 pm
TIL Attaching a generated file to a GitHub release using Actions — For [Datasette Desktop](https://github.com/simonw/datasette-app) I wanted to run an action which, when I created a release, would build an asset for that release and then upload and attach it.
8th Sep 2021, 5:04 am
TIL Open external links in an Electron app using the system browser — For [Datasette.app](https://github.com/simonw/datasette-app) I wanted to ensure that links to external URLs would [open in the system browser](https://github.com/simonw/datasette-app/issues/34).
2nd Sep 2021, 9:15 pm
TIL Using the Chrome DevTools console as a REPL for an Electron app — I figured out how to use the Chrome DevTools to execute JavaScript interactively inside the Electron main process. I always like having a REPL for exploring APIs, and this means I can explore the Electron and Node.js APIs interactively.
31st Aug 2021, 9:09 pm
TIL Calculating the AQI based on the Purple Air API for a sensor — [Purple Air](https://www.purpleair.com/) sensors have an API at `https://www.purpleair.com/map.json?show=SENSOR-ID-HERE`, which returns JSON that looks something like this:
31st Aug 2021, 7:46 pm
TIL Searching all columns of a table in Datasette — I came up with this trick today, when I wanted to run a `LIKE` search against every column in a table.
23rd Aug 2021, 6:48 pm

Years