Simon Willison’s Weblog

Subscribe

TILs

Filters: Sorted by date

TIL Pretty-printing all read-only JSON in the Django admin — I have a bunch of models with JSON fields that are marked as read-only in the Django admin - usually because they're recording the raw JSON that was imported from an API somewhere to create an object, for debugging purposes.
8th Mar 2021, 7:02 am
TIL How to run MediaWiki with SQLite on a macOS laptop — Today I [got curious](https://twitter.com/simonw/status/1368414296888070146) about how MediaWiki records page history, so I started digging around and in the process figured out how to run it against a SQLite database on my macOS laptop!
7th Mar 2021, 5:02 am
TIL Show the timezone for datetimes in the Django admin — Django supports storing dates in a database as UTC but displaying them in some other timezone - which is good. But... by default datetimes are shown in the Django admin interface without any clue as to what timezone they are being displayed in.
3rd Mar 2021, 5:17 am
TIL Writing tests for the Django admin with pytest-django — I'm using [pytest-django](https://pytest-django.readthedocs.io/) on a project and I wanted to write a test for a Django admin create form submission. Here's the pattern I came up with:
2nd Mar 2021, 9:08 pm
TIL Granting a PostgreSQL user read-only access to some tables — I wanted to grant a PostgreSQL user (or role) read-only access to a specific list of tables.
26th Feb 2021, 3:58 pm
TIL Adding extra read-only information to a Django admin change page — I figured out this pattern today for adding templated extra blocks of information to the Django admin change page for an object.
26th Feb 2021, 1:49 am
TIL Running tests against PostgreSQL in a service container — I wanted to run some Django tests - using `pytest-django` and with Django configured to pick up the `DATABASE_URL` environment variable via [dj-database-url](https://github.com/jacobian/dj-database-url) - against a PostgreSQL server running in GitHub Actions.
24th Feb 2021, 3:33 am
TIL Show the SQL schema for a PostgreSQL database — This took me longer to figure out than I care to admit.
24th Feb 2021, 3:03 am
TIL Using sphinx.ext.extlinks for issue links — Datasette's [release notes](https://github.com/simonw/datasette/blob/main/docs/changelog.rst) are formatted using Sphinx. Almost every bullet point links to the corresponding GitHub issue, so they were full of lines that look like this:
18th Feb 2021, 1:27 am
TIL Loading radio.garden into SQLite using jq — http://radio.garden/ is an amazing website which displays a 3D globe covered in radio stations and lets you listen to any of them.
18th Feb 2021, 12:44 am
TIL Using io.BufferedReader to peek against a non-peekable stream — When building the [--sniff option](https://github.com/simonw/sqlite-utils/issues/230) for `sqlite-utils insert` (which attempts to detect the correct CSV delimiter and quote character by looking at the first 2048 bytes of a CSV file) I had the need to peek ahead in an incoming stream of data.
15th Feb 2021, 7:17 pm
TIL Handling CSV files with wide columns in Python — Users [were reporting](https://github.com/simonw/sqlite-utils/issues/229) the following error using `sqlite-utils` to import some CSV files:
15th Feb 2021, 7 pm
TIL Shrinking PNG files with pngquant and oxipng — I usually use [Squoosh.app](https://squoosh.app/) to reduce the size of my PNGs, but in this case I had a folder with nearly 50 images in it so I wanted to do it using the command-line.
7th Feb 2021, 7:57 pm
TIL Downloading MapZen elevation tiles — [Via Tony Hirst](https://twitter.com/psychemedia/status/1357280624319553537) I found out about [MapZen's elevation tiles](https://www.mapzen.com/blog/terrain-tile-service/), which encode elevation data in PNG and other formats.
4th Feb 2021, 6:48 pm
TIL Serving MBTiles with datasette-media — The [MBTiles](https://github.com/mapbox/mbtiles-spec) format uses SQLite to bundle map tiles for use with libraries such as Leaflet.
3rd Feb 2021, 11:12 pm
TIL Splitting on commas in SQLite — I had an input string in `x,y,z` format and I needed to split it into three separate values in SQLite. I managed to do it using a confusing combination of the `instr()` and `substr()` functions.
1st Feb 2021, 7:02 pm
TIL Conditionally creating directories in cookiecutter — I wanted my [datasette-plugin](https://github.com/simonw/datasette-plugin) cookiecutter template to create empty `static` and `templates` directories if the user replied `y` to the `include_static_directory` and `include_templates_directory` prompts.
27th Jan 2021, 11:56 pm
TIL Testing cookiecutter templates with pytest — I added some unit tests to my [datasette-plugin](https://github.com/simonw/datasette-plugin) cookiecutter template today, since the latest features involved adding a `hooks/post_gen_project.py` script.
27th Jan 2021, 11:50 pm
TIL Embedding JavaScript in a Jupyter notebook — I recently found out modern browsers include a JavaScript API for creating public/private keys for cryptography.
22nd Jan 2021, 9:52 pm
TIL Syntax highlighting Python console examples with GFM — It turns out [GitHub Flavored Markdown](https://github.github.com/gfm/) can apply syntax highlighting to Python console examples, like this one:
19th Jan 2021, 1:21 am
TIL Fixing broken text encodings with sqlite-transform and ftfy — I was working with a database table that included values that were clearly in the wrong character encoding - values like this:
18th Jan 2021, 8:44 pm
TIL Bulk fetching repository details with the GitHub GraphQL API — I wanted to be able to fetch details of a list of different repositories from the GitHub GraphQL API by name in a single operation.
17th Jan 2021, 7:49 pm
TIL Recovering data from AWS Lightsail using EC2 — I ran into problems with my AWS Lightsail instance: it exceeded the CPU burst quota for too long and was suspended, and I couldn't figure out how to un-suspend it.
16th Jan 2021, 9:07 pm
TIL Identifying column combination patterns in a SQLite table — Given a large, heterogeneous table I wanted to identify patterns in the rows in terms of which columns were not null.
12th Jan 2021, 10 pm
TIL Packaging a Python app as a standalone binary with PyInstaller — [PyInstaller](https://www.pyinstaller.org/) can take a Python script and bundle it up as a standalone executable for macOS, Linux and apparently Windows too (I've not tried it on Windows yet).
4th Jan 2021, 9:54 pm
TIL Using Prettier to check JavaScript code style in GitHub Actions — I [decided](https://github.com/simonw/datasette/issues/1166) to adopt [Prettier](https://prettier.io/) as the JavaScript code style for Datasette, based on my success with [Black](https://github.com/psf/black) for Python code.
31st Dec 2020, 9:55 pm
TIL Using Jest without a package.json — I wanted to try out [Jest](https://jestjs.io/) for writing JavaScript unit tests, in a project that wasn't set up with `package.json` and other NPM related things.
30th Dec 2020, 10:36 pm
TIL Relinquishing control in Python asyncio — `asyncio` in Python is a form of co-operative multitasking, where everything runs in a single thread but asynchronous tasks can yield to other tasks to allow them to execute.
29th Dec 2020, 7:25 pm
TIL Replicating SQLite with rqlite — I tried out [rqlite](https://github.com/rqlite/rqlite) today, a "lightweight, distributed relational database, which uses SQLite as its storage engine". It uses the Raft consensus algorithm to allow multiple SQLite instances to elect a leader and replicate changes amongst themselves.
28th Dec 2020, 7:22 pm
TIL Skipping CSV rows with odd numbers of quotes using ripgrep — I'm working with several huge CSV files - over 5 million rows total - and I ran into a problem: it turned out there were a few lines in those files that imported incorrectly because they were not correctly escaped.
12th Dec 2020, 3:50 am

Years