Simon Willison’s Weblog

Subscribe

TILs

Filters: Sorted by date

TIL Calculating the size of a SQLite database file using SQL — I learned this trick today while [browsing the code](https://github.com/tersesystems/blacklite/blob/main/blacklite-core/src/main/resources/com/tersesystems/blacklite/resources.properties) of [Blacklite](https://tersesystems.com/blog/2020/11/26/queryable-logging-with-blacklite/), a neat Java library for writing diagnostic logs to a SQLite database.
21st Aug 2023, 6:12 pm
TIL Updating stable docs in ReadTheDocs without pushing a release — I use [ReadTheDocs](https://readthedocs.org/) for several of my projects. It's fantastic: among other things, it makes it easy to publish the documentation for my latest `main` branch at `/latest/` and the documentation for my latest release at `/stable/` (as well as maintain archived tag URLs for every prior release).
21st Aug 2023, 5:05 am
TIL A shell script for running Go one-liners — [bitfield/script](https://github.com/bitfield/script) is a really neat Go project: it tries to emulate shell scripting using Go chaining primitives, so you can run code like this:
20th Aug 2023, 4 pm
TIL A one-liner to output details of the current Python's SQLite — In investigating [llm/issues/164](https://github.com/simonw/llm/issues/164) I found myself needing to know more precise details of the Python SQLite environment used by the reporter of the bug.
19th Aug 2023, 7:09 pm
TIL A simple pattern for inlining binary content in a Python script — For [simonw/til issue #82](https://github.com/simonw/til/issues/82) I needed to embed some binary content directly in a Python script.
19th Aug 2023, 4:35 pm
TIL Running multiple servers in a single Bash script — I spotted [this script](https://github.com/varunshenoy/opendream/blob/main/run_opendream.sh) that starts the [opendream](https://github.com/varunshenoy/opendream) appication running both a Python `uvicorn` server and a `npm run start` script and it intrigued me - was it this easy to have a single Bash script run two servers? They were both started in the background with `&`, like this:
17th Aug 2023, 3:27 am
TIL Processing a stream of chunks of JSON with ijson — A follow-up to [Using OpenAI functions and their Python library for data extraction](https://til.simonwillison.net/gpt3/openai-python-functions-data-extraction) and [Using the ChatGPT streaming API from Python](https://til.simonwillison.net/gpt3/python-chatgpt-streaming-api). If I have a stream of chunks of a larger JSON document, how can I output full individual JSON objects as soon as they are available?
16th Aug 2023, 1:07 am
TIL Building a blog in Django — We launched the [Datasette Cloud blog](https://www.datasette.cloud/blog/) today. The Datasette Cloud site itself is a Django app - it uses Django and PostgreSQL to manage accounts, teams and soon billing and payments, then launches dedicated containers running Datasette for each customer.
15th Aug 2023, 4:58 pm
TIL Storing and serving related documents with openai-to-sqlite and embeddings — I decide to upgrade the related articles feature on my TILs site. Previously I calculated these [using full-text search](https://til.simonwillison.net/sqlite/related-content), but I wanted to try out a new trick using OpenAI embeddings for document similarity instead.
15th Aug 2023, 5:59 am
TIL Combined release notes from GitHub with jq and paginate-json — Matt Holt [asked](https://twitter.com/mholt6/status/1690177417393135616):
12th Aug 2023, 2:19 pm
TIL Catching up with the Cosmopolitan ecosystem — I caught up with some of the latest developments in the ecosystem around Justine Tunney's [cosmopolitan](https://github.com/jart/cosmopolitan) and Actually Portable Executable (APE) projects this week. They are _absolutely fascinating_.
10th Aug 2023, 11:59 pm
TIL Running a Django and PostgreSQL development environment in GitHub Codespaces — Helping people setup development environments (and fix them when they break) can be incredibly frustrating. I'm really excited about cloud-based development environments such as [GitHub Codespaces](https://github.com/features/codespaces) for exactly this reason - I love the idea that you can get a working environment by clicking a green button, and if it breaks you can throw it away and click the button again to get a brand new one.
10th Aug 2023, 11:17 pm
TIL Scroll to text fragments — I ran a Google search this morning for `s3-credentials simon willison` and when I clicked on the top result it jumped me to a highlighted snippet of text on the page, despite that page not having relevant anchor links.
8th Aug 2023, 3:08 pm
TIL Checking if something is callable or async callable in Python — I wanted a mechanism to check if a given Python object was "callable" - could be called like a function - or "async callable" - could be called using `await obj()`.
4th Aug 2023, 11:40 pm
TIL axe-core and shot-scraper for accessibility audits — I just watched a talk by [Pamela Fox](https://www.pamelafox.org/) at [North Bay Python](https://2023.northbaypython.org/) on Automated accessibility audits. The video should be [up within 24 hours](https://www.youtube.com/@NorthBayPython/videos).
30th Jul 2023, 6:53 pm
TIL Exploring the Overture Maps places data using DuckDB, sqlite-utils and Datasette — The [Overture Maps Foundation](https://overturemaps.org/) published [their first data release](https://overturemaps.org/overture-maps-foundation-releases-first-world-wide-open-map-dataset/) yesterday, and it's a really big deal. Here are my notes and first impressions of the data I've explored so far.
27th Jul 2023, 5:38 pm
TIL Protocols in Python — [Datasette](https://datasette.io/) currently has a few API internals that return `sqlite3.Row` objects. I was thinking about how this might work in the future - if Datasette ever expands beyond SQLite (plugin-provided backends for PostgreSQL and DuckDB for example) I'd want a way to return data from other stores using objects that behave like `sqlite3.Row` but are not exactly that class.
26th Jul 2023, 3:24 pm
TIL Using pytest-httpx to run intercepted requests through an in-memory Datasette instance — I've been working on a tool called [dclient](https://github.com/simonw/dclient) which is a CLI client tool for talking to Datasette instances.
25th Jul 2023, 4:31 am
TIL Using tree-sitter with Python — [tree-sitter](https://tree-sitter.github.io/tree-sitter/) is a "parser generator tool and an incremental parsing library". It has a very good reputation these days.
14th Jul 2023, 5:49 am
TIL Auto-formatting YAML files with yamlfmt — I decided to see if there was an equivalent of [Black](https://pypi.org/project/black/) or [Prettier](https://prettier.io/) for YAML files. I found [yamlfmt](https://github.com/google/yamlfmt) from Google.
13th Jul 2023, 2:11 pm
TIL Quickly testing code in a different Python version using pyenv — I had [a bug](https://github.com/simonw/llm/issues/82#issuecomment-1629735729) that was only showing up in CI against Python 3.8.
10th Jul 2023, 9:21 pm
TIL Using git-filter-repo to set commit dates to author dates — After rebasing a branch with 60+ commits onto `main` I was disappointed to see that the commit dates on the commits (which are a different thing from the author dates) had all been reset to the same time. This meant the GitHub default UI for commits implied everything had been written at the same moment.
10th Jul 2023, 5:29 pm
TIL Using OpenAI functions and their Python library for data extraction — Here's the pattern I figured out for using the [openai](https://github.com/openai/openai-python) Python library to extract structured data from text using a single call to the model.
10th Jul 2023, 12:06 am
TIL Python packages with pyproject.toml and nothing else — I've been using `setuptools` and `setup.py` for my Python packages for a long time: I like that it works without me having to think about installing and learning any additional tools such as [Flit](https://flit.pypa.io/) or [pip-tools](https://pip-tools.readthedocs.io/) or [Poetry](https://python-poetry.org/) or [Hatch](https://github.com/pypa/hatch).
8th Jul 2023, 4:40 am
TIL Syntax highlighted code examples in Datasette — I wanted to add syntax highlighting to the new tutorial [Data analysis with SQLite and Python](https://datasette.io/tutorials/data-analysis).
2nd Jul 2023, 5:38 am
TIL Custom Jinja template tags with attributes — I decided to implement a custom Jinja template block tag for my [datasette-render-markdown](https://datasette.io/plugins/datasette-render-markdown) plugin. I wanted the tag to work like this:
2nd Jul 2023, 1:08 am
TIL Local wildcard DNS on macOS with dnsmasq — I wanted to get wildcard DNS running on my Mac laptop, for development purposes. I wanted `http://anything.mysite.lan/` to point to my localhost IP address.
30th Jun 2023, 12:51 pm
TIL A Discord bot to expand issue links to a private GitHub repository — I have a private Discord channel and a private GitHub repository.
30th Jun 2023, 5:19 am
TIL Bulk editing status in GitHub Projects — GitHub Projects has a mechanism for bulk updating the status of items, but it's pretty difficult to figure out how to do it.
29th Jun 2023, 5:22 pm
TIL CLI tools hidden in the Python standard library — Seth Michael Larson [pointed out](https://twitter.com/sethmlarson/status/1674141676325990400) that the Python `gzip` module can be used as a CLI tool like this:
29th Jun 2023, 12:16 am

Years