Datasette 0.61: The annotated release notes
24th March 2022
In preparation for Datasette 1.0, this release includes two potentially backwards-incompatible changes. Hashed URL mode has been moved to a separate plugin, and the way Datasette generates URLs to databases and tables with special characters in their name such as
- URLs within Datasette now use a different encoding scheme for tables or databases that include “special” characters outside of the range of
a-zA-Z0-9_-. This scheme is explained here: Tilde encoding. (#1657)
- Removed hashed URL mode from Datasette. The new
datasette-hashed-urlsplugin can be used to achieve the same result, see datasette-hashed-urls for details. (#1661)
I talked about these changes in my weeknotes. These are two major steps forward towards Datasette 1.0, in that implementing them removed a lot of code and complexity from Datasette core.
Datasette also now requires Python 3.7 or higher.
0.60 was the final release to work with Python 3.6, which ended security support last December.
- Databases can now have a custom path within the Datasette instance that is independent of the database name, using the
This was the last change I made before the 0.61 release, and turned out to have a severe bug which prompted me to release 0.61.1 shortly afterwards.
I needed the feature in order to implement the datasette-hashed-urls plugin.
That plugin works by modifying the URL relating to a database in order to incorporate the SHA-256 hash of that database’s contents. This allows all JSON and HTML pages within that database to set a far-future cache header, providing a huge performance boost especially if run behind a caching proxy such as Cloudflare.
Initially it worked by changing the display name of the database too, but Forest Gregg pointed out that exposing all of those hashes in the UI was a pretty poor user experience.
So I decided to split the “route” (the URL to the database—“path” was already in use as the path to the file on disk) from the unique name used to refer to the database internally.
I caught almost all of the places in the code that needed to be updated, but shortly after shipping 0.61 I noticed that Forest had filed issues relating to the one place I had missed!
Those fixes are out now, and datasette-hashed-urls 0.3 should work as advertised.
It’s good open source governance to have one of these. I decided to adopt the Contributor Covenant because it reflected my own values for the project and is used by a large number of projects that I trust.
I didn’t test Datasette against Python 3.10 before it came out and was distressed to find an asyncio bug that caused errors with the project after that version of Python was released!
I’m not going to let that happen again, so Datasette’s test suite now runs against the 3.11 developer preview. Here’s my TIL about how I set that up.
- New datasette.ensure_permissions(actor, permissions) internal method for checking multiple permissions at once. (#1675)
- New datasette.check_visibility(actor, action, resource=None) internal method for checking if a user can see a resource that would otherwise be invisible to unauthenticated users. (#1678)
These two new permissions APIs came out of a larger effort to refactor and simplify Datasette’s core views.
Datasette’s BaseView class included permission logic. I’m trying to shrink that superclass down to the point where I can remove it entirely, and I also wanted to make that permission logic available to plugins as well. Moving those methods into the documented
Datasette class felt like a good way to achieve that.
- Table and row HTML pages now include a
<link rel="alternate" type="application/json+datasette" href="...">element and return a
Link: URL; rel="alternate"; type="application/json+datasette"HTTP header pointing to the JSON version of those pages. (#1533)
The idea behind this change originated with my experimental work on the datasette-notebook plugin, which aims to implement a combination wiki-dashboard system for Datasette. Development of that plugin is stalled for the moment.
I wanted to build a feature where you could paste in a URL to a Datasette query or filtered table and the plugin would then embed and display the results of that query on a page.
The problem I needed to solve was this: given a URL, how can I tell that it corresponds to a Datasette table or query? Especially if that URL might be hosted on a separate website entirely (why not support embedding Datasette tables from other instances?)
My solution was an HTTP header. You can now make a
HEAD request against a Datasette page and, if it corresponds to a table or view, you’ll get back a
Link: ... rel="alternate" header pointing to the JSON version of that page.
Here’s an example using
~ % curl -I https://latest.datasette.io/fixtures/facetable HTTP/2 200 link: https://latest.datasette.io/fixtures/facetable.json; rel="alternate"; type="application/json+datasette" cache-control: max-age=5 referrer-policy: no-referrer access-control-allow-origin: * access-control-allow-headers: Authorization access-control-expose-headers: Link content-type: text/html; charset=utf-8
I’m looking forward to building interesting features against this in the future.
- Canned queries are now shown at the top of the database page, directly below the SQL editor. Previously they were shown at the bottom, below the list of tables. (#1612)
- Datasette now has a default favicon. (#1603)
I originally created this in Figma, and then hand-edited it in Pixelmator.
SQLite and SpatiaLite occasionally use automatically created tables to power some of their functionality. These aren’t very interesting to regular users, so Datasette omits them from view by default on the homepage.
I have an open issue for Labels explaining what hidden tables are for since the current UI leads to legitimate questions from users who click on the “show hidden tables” link!
This was the impetus for dropping support for Python 3.6.
The tracer mechanism powers the debugging feature that shows all of the SQL queries that were executed to produce a page (demo here).
I’ve been experimenting with ways to run some of these queries in parallel, taking advantage of
asyncio. But the tracer mechanism wasn’t correctly tracking these, because queries executed in additional
asyncio tasks were not being correctly bundled together.
The Python standard library contextvars module provides a neat way to solve this, but it was introduced in Python 3.7. So I finally bit the bullet and dropped 3.6.
This means plugins can now do
from datasette import Response, where previously they had to use
from datasette.utils.asgi import Response.
I’ve long been frustrated that Django makes me remember where to import things from—so now Datasette lets the most commonly imported stuff (counted by running grep against my own plugins) from the root of the package.
/-/versionspage now returns additional details for libraries used by SpatiaLite. (#1607)
You can see a demo of that here.
- Documentation now links to the Datasette Tutorials.
I wrote about these new tutorials a few weeks ago.
And the rest:
More recent articles
- Datasette Enrichments: a new plugin framework for augmenting your data - 1st December 2023
- llamafile is the new best way to run a LLM on your own computer - 29th November 2023
- Prompt injection explained, November 2023 edition - 27th November 2023
- I'm on the Newsroom Robots podcast, with thoughts on the OpenAI board - 25th November 2023
- Weeknotes: DevDay, GitHub Universe, OpenAI chaos - 22nd November 2023
- Deciphering clues in a news article to understand how it was reported - 22nd November 2023
- Exploring GPTs: ChatGPT in a trench coat? - 15th November 2023
- Financial sustainability for open source projects at GitHub Universe - 10th November 2023
- ospeak: a CLI tool for speaking text in the terminal via OpenAI - 7th November 2023
- DALL-E 3, GPT4All, PMTiles, sqlite-migrate, datasette-edit-schema - 30th October 2023