Simon Willison’s Weblog

Datasette 0.61: The annotated release notes

24th March 2022

I released Datasette 0.61 this morning—closely followed by 0.61.1 to fix a minor bug. Here are the annotated release notes.

In preparation for Datasette 1.0, this release includes two potentially backwards-incompatible changes. Hashed URL mode has been moved to a separate plugin, and the way Datasette generates URLs to databases and tables with special characters in their name such as / and . has changed.

• URLs within Datasette now use a different encoding scheme for tables or databases that include “special” characters outside of the range of a-zA-Z0-9_-. This scheme is explained here: Tilde encoding. (#1657)
• Removed hashed URL mode from Datasette. The new datasette-hashed-urls plugin can be used to achieve the same result, see datasette-hashed-urls for details. (#1661)

I talked about these changes in my weeknotes. These are two major steps forward towards Datasette 1.0, in that implementing them removed a lot of code and complexity from Datasette core.

Datasette also now requires Python 3.7 or higher.

0.60 was the final release to work with Python 3.6, which ended security support last December.

• Databases can now have a custom path within the Datasette instance that is independent of the database name, using the db.route property. (#1668)

This was the last change I made before the 0.61 release, and turned out to have a severe bug which prompted me to release 0.61.1 shortly afterwards.

I needed the feature in order to implement the datasette-hashed-urls plugin.

That plugin works by modifying the URL relating to a database in order to incorporate the SHA-256 hash of that database’s contents. This allows all JSON and HTML pages within that database to set a far-future cache header, providing a huge performance boost especially if run behind a caching proxy such as Cloudflare.

Initially it worked by changing the display name of the database too, but Forest Gregg pointed out that exposing all of those hashes in the UI was a pretty poor user experience.

So I decided to split the “route” (the URL to the database—“path” was already in use as the path to the file on disk) from the unique name used to refer to the database internally.

I caught almost all of the places in the code that needed to be updated, but shortly after shipping 0.61 I noticed that Forest had filed issues relating to the one place I had missed!

Those fixes are out now, and datasette-hashed-urls 0.3 should work as advertised.

• Datasette is now covered by a Code of Conduct. (#1654)

It’s good open source governance to have one of these. I decided to adopt the Contributor Covenant because it reflected my own values for the project and is used by a large number of projects that I trust.

• Python 3.6 is no longer supported. (#1577)
• Tests now run against Python 3.11-dev. (#1621)

I didn’t test Datasette against Python 3.10 before it came out and was distressed to find an asyncio bug that caused errors with the project after that version of Python was released!

I’m not going to let that happen again, so Datasette’s test suite now runs against the 3.11 developer preview. Here’s my TIL about how I set that up.

• New datasette.ensure_permissions(actor, permissions) internal method for checking multiple permissions at once. (#1675)
• New datasette.check_visibility(actor, action, resource=None) internal method for checking if a user can see a resource that would otherwise be invisible to unauthenticated users. (#1678)

These two new permissions APIs came out of a larger effort to refactor and simplify Datasette’s core views.

Datasette’s BaseView class included permission logic. I’m trying to shrink that superclass down to the point where I can remove it entirely, and I also wanted to make that permission logic available to plugins as well. Moving those methods into the documented Datasette class felt like a good way to achieve that.

• Table and row HTML pages now include a <link rel="alternate" type="application/json+datasette" href="..."> element and return a Link: URL; rel="alternate"; type="application/json+datasette" HTTP header pointing to the JSON version of those pages. (#1533)
• Access-Control-Expose-Headers: Link is now added to the CORS headers, allowing remote JavaScript to access that header.

The idea behind this change originated with my experimental work on the datasette-notebook plugin, which aims to implement a combination wiki-dashboard system for Datasette. Development of that plugin is stalled for the moment.

I wanted to build a feature where you could paste in a URL to a Datasette query or filtered table and the plugin would then embed and display the results of that query on a page.

To support this, I built an experimental Web Component, datasette-table, and published it to npm (TIL).

The problem I needed to solve was this: given a URL, how can I tell that it corresponds to a Datasette table or query? Especially if that URL might be hosted on a separate website entirely (why not support embedding Datasette tables from other instances?)

My solution was an HTTP header. You can now make a HEAD request against a Datasette page and, if it corresponds to a table or view, you’ll get back a Link: ... rel="alternate" header pointing to the JSON version of that page.

Here’s an example using curl:

~ % curl -I https://latest.datasette.io/fixtures/facetable
HTTP/2 200 
link: https://latest.datasette.io/fixtures/facetable.json; rel="alternate"; type="application/json+datasette"
cache-control: max-age=5
referrer-policy: no-referrer
access-control-allow-origin: *
access-control-allow-headers: Authorization
access-control-expose-headers: Link
content-type: text/html; charset=utf-8

I’m looking forward to building interesting features against this in the future.

• Canned queries are now shown at the top of the database page, directly below the SQL editor. Previously they were shown at the bottom, below the list of tables. (#1612)

Jacob Fenton suggested this. Canned queries were previously listed at the very bottom of the database page, below the list of tables. They’re now shown at the top. I think this is a big improvement!

• Datasette now has a default favicon. (#1603)

I originally created this in Figma, and then hand-edited it in Pixelmator.

• sqlite_stat tables are now hidden by default. (#1587)
• SpatiaLite tables data_licenses, KNN and KNN2 are now hidden by default. (#1601)

SQLite and SpatiaLite occasionally use automatically created tables to power some of their functionality. These aren’t very interesting to regular users, so Datasette omits them from view by default on the homepage.

I have an open issue for Labels explaining what hidden tables are for since the current UI leads to legitimate questions from users who click on the “show hidden tables” link!

• SQL query tracing mechanism now works for queries executed in asyncio sub-tasks, such as those created by asyncio.gather(). (#1576)
• datasette.tracer mechanism is now documented.

This was the impetus for dropping support for Python 3.6.

The tracer mechanism powers the debugging feature that shows all of the SQL queries that were executed to produce a page (demo here).

I’ve been experimenting with ways to run some of these queries in parallel, taking advantage of asyncio. But the tracer mechanism wasn’t correctly tracking these, because queries executed in additional asyncio tasks were not being correctly bundled together.

The Python standard library contextvars module provides a neat way to solve this, but it was introduced in Python 3.7. So I finally bit the bullet and dropped 3.6.

• Common Datasette symbols can now be imported directly from the top-level datasette package, see Import shortcuts. Those symbols are Response, Forbidden, NotFound, hookimpl, actor_matches_allow. (#957)

This means plugins can now do from datasette import Response, where previously they had to use from datasette.utils.asgi import Response.

I’ve long been frustrated that Django makes me remember where to import things from—so now Datasette lets the most commonly imported stuff (counted by running grep against my own plugins) from the root of the package.

• /-/versions page now returns additional details for libraries used by SpatiaLite. (#1607)

You can see a demo of that here.

• Documentation now links to the Datasette Tutorials.

I wrote about these new tutorials a few weeks ago.

And the rest:

• Datasette will now also look for SpatiaLite in /opt/homebrew—thanks, Dan Peterson. (#1649)
• Fixed bug where custom pages did not work on Windows. Thanks, Robert Christie. (#1545)
• Fixed error caused when a table had a column named n. (#1228)