Weeknotes: Datasette 0.33

23rd December 2019

I released Datasette 0.33 yesterday. The release represents an accumulation of small changes and features since Datasette 0.32 back in November. Duplicating the release notes:

rowid is now included in dropdown menus for filtering tables (#636)
Columns are now only suggested for faceting if they have at least one value with more than one record (#638)
Queries with no results now display “0 results” (#637)
Improved documentation for the --static option (#641)
asyncio task information is now included on the /-/threads debug page
Bumped Uvicorn dependency 0.11
You can now use --port 0 to listen on an available port
New template_debug setting for debugging templates, e.g. https://latest.datasette.io/fixtures/roadsideattractions?context=1 (#654)

The last three items deserve extra explanation.

Port 0

I run a lot of Datasette instances on my laptop—I often have 5 or 6 running representing different projects or experiments.

Eventually this means I start to hit port conflicts. I’ll run datasette mydb.db -p 8005 and get an error because port 8005 is already in use by something else.

I asked on Twitter if there were any downsides to having Datasette automatically pick an unused port... and @davids and @dominicrodger pointed out that port 0 is a Unix convention for “pick a port for me”.

It turned out Datasette already supported this! If you ran datasette --port=0 the underlying Uvicorn server would indeed pick an unused port and start serving on it.

There was just one catch: the URL output to the console would show port 0, so it wasn’t clear which port to actually visit. This was a bug in Uvicorn.

Open source to the rescue... I filed an issue with Uvicorn and then constructed a pull request. Tom Christie merged it five hours later and shipped a release. Datasette 0.33 depends on that updated version of Uvicorn, so --port=0 now works as expected!

template_debug

I’m always looking for new ways to add debugging and development tools to Datasette. I added support for ?_trace=1 to see executed SQL queries back in May for example.

While thinking through a refactoring of Datasette’s template rendering to better support plugins, I realized there was an opportunity for better debugging around templates.

Datasette supports custom templates, which means template authors need a clear way to see what variables are available to their template logic.

I tried adding a ?_context=1 parameter to dump out the template context, and it instantly helped me better understand what was going on.

There was just one problem: what about secret information? By default Datasette doesn’t include any secrets (signing keys, API tokens etc) in its template context—but Datasette plugins are allowed to add things to the context, and I can’t guarantee that a future plugin won’t add something that shouldn’t be exposed by the ?_context=1 debugging parameter.

The solution was to have context debugging only available if the new template_debug configuration setting is turned on:

$ datasette --memory --config template_debug:1

I’ve set up the latest.datasette.io demo to run with this setting, so you can now view the context for any page on that site, like so:

Niche Museums

Natalie and I are in the UK this week. We spent a few days in London and managed to get in a few niche museums, including the first four on this week’s new listings on Niche Museums:

I also made a small fix to the site’s Atom feed: it now respects newlines in the entries, by rendering the description using Markdown.

Here’s the implementation. Since the Atom feed is defined by a SQL query (using datasette-atom) I took advantage of Datasette’s ability to load custom plugins from a per-site plugins/ directory and added a custom SQLite SQL function for rendering markdown, then added that function call to the query that defines the feed.

Posted 23rd December 2019 at 9:38 pm · Follow me on Mastodon or Twitter or subscribe to my newsletter

Simon Willison’s Weblog