Simon Willison’s Weblog

Datasette 1.0a8: JavaScript plugins, new plugin hooks and plugin configuration in datasette.yaml

7th February 2024

I just released Datasette 1.0a8. These are the annotated release notes.

This alpha release continues the migration of Datasette’s configuration from metadata.yaml to the new datasette.yaml configuration file, introduces a new system for JavaScript plugins and adds several new plugin hooks.

My plan is for this to be the last alpha that adds new features—the new plugin hooks, in this case. The next release will focus on wrapping up the stable APIs for 1.0, with a particular focus on template stability (so users can customize Datasette without fear of it breaking in future minor releases) and wrapping up the work on the stable JSON API.

Configuration

• Plugin configuration now lives in the datasette.yaml configuration file, passed to Datasette using the -c/--config option. Thanks, Alex Garcia. (#2093)

  datasette -c datasette.yaml

  Where datasette.yaml contains configuration that looks like this:

  plugins:
    datasette-cluster-map:
      latitude_column: xlat
      longitude_column: xlon
  

• Previously plugins were configured in metadata.yaml, which was confusing as plugin settings were unrelated to database and table metadata.

This almost concludes the work (driven mainly by Alex Garcia) to clean up how Datasette is configured prior to the 1.0 release. Moving things that aren’t metadata out of the metadata.yaml/json file is a big conceptual improvement, and one that absolutely needed to happen before 1.0.

• The -s/--setting option can now be used to set plugin configuration as well. See Configuration via the command-line for details. (#2252)

  The above YAML configuration example using -s/--setting looks like this:

  datasette mydatabase.db\
    -s plugins.datasette-cluster-map.latitude_column xlat \
    -s plugins.datasette-cluster-map.longitude_column xlon

This feature is mainly for me. I start new Datasette instances dozens of times a day to try things out, and having to manually edit a datasette.yaml file before trying something new is an annoying little piece of friction.

With the -s option anything that can be represented in JSON or YAML can also be passed on the command-line.

I mainly love this as a copy-and-paste mechanism: my notes are crammed with datasette shell one-liners, and being able to paste something into my terminal to recreate a Datasette instance with a specific configuration is a big win.

The -s command uses dot-notation to specify nested keys, but it has a simple mechanism for representing more complex objects too: you can pass them in as JSON literal strings and Datasette will parse them. The --setting documentation includes this example of configuring datasette-proxy-url:

datasette mydatabase.db \
  -s plugins.datasette-proxy-url.paths '[{"path": "/proxy", "backend": "http://example.com/"}]'

Which is equivalent to the following datasette.yaml file:

plugins:
  datasette-proxy-url:
    paths:
    - path: /proxy
      backend: http://example.com/

• The new /-/config page shows the current instance configuration, after redacting keys that could contain sensitive data such as API keys or passwords. (#2254)

Datasette has a set of introspection endpoints like this—/-/metadata and /-/settings and /-/threads, all of which can have .json added to get back the raw JSON. I find them really useful for debugging instances and understanding how they have been configured.

The redaction is new: previously I had designed a mechanism for passing secrets as environment variables in a way that would avoid them being exposed here, but I realized automated redaction is less likely to cause people to leak secrets by accident.

• Existing Datasette installations may already have configuration set in metadata.yaml that should be migrated to datasette.yaml. To avoid breaking these installations, Datasette will silently treat table configuration, plugin configuration and allow blocks in metadata as if they had been specified in configuration instead. (#2247) (#2248) (#2249)

Originally the plan was to have Datasette fail to load if it spotted configuration in metadata.yaml that should have been migrated to datasette.yaml.

I changed my mind about this mainly as I experienced the enormous inconvenience of updating all of my Datasette instances to the new format—including rewriting the automated tests for my plugins.

I think my philosophy on this going forward is going to be that Datasette will take extra effort to keep older things working provided the additional code complexity in doing so is low enough to make it worth the trade-off. In this case I think it is.

Note that the datasette publish command has not yet been updated to accept a datasette.yaml configuration file. This will be addressed in #2195 but for the moment you can include those settings in metadata.yaml instead.

I promised myself I would ship 1.0a8 today no matter what, so I cut this feature at the last moment.

JavaScript plugins

Datasette now includes a JavaScript plugins mechanism, allowing JavaScript to customize Datasette in a way that can collaborate with other plugins.

This provides two initial hooks, with more to come in the future:

• makeAboveTablePanelConfigs() can add additional panels to the top of the table page.
• makeColumnActions() can add additional actions to the column menu.

Thanks Cameron Yick for contributing this feature. (#2052)

The core problem we are trying to solve here comes from what happens when multiple plugins all try to customize the Datasette instance at the same time.

This is particularly important for visualization plugins.

An example: datasette-cluster-map and datasette-geojson-map both add a map to the top of the table page. This means if you have both plugins installed you can end up with two maps!

The new mechanism allows plugins to collaborate: each plugin can contribute one or more “panels” which will then be shown above the table view in an interface with toggles to switch between them.

The column actions mechanism is similar: it allows plugins to contribute additional actions to the column menu, which appears when you click the cog icon in the header of a table column.

Cameron Yick did a great job with this feature. I’ve been slow in getting a release out with it though—my hope is that we can iterate more productively on it now that it’s in an alpha release.

Plugin hooks

• New jinja2_environment_from_request(datasette, request, env) plugin hook, which can be used to customize the current Jinja environment based on the incoming request. This can be used to modify the template lookup path based on the incoming request hostname, among other things. (#2225)

I wrote about my need for this in Page caching and custom templates for Datasette Cloud: I wanted a way to modify the Jinja environment based on the requested HTTP host, and this lets me do that.

• New family of template slot plugin hooks: top_homepage, top_database, top_table, top_row, top_query, top_canned_query. Plugins can use these to provide additional HTML to be injected at the top of the corresponding pages. (#1191)

Another long-running need (the issue is from January 2021). Similar to the JavaScript plugin mechanism, this allows multiple plugins to add content to the page without one plugin overwriting the other.

The new Datasette Events system

• New track_event() mechanism for plugins to emit and receive events when certain events occur within Datasette. (#2240)

  • Plugins can register additional event classes using register_events(datasette).
  • They can then trigger those events with the datasette.track_event(event) internal method.
  • Plugins can subscribe to notifications of events using the track_event(datasette, event) plugin hook.
  • Datasette core now emits login, logout, create-token, create-table, drop-table, insert-rows, upsert-rows, update-row, delete-row events, documented here.

Another hook inspired by Datasette Cloud. I want better analytics for that product to help track which features are being used, but I also wanted to do that in a privacy-forward manner. I decided to bake it into Datasette core and I intend to make it visible to the administrators of Datasette Cloud instances—so that it doubles as an audit log for what’s happening in their instances.

I realized that this has uses beyond analytics: if a plugin wants to do something extra any time a new table is created within Datasette it can use the track_events() plugin hook to listen out for the create-table event and take action when it occurs.

• New internal function for plugin authors: await db.execute_isolated_fn(fn), for creating a new SQLite connection, executing code and then closing that connection, all while preventing other code from writing to that particular database. This connection will not have the prepare_connection() plugin hook executed against it, allowing plugins to perform actions that might otherwise be blocked by existing connection configuration. (#2218)

This came about because I was trying to figure out a way to use prepare_connection() hook to add authorizers that prevent users from deleting certain tables, but found that doing this prevented VACUUM from working.

The new internal function provides a clean slate for plugins to do anything they like with a SQLite connection, while simultaneously preventing any write operations from other code from executing (even against other connections) until that isolated operation is complete.

Documentation

• Documentation describing how to write tests that use signed actor cookies using datasette.client.actor_cookie(). (#1830)
• Documentation on how to register a plugin for the duration of a test. (#2234)
• The configuration documentation now shows examples of both YAML and JSON for each setting.

I like including links to new documentation in the release notes, to give people a chance to catch useful new documentation that they might otherwise miss.

Minor fixes

• Datasette no longer attempts to run SQL queries in parallel when rendering a table page, as this was leading to some rare crashing bugs. (#2189)
• Fixed warning: DeprecationWarning: pkg_resources is deprecated as an API (#2057)
• Fixed bug where ?_extra=columns parameter returned an incorrectly shaped response. (#2230)

Surprisingly few bug fixes in this alpha—most of the work in the last few months has been new features. I think this is a good sign in terms of working towards a stable 1.0.