Simon Willison’s Weblog


Weeknotes: more datasette-secrets, plus a mystery video project

7th May 2024

I introduced datasette-secrets two weeks ago. The core idea is to provide a way for end-users to store secrets such as API keys in Datasette, allowing other plugins to access them.

datasette-secrets 0.2 is the first non-alpha release of that project. The big new feature is that the plugin is now compatible with both the Datasette 1.0 alphas and the stable releases of Datasette (currently Datasette 0.64.6).

My policy at the moment is that a plugin that only works with the Datasette 1.0 alphas must itself be an alpha release. I’ve been feeling the weight of this as the number of plugins that depend on 1.0a has grown—on the one hand it’s a great reason to push through to that 1.0 stable release, but it’s painful to have so many features that are incompatible with current Datasette.

This came to a head with Datasette Enrichments. I wanted to start consuming secrets from enrichments such as datasette-enrichments-gpt and datasette-enrichments-opencage, but I didn’t want the whole enrichments ecosystem to become 1.0a only.

Patterns for plugins that work against multiple Datasette versions

I ended up building out quite a bit of infrastructure to help support plugins that work with both versions.

I already have a GitHub Actions pattern for running tests against both versions, which looks like this:

    runs-on: ubuntu-latest
        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
        datasette-version: ["<1.0", ">=1.0a13"]
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v5
        python-version: ${{ matrix.python-version }}
        cache: pip
        cache-dependency-path: pyproject.toml
    - name: Install dependencies
      run: |
        pip install '.[test]'
        pip install "datasette${{ matrix.datasette-version }}"
    - name: Run tests
      run: |

This uses a GitHub Actions matrix to run the test suite ten times—five against Datasette <1.0 on different Python versions and then five again on Datasette >=1.0a13.

One of the big changes in Datasette 1.0 involves the way plugins are configured. I have a datasette-test library to help paper over those differences, which can be used like this:

from datasette_test import Datasette

def test_something():
    datasette = Datasette(
            "datasette-secrets": {
                "database": "_internal",
                "encryption-key": TEST_ENCRYPTION_KEY,
        permissions={"manage-secrets": {"id": "admin"}},

The plugin_config= argument there is unique to that datasette_test.Datasette() class constructor, and does the right thing against both versions of Datasette. permissions= is a similar utility function. Both are described in the datasette-test README.

The PR adding <1.0 and >1.0a compatibility has a few more details of changes I made to get datasette-secrets to work with both versions.

Here’s what the secrets management interface looks like now:

Manage secrets creen in Datasette Cloud. Simon Willison is logged in. A secret called OpenAI_API_KEY is at version 1, last updated by swillison on 25th April.

Adding secrets to enrichments

I ended up changing the core enrichments framework to add support for secrets. The new mechanism is documented here—but the short version is you can now define an Enrichments subclass that looks like this:

from datasette_enrichments import Enrichment
from datasette_secrets import Secret

class TrainEnthusiastsEnrichment(Enrichment):
    name = "Train Enthusiasts"
    slug = "train-enthusiasts"
    description = "Enrich with extra data from the Train Enthusiasts API"
    secret = Secret(
        description="An API key from train-enthusiasts.doesnt.exist",
        obtain_label="Get an API key"

This imaginary enrichment will now do the following:

  1. If a TRAIN_ENTHUSIASTS_API_KEY environment variable is present it will use that without asking for an API key.
  2. A user with sufficient permissions, in a properly configured Datasette instance, can visit the “Manage secrets” page to set that API key such that it will be encrypted and persisted in the Datasette invisible “internal” database.
  3. If neither of those are true, the enrichment will ask for an API key every time a user tries to run it. That API key will be kept in memory, used and then discarded—it will not be persisted anywhere.

There are still a bunch more enrichments that need to be upgraded to the new pattern, but those upgrades are now a pretty straightforward process.

Mystery video

I’ve been collaborating on a really fun video project for the past few weeks. More on this when it’s finished, but it’s been a wild experience. I can’t wait to see how it turns out, and share it with the world.