Simon Willison’s Weblog

Weeknotes: more datasette-secrets, plus a mystery video project

7th May 2024

I introduced datasette-secrets two weeks ago. The core idea is to provide a way for end-users to store secrets such as API keys in Datasette, allowing other plugins to access them.

datasette-secrets 0.2 is the first non-alpha release of that project. The big new feature is that the plugin is now compatible with both the Datasette 1.0 alphas and the stable releases of Datasette (currently Datasette 0.64.6).

My policy at the moment is that a plugin that only works with the Datasette 1.0 alphas must itself be an alpha release. I’ve been feeling the weight of this as the number of plugins that depend on 1.0a has grown—on the one hand it’s a great reason to push through to that 1.0 stable release, but it’s painful to have so many features that are incompatible with current Datasette.

This came to a head with Datasette Enrichments. I wanted to start consuming secrets from enrichments such as datasette-enrichments-gpt and datasette-enrichments-opencage, but I didn’t want the whole enrichments ecosystem to become 1.0a only.

Patterns for plugins that work against multiple Datasette versions

I ended up building out quite a bit of infrastructure to help support plugins that work with both versions.

I already have a GitHub Actions pattern for running tests against both versions, which looks like this:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
        datasette-version: ["<1.0", ">=1.0a13"]
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}
        cache: pip
        cache-dependency-path: pyproject.toml
    - name: Install dependencies
      run: |
        pip install '.[test]'
        pip install "datasette${{ matrix.datasette-version }}"
    - name: Run tests
      run: |
        pytest

This uses a GitHub Actions matrix to run the test suite ten times—five against Datasette <1.0 on different Python versions and then five again on Datasette >=1.0a13.

One of the big changes in Datasette 1.0 involves the way plugins are configured. I have a datasette-test library to help paper over those differences, which can be used like this:

from datasette_test import Datasette

def test_something():
    datasette = Datasette(
        plugin_config={
            "datasette-secrets": {
                "database": "_internal",
                "encryption-key": TEST_ENCRYPTION_KEY,
            }
        },
        permissions={"manage-secrets": {"id": "admin"}},
    )

The plugin_config= argument there is unique to that datasette_test.Datasette() class constructor, and does the right thing against both versions of Datasette. permissions= is a similar utility function. Both are described in the datasette-test README.

The PR adding <1.0 and >1.0a compatibility has a few more details of changes I made to get datasette-secrets to work with both versions.

Here’s what the secrets management interface looks like now:

Adding secrets to enrichments

I ended up changing the core enrichments framework to add support for secrets. The new mechanism is documented here—but the short version is you can now define an Enrichments subclass that looks like this:

from datasette_enrichments import Enrichment
from datasette_secrets import Secret


class TrainEnthusiastsEnrichment(Enrichment):
    name = "Train Enthusiasts"
    slug = "train-enthusiasts"
    description = "Enrich with extra data from the Train Enthusiasts API"
    secret = Secret(
        name="TRAIN_ENTHUSIASTS_API_KEY",
        description="An API key from train-enthusiasts.doesnt.exist",
        obtain_url="https://train-enthusiasts.doesnt.exist/api-keys",
        obtain_label="Get an API key"
    )

This imaginary enrichment will now do the following:

1. If a TRAIN_ENTHUSIASTS_API_KEY environment variable is present it will use that without asking for an API key.
2. A user with sufficient permissions, in a properly configured Datasette instance, can visit the “Manage secrets” page to set that API key such that it will be encrypted and persisted in the Datasette invisible “internal” database.
3. If neither of those are true, the enrichment will ask for an API key every time a user tries to run it. That API key will be kept in memory, used and then discarded—it will not be persisted anywhere.

There are still a bunch more enrichments that need to be upgraded to the new pattern, but those upgrades are now a pretty straightforward process.

Mystery video

I’ve been collaborating on a really fun video project for the past few weeks. More on this when it’s finished, but it’s been a wild experience. I can’t wait to see how it turns out, and share it with the world.

Releases

• llm-openrouter 0.2—2024-05-03
  LLM plugin for models hosted by OpenRouter
• datasette-upload-dbs 0.3.2—2024-05-03
  Upload SQLite database files to Datasette
• ttok 0.3—2024-05-02
  Count and truncate text based on tokens
• datasette-enrichments 0.4.2—2024-04-27
  Tools for running enrichments against data stored in Datasette
• datasette-secrets 0.2—2024-04-26
  Manage secrets such as API keys for use with other Datasette plugins
• datasette-test 0.3.2—2024-04-26
  Utilities to help write tests for Datasette plugins and applications
• datasette-test-plugin 0.1—2024-04-26
  Part of datasette-test
• datasette-extract 0.1a6—2024-04-25
  Import unstructured data (text and images) into structured tables
• datasette-leaflet-geojson 0.8.2—2024-04-25
  Datasette plugin that replaces any GeoJSON column values with a Leaflet map.
• datasette-edit-schema 0.8a2—2024-04-24
  Datasette plugin for modifying table schemas

TILs

• Transcribing MP3s with whisper-cpp on macOS—2024-04-26