Simon Willison’s Weblog

A cookiecutter template for writing Datasette plugins

20th June 2020

Datasette’s plugin system is one of the most interesting parts of the entire project. As I explained to Matt Asay in this interview, the great thing about plugins is that Datasette can gain new functionality overnight without me even having to review a pull request. I just need to get more people to write them!

datasette-plugin is my most recent effort to help make that as easy as possible. It’s a cookiecutter template that sets up the outline of a new plugin, combining various best patterns I’ve discovered over the past two years of writing my own plugins.

Once you’ve installed cookiecutter you can start building a new plugin by running:

cookiecutter gh:simonw/datasette-plugin

Cookiecutter will run a quick interactive session asking for a few details. It will then use those details to generate a new directory structure ready for you to start work on the plugin.

The datasette-plugin README describes the next steps. A couple of things are worth exploring in more detail.

Writing tests for plugins

I’m a big believer in automated testing: every single one of my plugins includes tests, and those test are run against every commit and must pass before new packages are shipped to PyPI.

In my experience the hardest part of writing tests is getting them started: setting up an initial test harness and ensuring that new tests can be easily written.

datasette-plugin adds pytest as a testing dependency and creates a tests/ folder with an initial, passing unit test in it.

The test confirms that the new plugin has been correctly installed, by running a request through a configured Datasette instance and hitting the /-/plugins.json introspection endpoint.

In doing so, it demonstrates how to run tests that interact with Datasette’s HTTP API. This is a very productive way to write tests.

The example test uses the HTTPX Python library. HTTPX offers a requests-style API but with a couple of crucial improvements. Firstly, it’s been built with asyncio support as a top-level concern. Secondly, it understands the ASGI protocol and can be run directly against an ASGI Python interface without needing to spin up an actual HTTP server. Since Datasette speaks ASGI this makes it the ideal tool for testing Datasette plugins.

Here’s that first test that gets created by the cookiecutter template:

from datasette.app import Datasette
import pytest
import httpx

@pytest.mark.asyncio
async def test_plugin_is_installed():
    app = Datasette([], memory=True).app()
    async with httpx.AsyncClient(app=app) as client:
        response = await client.get(
            "http://localhost/-/plugins.json"
        )
        assert 200 == response.status_code
        installed_plugins = {
            p["name"] for p in response.json()
        }
        assert "datasette-plugin-template-demo" in installed_plugins

My hope is that including a passing test that demonstrates how to execute test requests will make it much easier for plugin authors to start building out their own custom test suite.

Continuous integration with GitHub Actions

My favourite thing about GitHub Actions is that they’re enabled on every GitHub repository for free, without any extra configuration necessary.

The datasette-plugin template takes advantage of this. Not only does every new project get a passing test—it also gets a GitHub Action—in .github/workflows/test.yml—that executes the tests on every commit.

It even runs the test suite in parallel against Python 3.6, 3.7 and 3.8—the versions currently supported by Datasette itself.

A second action in .github/workflows/publish.yml bakes in my opinions on the best way to manage plugin releases: it builds and ships a new package to PyPI every time a new tag (and corresponding GitHub release) is added to the repository.

For this to work you’ll need to create a PyPI API token and add it to your plugin’s GitHub repository as a PYPI_TOKEN secret. This is explained in the README.

Deploying a live demo of the template with GitHub Actions

Whenever possible, I like to ship my projects with live demos. The Datasette repository publishes a demo of the latest commit to https://latest.datasette.io/ on every commit. I try to do the same for my plugins, where it makes sense to do so.

What could a live demo of a cookiecutter template look like?

Ideally it would show a complete, generated project. I love GitHub’s code browsing interface, so a separate repository containing that generated project would be ideal.

So that’s what https://github.com/simonw/datasette-plugin-template-demo is: it’s a repository showing the most recent output of the latest version of the cookiecutter template that lives in https://github.com/simonw/datasette-plugin.

It’s powered by this GitHub Action, which runs on every push to the datasette-plugin repo, installs cookiecutter, uses cookiecutter against some fixed inputs to re-generate the project and then pushes the results up to datasette-plugin-template-demo as a new commit.

As a fun final touch, it uses the GitHub commit comments API to add a comment to the commit to datasette-plugin linking to the “browse” view on the resulting code in the datasette-plugin-template-demo repository. Here’s one of those commit comments.

Figuring out how to build this took quite a bit of work. Issue #4 has a blow-by-blow rundown of how I got it working.

I couldn’t resist tweeting about it:

Writing a GitHub Action for a repo that generates content for a second repo and pushes that content to the second repo and then posts a comment to the commit on the first repo that links to the newly committed code in the second repo

- Simon Willison (@simonw) June 19, 2020