DJP: A plugin system for Django

25th September 2024

DJP is a new plugin mechanism for Django, built on top of Pluggy. I announced the first version of DJP during my talk yesterday at DjangoCon US 2024, How to design and implement extensible software with plugins. I’ll post a full write-up of that talk once the video becomes available—this post describes DJP and how to use what I’ve built so far.

Why plugins?

Django already has a thriving ecosystem of third-party apps and extensions. What can a plugin system add here?

If you’ve ever installed a Django extension—such as django-debug-toolbar or django-extensions—you’ll be familiar with the process. You pip install the package, then add it to your list of INSTALLED_APPS in settings.py—and often configure other picees, like adding something to MIDDLEWARE or updating your urls.py with new URL patterns.

This isn’t exactly a huge burden, but it’s added friction. It’s also the exact kind of thing plugin systems are designed to solve.

DJP addresses this. You configure DJP just once, and then any additional DJP-enabled plugins you pip install can automatically register configure themselves within your Django project.

Setting up DJP

There are three steps to adding DJP to an existing Django project:

pip install djp—or add it to your requirements.txt or similar.

Modify your settings.py to add these two lines:

# Can be at the start of the file:
import djp

# This MUST be the last line:
djp.settings(globals())

Modify your urls.py to contain the following:

import djp

urlpatterns = [
    # Your existing URL patterns
] + djp.urlpatterns()

That’s everything. The djp.settings(globals()) line is a little bit of magic—it gives djp an opportunity to make any changes it likes to your configured settings.

You can see what that does here. Short version: it adds "djp" and any other apps from plugins to INSTALLED_APPS, modifies MIDDLEWARE for any plugins that need to do that and gives plugins a chance to modify any other settings they need to.

One of my personal rules of plugin system design is that you should never ship a plugin hook (a customization point) without releasing at least one plugin that uses it. This validates the design and provides executable documentation in the form of working code.

I’ve released three plugins for DJP so far.

django-plugin-django-header

django-plugin-django-header is a very simple initial example. It registers a Django middleware class that adds a Django-Composition: HTTP header to every response with the name of a random Composition by Django Reinhardt (thanks,Wikipedia).

pip install django-plugin-django-header

Then try it out with curl:

curl -I http://localhost:8000/

You should get back something like this:

...
Django-Composition: Nuages
...

I’m running this on my blog right now! Try this command to see it in action:

curl -I https://simonwillison.net/

The plugin is very simple. Its __init__.py registers middleware like this:

import djp

@djp.hookimpl
def middleware():
    return [
        "django_plugin_django_header.middleware.DjangoHeaderMiddleware"
    ]

That string references the middleware class in this file.

django-plugin-blog

django-plugin-blog is a much bigger example. It implements a full blog system for your Django application, with bundled models and templates and views and a URL configuration.

You’ll need to have configured auth and the Django admin already (those already there by default in the django-admin startproject template). Now install the plugin:

pip install django-plugin-blog

And run migrations to create the new database tables:

python manage.py migrate

That’s all you need to do. Navigating to /blog/ will present the index page of the blog, including a link to a working Atom feed.

You can add entries and tags through the Django admin (configured for you by the plugin) and those will show up on /blog/, get their own URLs at /blog/2024/<slug>/ and be included in the Atom feed, the /blog/archive/ list and the /blog/2024/ year-based index too.

The default design is very basic, but you can customize that by providing your own base template or providing custom templates for each of the pages. There are details on the templates in the README.

The blog implementation is directly adapted from my Building a blog in Django TIL.

The primary goal of this plugin is to demonstrate what a plugin with views, templates, models and a URL configuration looks like. Here’s the full __init__.py for the plugin:

from django.urls import path
from django.conf import settings
import djp

@djp.hookimpl
def installed_apps():
    return ["django_plugin_blog"]

@djp.hookimpl
def urlpatterns():
    from .views import index, entry, year, archive, tag, BlogFeed

    blog = getattr(settings, "DJANGO_PLUGIN_BLOG_URL_PREFIX", None) or "blog"
    return [
        path(f"{blog}/", index, name="django_plugin_blog_index"),
        path(f"{blog}/<int:year>/<slug:slug>/", entry, name="django_plugin_blog_entry"),
        path(f"{blog}/archive/", archive, name="django_plugin_blog_archive"),
        path(f"{blog}/<int:year>/", year, name="django_plugin_blog_year"),
        path(f"{blog}/tag/<slug:slug>/", tag, name="django_plugin_blog_tag"),
        path(f"{blog}/feed/", BlogFeed(), name="django_plugin_blog_feed"),
    ]

It still only needs to implement two hooks: one to add django_plugin_blog to the INSTALLED_APPS list and another to add the necessary URL patterns to the project.

The from .views import ... line is nested inside the urlpatterns() hook because I was hitting circular import issues with those imports at the top of the module.

django-plugin-database-url

django-plugin-database-url is the smallest of my example plugins. It exists mainly to exercise the settings() plugin hook, which allows plugins to further manipulate settings in any way they like.

Quoting the README:

Once installed, any DATABASE_URL environment variable will be automatically used to configure your Django database setting, using dj-database-url.

Here’s the full implementation of that plugin, most of which is copied straight from the dj-database-url documentation:

import djp
import dj_database_url

@djp.hookimpl
def settings(current_settings):
    current_settings["DATABASES"]["default"] = dj_database_url.config(
        conn_max_age=600,
        conn_health_checks=True,
    )

If DJP gains traction, I expect that a lot of plugins will look like this—thin wrappers around existing libraries where the only added value is that they configure those libraries automatically once the plugin is installed.

Writing a plugin

A plugin is a Python package bundling a module that implements one or more of the DJP plugin hooks.

As I’ve shown above, the Python code for plugins can be very short. The larger challenge is correctly packaging and distributing the plugin—plugins are discovered using Entry Points which are defined in a pyproject.toml file, and you need to get those exactly right for your plugin to be discovered.

DJP includes documentation on creating a plugin, but to make it as frictionless as possible I’ve released a new django-plugin cookiecutter template.

This means you can start a new plugin like this:

pip install cookiecutter
cookiecutter gh:simonw/django-plugin

Then answer the questions:

  [1/6] plugin_name (): django-plugin-example
  [2/6] description (): A simple example plugin
  [3/6] hyphenated (django-plugin-example):
  [4/6] underscored (django_plugin_example):
  [5/6] github_username (): simonw
  [6/6] author_name (): Simon Willison

And you’l get a django-plugin-example directory with a fully configured plugin ready to be published to PyPI.

The template includes a .github/workflows directory with actions that can run tests, and an action that publishes your plugin to PyPI any time you create a new release on GitHub.

I’ve used that pattern myself for hundreds of plugin projects for Datasette and LLM, so I’m confident this is an effective way to release plugins.

The workflows use PyPI’s Trusted Publishers mechanism (see my TIL), which means you don’t need to worry about API keys or PyPI credentials—configure the GitHub repo once using the PyPI UI and everything should just work.

Writing tests for plugins

Writing tests for plugins can be a little tricky, especially if they need to spin up a full Django environemnt in order to run the tests.

I previously published a TIL about that, showing how to have tests with their own tests/test_project project that can be used by pytest-django.

I’ve baked that pattern into the simon/django-plugin cookiecutter template as well, plus a single default test which checks that a hit to the / index page returns a 200 status code—still a valuable default test since it confirms the plugin hasn’t broken everything!

The tests for django-plugin-django-header and for django-plugin-blog should provide a useful starting point for writing tests for your own plugins.

Why call it DJP?

Because django-plugins already existed on PyPI, and I like my three letter acronyms there!

What’s next for DJP?

I presented this at DjangoCon US 2024 yesterday afternoon. Initial response seemed positive, and I’m going to be attending the conference sprints on Thursday morning to see if anyone wants to write their own plugin or help extend the system further.

Is this a good idea? I think so. Plugins have been transformative for both Datasette and LLM, and I think Pluggy provides a mature, well-designed foundation for this kind of system.

I’m optimistic about plugins as a natural extension of Django’s existing ecosystem. Let’s see where this goes.

Posted 25th September 2024 at 2 pm · Follow me on Mastodon or Twitter or subscribe to my newsletter

Simon Willison’s Weblog