Simon Willison’s Weblog

Publish Python packages to PyPI with a python-lib cookiecutter template and GitHub Actions

16th January 2024

I use cookiecutter to start almost all of my Python projects. It helps me quickly generate a skeleton of a project with my preferred directory structure and configured tools.

I made some major upgrades to my python-lib cookiecutter template today. Here’s what it can now do to help you get started with a new Python library:

• Create a pyproject.toml file configured for use with setuptools. In my opinion this is the pattern with the current lowest learning curve—I wrote about that in detail in this TIL.
• Add a skeleton README and an Apache 2.0 LICENSE file.
• Create your_package/__init__.py for your code to go in.
• Create tests/test_your_package.py with a skeleton test.
• Include pytest as a test dependency.
• Configure GitHub Actions with two workflows in .github/workflows—one for running the tests against Python 3.8 through 3.12, and one for publishing releases of your package to PyPI.

The changes I made today are that I switched from setup.py to pyproject.toml, and I made a big improvement to how the publishing workflow authenticates with PyPI.

Publishing to PyPI with Trusted Publishing

My previous version of this template required you to jump through quite a few hoops to get PyPI publishing to work. You needed to create a PyPI token that could publish a new package, then paste that token into a GitHub Actions secret, then publish the package, and then disable that token and create a new one dedicated to just updating this package in the future.

The new version is much simpler, thanks to PyPI’s relatively new Trusted Publishers mechanism.

To publish a new package, you need to sign into PyPI and create a new “pending publisher”. Effectively you tell PyPI "My GitHub repository myname/name-of-repo should be allowed to publish packages with the name name-of-package".

Here’s that form for my brand new datasette-test library, the first library I published using this updated template:

Then create a release on GitHub, with a name that matches the version number from your pyproject.toml. Everything else should Just Work.

I wrote more about Trusted Publishing in this TIL.

Creating a package using a GitHub repository template

The most time consuming part of this project was getting my GitHub repository template to work properly.

There are two ways to use my cookiecutter template. You can use the cookiecutter command-line tool like this:

pipx install cookiecutter
cookiecutter gh:simonw/python-lib
# Answer a few questions here

But a more fun and convenient option is to use my GitHub repository template, simonw/python-lib-template-repository.

This lets you fill in a form on GitHub to create a new repository which will then execute the cookiecutter template for you and update itself with the result.

You can see an example of a repository created using this template at datasette/datasette-test.

Adding it all together

There are quite a lot of moving parts under the scenes here, but the end result is that anyone can now create a Python library with test coverage, GitHub CI and release automation by filling in a couple of forms and clicking some buttons.

For more details on how this all works, and how it’s evolved over time:

• A cookiecutter template for writing Datasette plugins from June 2020 describes my first experiments with cookiecutter
• Dynamic content for GitHub repository templates using cookiecutter and GitHub Actions from August 2021 describes my earliest attempts at using GitHub repository templates for this
• How to build, test and publish an open source Python library is a ten minute talk I gave at PyGotham in November 2021. It describes setup.py in detail, which is no longer my preferred approach.